2024-03-24T11:35:08.205 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T11:35:08.205 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942 --verbose --description orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942/orig.config.yaml 2024-03-24T11:35:08.245 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:35:08.336 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617942' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3JepwCWgr2yIJfaYMnolJhwy5Vh+NFlByQ2I6OMMzvfbKd8fFl+1ZS7elOZvNvwMQvB4EBeUM23bQLAmhU8R0= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOk9H8eEFFWVNlMKDLkBlGm/7Z0sxrZzum1Fu/7ki/WiPNgwmx2N/KtZ6zWh2I40SNDK2JBOx8Ug/KITGNVkyCM= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T11:35:08.336 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T11:35:08.337 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T11:35:08.337 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T11:35:08.339 INFO:teuthology.task.internal:Checking packages... 2024-03-24T11:35:08.362 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T11:35:08.362 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:35:08.362 INFO:teuthology.packaging:ref: None 2024-03-24T11:35:08.362 INFO:teuthology.packaging:tag: None 2024-03-24T11:35:08.362 INFO:teuthology.packaging:branch: squid 2024-03-24T11:35:08.362 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:35:08.363 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T11:35:08.509 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T11:35:08.511 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T11:35:08.518 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T11:35:08.518 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T11:35:08.524 INFO:teuthology.task.internal:Saving configuration 2024-03-24T11:35:08.537 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T11:35:08.544 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:35:08.568 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:33.339692', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3JepwCWgr2yIJfaYMnolJhwy5Vh+NFlByQ2I6OMMzvfbKd8fFl+1ZS7elOZvNvwMQvB4EBeUM23bQLAmhU8R0='} 2024-03-24T11:35:08.593 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:33.340900', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOk9H8eEFFWVNlMKDLkBlGm/7Z0sxrZzum1Fu/7ki/WiPNgwmx2N/KtZ6zWh2I40SNDK2JBOx8Ug/KITGNVkyCM='} 2024-03-24T11:35:08.593 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T11:35:08.600 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-24T11:35:08.600 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-24T11:35:08.600 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T11:35:08.704 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0ebcecd0d0>, signals=[15]) 2024-03-24T11:35:08.704 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T11:35:08.710 INFO:teuthology.task.internal:Opening connections... 2024-03-24T11:35:08.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-03-24T11:35:08.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:35:08.819 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-03-24T11:35:08.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:35:08.909 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T11:35:08.916 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-03-24T11:35:08.934 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-03-24T11:35:08.935 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-03-24T11:35:08.991 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-03-24T11:35:08.991 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:35:08.992 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:35:08.993 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-03-24T11:35:09.020 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-03-24T11:35:09.038 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-03-24T11:35:09.038 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-03-24T11:35:09.094 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-03-24T11:35:09.094 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-03-24T11:35:09.094 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-03-24T11:35:09.094 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:35:09.094 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-03-24T11:35:09.094 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:35:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:35:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-03-24T11:35:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:35:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:35:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:35:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:35:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:35:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:35:09.095 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-03-24T11:35:09.119 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T11:35:09.128 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T11:35:09.133 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T11:35:09.133 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:35:09.135 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:35:09.151 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T11:35:09.157 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T11:35:09.157 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:35:09.193 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:35:09.218 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T11:35:09.278 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T11:35:09.292 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T11:35:09.292 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T11:35:09.293 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:35:09.293 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:35:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.310 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-03-24T11:35:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.311 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-24T11:35:09.368 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:35:09.368 INFO:teuthology.task.kernel:Running kernel on smithi193: 5.14.0-361.el9.x86_64 2024-03-24T11:35:09.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2024-03-24T11:35:09.369 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:35:09.369 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-361.el9.x86_64 2024-03-24T11:35:09.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-03-24T11:35:11.536 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-24T11:35:11.603 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T11:35:12.987 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-24T11:35:13.353 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-24T11:35:14.011 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T11:35:16.029 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-24T11:35:22.759 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 881 kB/s | 8.0 MB 00:09 2024-03-24T11:35:23.879 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.1 MB 00:04 2024-03-24T11:35:25.230 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 16 kB 00:00 2024-03-24T11:35:25.335 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-24T11:35:29.144 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.1 MB 00:00 2024-03-24T11:35:31.345 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-03-24T11:35:32.121 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-03-24T11:35:37.104 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-24T11:35:39.227 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:35:39.228 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:35:39.317 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:35:39.328 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:35:39.328 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-24T11:35:39.328 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout:Install 4 Packages 2024-03-24T11:35:39.329 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:35:39.330 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 76 M 2024-03-24T11:35:39.330 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 123 M 2024-03-24T11:35:39.330 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:35:39.480 INFO:teuthology.orchestra.run.smithi193.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 408 kB/s | 44 kB 00:00 2024-03-24T11:35:39.814 INFO:teuthology.orchestra.run.smithi193.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 34 MB/s | 15 MB 00:00 2024-03-24T11:35:40.222 INFO:teuthology.orchestra.run.smithi193.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 39 MB/s | 33 MB 00:00 2024-03-24T11:35:40.473 INFO:teuthology.orchestra.run.smithi193.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 28 MB/s | 27 MB 00:00 2024-03-24T11:35:40.473 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:35:40.473 INFO:teuthology.orchestra.run.smithi193.stdout:Total 66 MB/s | 76 MB 00:01 2024-03-24T11:35:41.136 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:35:41.503 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:35:41.503 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:35:42.556 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:35:42.557 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:35:45.056 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:35:45.457 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-24T11:35:45.732 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:35:45.822 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:35:48.544 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:35:48.683 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:35:50.529 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-03-24T11:35:52.677 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:35:52.678 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:35:52.779 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-03-24T11:35:52.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:35:52.787 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 76 M 2024-03-24T11:35:52.787 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 123 M 2024-03-24T11:35:52.787 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:35:53.038 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 340 kB/s | 44 kB 00:00 2024-03-24T11:35:53.384 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:35:53.425 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-24T11:35:54.139 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-24T11:35:54.806 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-24T11:35:54.806 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:35:54.806 INFO:teuthology.orchestra.run.smithi064.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-24T11:35:55.498 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:35:55.930 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:35:55.930 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:35:56.988 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:35:56.988 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:35:58.156 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:35:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:36:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:36:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:03.193 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:03.331 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:08.433 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:08.474 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:27.419 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:27.419 INFO:teuthology.orchestra.run.smithi193.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:36:27.419 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:36:27.463 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:27.812 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:27.813 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:36:27.813 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:27.813 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:28.249 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:28.249 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:36:28.249 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:36:28.249 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:28.250 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:36:28.250 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:36:28.250 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:36:28.250 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:36:28.250 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:36:28.569 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:36:28.681 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:28.681 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:36:28.681 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:36:28.681 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:36:28.681 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:36:28.682 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T11:36:28.682 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:36:28.682 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:36:28.762 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:28.763 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:36:28.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo rpm -qi grub2-tools 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Name : grub2-tools 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Epoch : 1 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Version : 2.06 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Release : 68.el9 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: x86_64 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Group : Unspecified 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Size : 8264254 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:License : GPLv3+ 2024-03-24T11:36:28.801 INFO:teuthology.orchestra.run.smithi193.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:Packager : builder@centos.org 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:Vendor : CentOS 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:Summary : Support tools for GRUB. 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:Description : 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:36:28.802 INFO:teuthology.orchestra.run.smithi193.stdout:hardware devices. 2024-03-24T11:36:28.803 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:36:28.803 INFO:teuthology.orchestra.run.smithi193.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:36:28.803 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:36:28.804 INFO:teuthology.task.kernel:Updating grub on smithi193 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:36:28.804 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:36:29.451 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2024-03-24T11:36:30.889 INFO:teuthology.orchestra.run.smithi193.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:36:30.910 INFO:teuthology.orchestra.run.smithi193.stderr:done 2024-03-24T11:36:30.912 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-03-24T11:36:30.967 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.4g9JTTI3zz 2024-03-24T11:36:30.967 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4g9JTTI3zz 2024-03-24T11:36:31.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.4g9JTTI3zz 2024-03-24T11:36:31.187 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.4g9JTTI3zz is 6KB 2024-03-24T11:36:31.236 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.4g9JTTI3zz 2024-03-24T11:36:31.251 DEBUG:teuthology.orchestra.run.smithi193:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:36:31.315 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:36:31.315 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:36:31.315 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:36:31.316 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:36:31.317 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:36:31.462 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2024-03-24T11:36:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:44.418 INFO:teuthology.orchestra.run.smithi064.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:36:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:36:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:36:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:36:45.287 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:36:45.629 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:36:45.771 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:45.771 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:36:45.771 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:36:45.772 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:36:45.772 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:36:45.772 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T11:36:45.772 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:36:45.772 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:36:45.854 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:45.855 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:36:45.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:36:45.892 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:36:45.893 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:36:45.894 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:36:45.894 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:36:45.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:36:46.509 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-03-24T11:36:48.010 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:36:48.030 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-03-24T11:36:48.032 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-24T11:36:48.048 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.Bj2zwkgy3X 2024-03-24T11:36:48.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Bj2zwkgy3X 2024-03-24T11:36:48.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.Bj2zwkgy3X 2024-03-24T11:36:48.266 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.Bj2zwkgy3X is 6KB 2024-03-24T11:36:48.318 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.Bj2zwkgy3X 2024-03-24T11:36:48.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:36:48.400 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:36:48.400 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:36:48.400 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:36:48.401 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:36:48.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:36:48.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-03-24T11:37:01.478 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T11:37:01.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T11:37:01.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:37:18.582 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T11:37:18.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-24T11:37:18.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:37:19.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2024-03-24T11:37:28.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T11:37:28.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:37:31.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2024-03-24T11:37:37.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-24T11:37:43.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T11:37:43.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:37:46.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-24T11:37:46.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:37:49.121 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-24T11:38:01.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-24T11:38:01.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:02.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2024-03-24T11:38:04.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-24T11:38:17.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T11:38:17.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:17.847 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-24T11:38:18.248 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T11:38:18.249 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:38:18.249 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-03-24T11:38:18.263 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:38:18.263 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:38:18.263 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:38:18.263 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:38:19.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-24T11:38:19.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:19.264 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T11:38:19.264 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:38:19.325 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2024-03-24T11:38:19.380 DEBUG:teuthology.parallel:result is None 2024-03-24T11:38:22.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-24T11:38:40.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-24T11:38:40.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:40.599 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-24T11:38:41.268 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-24T11:38:41.268 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:38:41.268 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-24T11:38:41.281 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:38:41.281 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:38:41.281 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:38:41.282 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:38:42.282 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T11:38:42.283 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:38:42.344 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-03-24T11:38:42.389 DEBUG:teuthology.parallel:result is None 2024-03-24T11:38:42.389 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T11:38:42.397 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T11:38:42.398 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:38:42.400 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:38:42.419 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T11:38:42.426 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T11:38:42.433 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T11:38:42.433 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:38:42.459 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:38:42.492 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T11:38:42.499 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T11:38:42.500 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:38:42.524 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:38:42.560 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:38:42.560 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:38:42.573 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:38:42.574 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:38:42.575 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T11:38:42.582 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T11:38:42.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:38:42.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:38:42.651 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T11:38:42.661 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T11:38:42.661 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:38:42.693 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:38:42.709 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:38:42.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:38:42.852 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:38:42.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:38:42.983 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:38:42.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:38:43.052 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:38:43.096 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:38:43.161 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:38:43.206 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:38:43.250 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:38:43.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:38:43.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-03-24T11:38:43.322 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-03-24T11:38:43.369 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:38:43.415 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:38:43.839 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T11:38:43.847 INFO:teuthology.task.internal:Starting timer... 2024-03-24T11:38:43.847 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T11:38:43.858 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T11:38:43.868 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T11:38:43.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-03-24T11:38:43.925 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-03-24T11:38:43.927 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-03-24T11:38:43.988 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-03-24T11:38:43.990 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T11:38:43.990 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-03-24T11:38:44.014 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-03-24T11:38:44.014 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-03-24T11:38:44.040 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-03-24T11:38:44.040 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-03-24T11:38:44.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:38:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:38:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.111 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T11:38:44.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:38:44.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:38:44.195 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T11:38:44.204 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T11:38:44.221 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T11:38:44.222 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T11:43:03.886 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-03-24T11:43:03.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-24T11:43:03.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:43:03.980 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-24T11:43:04.065 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-24T11:43:04.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T11:43:04.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:43:04.155 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-24T11:43:04.233 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T11:43:04.233 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T11:43:04.244 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T11:43:04.244 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:43:04.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:43:04.246 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:43:04.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:43:04.275 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:43:04.291 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:43:04.305 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:43:04.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-03-24T11:43:04.321 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:43:04.334 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-03-24T11:43:04.351 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-03-24T11:43:04.351 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:43:04.364 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-03-24T11:43:04.367 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:43:04.381 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:43:04.397 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:43:04.422 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:43:04.424 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:43:04.424 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-24T11:43:04.424 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:43:04.424 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:43:04.424 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:43:04.424 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:43:04.451 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:43:04.454 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:43:04.454 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-24T11:43:04.455 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:43:04.455 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:43:04.455 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:43:04.455 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:43:04.456 INFO:teuthology.run_tasks:Running task pexec... 2024-03-24T11:43:04.465 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-24T11:43:04.465 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi064.front.sepia.ceph.com 2024-03-24T11:43:04.465 DEBUG:teuthology.orchestra.run.smithi064:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:43:04.466 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi193.front.sepia.ceph.com 2024-03-24T11:43:04.466 DEBUG:teuthology.orchestra.run.smithi193:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:43:05.057 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:48 ago on Sun 24 Mar 2024 11:42:16 AM UTC. 2024-03-24T11:43:05.078 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:51 ago on Sun 24 Mar 2024 11:42:13 AM UTC. 2024-03-24T11:43:05.209 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-24T11:43:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:43:05.222 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:43:05.225 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-03-24T11:43:05.225 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 9.8 M 2024-03-24T11:43:05.225 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:43:05.238 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:43:05.238 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-24T11:43:05.238 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:43:05.238 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:43:05.238 INFO:teuthology.orchestra.run.smithi193.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:43:05.239 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:43:05.239 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:43:05.239 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:43:05.239 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-24T11:43:05.239 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:43:05.246 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 3.1 M 2024-03-24T11:43:05.246 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 9.8 M 2024-03-24T11:43:05.246 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:43:05.505 INFO:teuthology.orchestra.run.smithi193.stdout:runc-1.1.12-2.el9.x86_64.rpm 16 MB/s | 3.1 MB 00:00 2024-03-24T11:43:05.505 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:43:05.505 INFO:teuthology.orchestra.run.smithi193.stdout:Total 12 MB/s | 3.1 MB 00:00 2024-03-24T11:43:05.541 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:43:05.546 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:43:05.546 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:43:05.625 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:43:05.625 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:43:05.827 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:43:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:43:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:43:06.792 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:43:06.792 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:43:06.792 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:43:06.792 INFO:teuthology.orchestra.run.smithi193.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:43:06.792 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:43:06.792 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:43:07.054 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:09.771 INFO:teuthology.orchestra.run.smithi064.stdout:runc-1.1.12-2.el9.x86_64.rpm 715 kB/s | 3.1 MB 00:04 2024-03-24T11:43:09.771 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:43:09.771 INFO:teuthology.orchestra.run.smithi064.stdout:Total 705 kB/s | 3.1 MB 00:04 2024-03-24T11:43:09.810 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:43:09.817 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:43:09.817 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:43:09.902 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:43:09.902 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:43:10.131 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:43:10.226 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:43:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:43:11.288 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:43:11.289 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:43:11.289 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:43:11.289 INFO:teuthology.orchestra.run.smithi064.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:43:11.289 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:43:11.289 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:43:11.533 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:11.534 INFO:teuthology.run_tasks:Running task install... 2024-03-24T11:43:11.542 DEBUG:teuthology.task.install:project ceph 2024-03-24T11:43:11.542 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}} 2024-03-24T11:43:11.542 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:43:11.542 INFO:teuthology.task.install:Using flavor: default 2024-03-24T11:43:11.548 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T11:43:11.548 INFO:teuthology.task.install:extra packages: [] 2024-03-24T11:43:11.549 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T11:43:11.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:43:11.551 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T11:43:11.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:43:11.692 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/ 2024-03-24T11:43:11.692 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1667.gdb0330b1 2024-03-24T11:43:11.699 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/ 2024-03-24T11:43:11.699 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1667.gdb0330b1 2024-03-24T11:43:11.810 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T11:43:11.811 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:43:11.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T11:43:11.844 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T11:43:11.844 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/db0330b1e4e2470d52b750e251e55a522b4f7d69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T11:43:11.861 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T11:43:11.861 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:43:11.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T11:43:11.888 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T11:43:11.889 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/db0330b1e4e2470d52b750e251e55a522b4f7d69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T11:43:11.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T11:43:11.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T11:43:11.998 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T11:43:12.036 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T11:43:12.067 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-03-24T11:43:12.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-03-24T11:43:12.102 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-03-24T11:43:12.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-03-24T11:43:12.313 INFO:teuthology.orchestra.run.smithi193.stdout:74 files removed 2024-03-24T11:43:12.340 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw 2024-03-24T11:43:12.369 INFO:teuthology.orchestra.run.smithi064.stdout:74 files removed 2024-03-24T11:43:12.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-03-24T11:43:13.009 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 221 kB/s | 82 kB 00:00 2024-03-24T11:43:13.229 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 155 kB/s | 82 kB 00:00 2024-03-24T11:43:13.369 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-24T11:43:13.604 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-24T11:43:13.711 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-03-24T11:43:14.138 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-24T11:43:14.154 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-03-24T11:43:14.557 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-24T11:43:15.522 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-24T11:43:16.098 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-24T11:43:16.479 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-24T11:43:17.814 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-03-24T11:43:18.230 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-24T11:43:19.874 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-24T11:43:22.172 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-24T11:43:23.623 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-24T11:43:31.399 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 790 kB/s | 6.1 MB 00:07 2024-03-24T11:43:32.741 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 106 kB/s | 16 kB 00:00 2024-03-24T11:43:33.602 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-03-24T11:43:38.326 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 21 MB 00:14 2024-03-24T11:43:38.668 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-24T11:43:41.199 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:43:41.201 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:43:41.201 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:43:41.201 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:43:41.201 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:43:41.201 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 9.8 M 2024-03-24T11:43:41.201 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.4 M 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.3 M 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 20 M 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 25 k 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 742 k 2024-03-24T11:43:41.202 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 498 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.1 M 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 45 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 131 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 161 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T11:43:41.203 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 320 k 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 298 k 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 100 k 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-03-24T11:43:41.204 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:43:41.205 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 58 M 2024-03-24T11:43:41.205 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:43:42.836 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-radosgw-19.0.0-1667.gdb0330b1.el9. 8.1 MB/s | 9.8 MB 00:01 2024-03-24T11:43:42.903 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.0.0-1667.gdb0330b1.el9. 378 kB/s | 25 kB 00:00 2024-03-24T11:43:43.036 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): libcephfs2-19.0.0-1667.gdb0330b1.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-24T11:43:43.174 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-24T11:43:43.220 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): ceph-common-19.0.0-1667.gdb0330b1.el9.x 13 MB/s | 20 MB 00:01 2024-03-24T11:43:43.254 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): libradosstriper1-19.0.0-1667.gdb0330b1. 2.2 MB/s | 498 kB 00:00 2024-03-24T11:43:43.320 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): python3-ceph-argparse-19.0.0-1667.gdb03 679 kB/s | 45 kB 00:00 2024-03-24T11:43:43.396 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-common-19.0.0-1667.gdb0330 1.7 MB/s | 131 kB 00:00 2024-03-24T11:43:43.463 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-cephfs-19.0.0-1667.gdb0330b1.el 2.4 MB/s | 161 kB 00:00 2024-03-24T11:43:43.530 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): librgw2-19.0.0-1667.gdb0330b1.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-24T11:43:43.555 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-rados-19.0.0-1667.gdb0330b1.el 3.4 MB/s | 320 kB 00:00 2024-03-24T11:43:43.580 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): python3-rbd-19.0.0-1667.gdb0330b1.el9. 5.8 MB/s | 298 kB 00:00 2024-03-24T11:43:43.605 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): python3-rgw-19.0.0-1667.gdb0330b1.el9. 1.9 MB/s | 100 kB 00:00 2024-03-24T11:43:43.731 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): ceph-base-19.0.0-1667.gdb0330b1.el9.x8 2.5 MB/s | 5.3 MB 00:02 2024-03-24T11:43:43.764 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-03-24T11:43:43.790 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 44 kB 00:00 2024-03-24T11:43:43.823 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 833 kB/s | 159 kB 00:00 2024-03-24T11:43:43.899 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-24T11:43:43.924 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-24T11:43:43.949 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 327 kB/s | 41 kB 00:00 2024-03-24T11:43:44.041 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 381 kB/s | 35 kB 00:00 2024-03-24T11:43:44.066 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 381 kB/s | 44 kB 00:00 2024-03-24T11:43:44.092 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-24T11:43:44.259 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 228 kB/s | 26 kB 00:00 2024-03-24T11:43:44.317 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-24T11:43:44.384 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-03-24T11:43:44.451 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 508 kB/s | 67 kB 00:00 2024-03-24T11:43:44.602 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-24T11:43:44.627 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-24T11:43:44.793 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-03-24T11:43:44.960 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-03-24T11:43:45.060 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librbd1-19.0.0-1667.gdb0330b1.el9.x86_ 12 MB/s | 3.1 MB 00:00 2024-03-24T11:43:45.286 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): librados2-19.0.0-1667.gdb0330b1.el9.x8 5.3 MB/s | 3.4 MB 00:00 2024-03-24T11:43:45.289 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:43:45.289 INFO:teuthology.orchestra.run.smithi064.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-24T11:43:45.318 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:43:45.356 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:43:45.356 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:43:45.580 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:43:45.582 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:43:45.582 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 9.8 M 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.4 M 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options 2024-03-24T11:43:45.583 INFO:teuthology.orchestra.run.smithi193.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.3 M 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 20 M 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 25 k 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 742 k 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T11:43:45.584 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 498 k 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.1 M 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse 2024-03-24T11:43:45.585 INFO:teuthology.orchestra.run.smithi193.stdout: x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 45 k 2024-03-24T11:43:45.586 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 131 k 2024-03-24T11:43:45.586 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 161 k 2024-03-24T11:43:45.586 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T11:43:45.586 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 320 k 2024-03-24T11:43:45.586 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 298 k 2024-03-24T11:43:45.586 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 100 k 2024-03-24T11:43:45.586 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout:Install 30 Packages 2024-03-24T11:43:45.587 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 2 Packages 2024-03-24T11:43:45.588 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:43:45.588 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 58 M 2024-03-24T11:43:45.588 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:43:45.869 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:43:45.870 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:43:46.382 INFO:teuthology.orchestra.run.smithi193.stdout:(1/32): ceph-base-19.0.0-1667.gdb0330b1.el9.x86 7.5 MB/s | 5.3 MB 00:00 2024-03-24T11:43:46.432 INFO:teuthology.orchestra.run.smithi193.stdout:(2/32): ceph-selinux-19.0.0-1667.gdb0330b1.el9. 505 kB/s | 25 kB 00:00 2024-03-24T11:43:46.541 INFO:teuthology.orchestra.run.smithi193.stdout:(3/32): libcephfs2-19.0.0-1667.gdb0330b1.el9.x8 6.7 MB/s | 742 kB 00:00 2024-03-24T11:43:46.650 INFO:teuthology.orchestra.run.smithi193.stdout:(4/32): ceph-radosgw-19.0.0-1667.gdb0330b1.el9. 10 MB/s | 9.8 MB 00:00 2024-03-24T11:43:46.683 INFO:teuthology.orchestra.run.smithi193.stdout:(5/32): libradosstriper1-19.0.0-1667.gdb0330b1. 3.4 MB/s | 498 kB 00:00 2024-03-24T11:43:46.758 INFO:teuthology.orchestra.run.smithi193.stdout:(6/32): python3-ceph-argparse-19.0.0-1667.gdb03 602 kB/s | 45 kB 00:00 2024-03-24T11:43:46.817 INFO:teuthology.orchestra.run.smithi193.stdout:(7/32): python3-ceph-common-19.0.0-1667.gdb0330 2.2 MB/s | 131 kB 00:00 2024-03-24T11:43:46.884 INFO:teuthology.orchestra.run.smithi193.stdout:(8/32): python3-cephfs-19.0.0-1667.gdb0330b1.el 2.4 MB/s | 161 kB 00:00 2024-03-24T11:43:46.951 INFO:teuthology.orchestra.run.smithi193.stdout:(9/32): librgw2-19.0.0-1667.gdb0330b1.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-24T11:43:46.976 INFO:teuthology.orchestra.run.smithi193.stdout:(10/32): python3-rados-19.0.0-1667.gdb0330b1.el 3.4 MB/s | 320 kB 00:00 2024-03-24T11:43:47.001 INFO:teuthology.orchestra.run.smithi193.stdout:(11/32): python3-rbd-19.0.0-1667.gdb0330b1.el9. 5.8 MB/s | 298 kB 00:00 2024-03-24T11:43:47.027 INFO:teuthology.orchestra.run.smithi193.stdout:(12/32): python3-rgw-19.0.0-1667.gdb0330b1.el9. 1.9 MB/s | 100 kB 00:00 2024-03-24T11:43:47.144 INFO:teuthology.orchestra.run.smithi193.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-03-24T11:43:47.168 INFO:teuthology.orchestra.run.smithi193.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-24T11:43:47.194 INFO:teuthology.orchestra.run.smithi193.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-24T11:43:47.244 INFO:teuthology.orchestra.run.smithi193.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-24T11:43:47.278 INFO:teuthology.orchestra.run.smithi193.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-24T11:43:47.295 INFO:teuthology.orchestra.run.smithi193.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 823 kB/s | 41 kB 00:00 2024-03-24T11:43:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:43:47.503 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-24T11:43:47.532 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-24T11:43:47.558 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-24T11:43:47.588 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:43:47.604 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:43:47.629 INFO:teuthology.orchestra.run.smithi193.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 104 kB/s | 35 kB 00:00 2024-03-24T11:43:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:43:47.804 INFO:teuthology.orchestra.run.smithi193.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 412 kB/s | 216 kB 00:00 2024-03-24T11:43:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:43:47.929 INFO:teuthology.orchestra.run.smithi193.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 148 kB/s | 44 kB 00:00 2024-03-24T11:43:48.187 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-24T11:43:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:43:48.327 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:43:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 8/34 2024-03-24T11:43:48.438 INFO:teuthology.orchestra.run.smithi193.stdout:(22/32): ceph-common-19.0.0-1667.gdb0330b1.el9. 7.2 MB/s | 20 MB 00:02 2024-03-24T11:43:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:43:48.490 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:43:48.518 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-24T11:43:48.562 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 11/34 2024-03-24T11:43:48.632 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:43:48.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:43:48.655 INFO:teuthology.orchestra.run.smithi193.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 363 kB/s | 309 kB 00:00 2024-03-24T11:43:48.702 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:43:48.728 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-24T11:43:48.772 INFO:teuthology.orchestra.run.smithi193.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 419 kB/s | 49 kB 00:00 2024-03-24T11:43:48.842 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-24T11:43:48.855 INFO:teuthology.orchestra.run.smithi193.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 62 kB/s | 26 kB 00:00 2024-03-24T11:43:48.879 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-24T11:43:48.897 INFO:teuthology.orchestra.run.smithi193.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-03-24T11:43:48.919 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-24T11:43:48.953 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-24T11:43:48.982 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-24T11:43:49.014 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-24T11:43:49.048 INFO:teuthology.orchestra.run.smithi193.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-24T11:43:49.114 INFO:teuthology.orchestra.run.smithi193.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-03-24T11:43:49.343 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-24T11:43:49.348 INFO:teuthology.orchestra.run.smithi193.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.7 MB/s | 838 kB 00:00 2024-03-24T11:43:49.368 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:43:49.407 INFO:teuthology.orchestra.run.smithi193.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-03-24T11:43:49.443 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:43:49.465 INFO:teuthology.orchestra.run.smithi193.stdout:(31/32): librados2-19.0.0-1667.gdb0330b1.el9.x8 9.6 MB/s | 3.4 MB 00:00 2024-03-24T11:43:49.473 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 23/34 2024-03-24T11:43:49.514 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-24T11:43:49.707 INFO:teuthology.orchestra.run.smithi193.stdout:(32/32): librbd1-19.0.0-1667.gdb0330b1.el9.x86_ 8.7 MB/s | 3.1 MB 00:00 2024-03-24T11:43:49.716 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:43:49.716 INFO:teuthology.orchestra.run.smithi193.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-24T11:43:49.748 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:43:49.786 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:43:49.786 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:43:49.992 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-24T11:43:50.021 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:43:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:43:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 27/34 2024-03-24T11:43:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 28/34 2024-03-24T11:43:50.278 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:43:50.278 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:43:51.689 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:43:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-24T11:43:51.795 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-24T11:43:51.821 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-24T11:43:51.843 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:43:51.858 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:43:52.030 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:43:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:43:52.408 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-24T11:43:52.472 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:43:52.472 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:43:52.503 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:43:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:43:52.638 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 8/34 2024-03-24T11:43:52.665 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:43:52.711 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:43:52.740 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-24T11:43:52.792 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 11/34 2024-03-24T11:43:52.861 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:43:52.884 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:43:52.947 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:43:52.974 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-24T11:43:53.085 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-24T11:43:53.096 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:43:53.125 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-24T11:43:53.131 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:43:53.166 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-24T11:43:53.198 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-24T11:43:53.223 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:43:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T11:43:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T11:43:53.224 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:43:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-24T11:43:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-24T11:43:53.266 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:43:53.584 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-24T11:43:53.615 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:43:53.713 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:43:53.744 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 23/34 2024-03-24T11:43:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-24T11:43:54.235 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-24T11:43:54.259 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:43:54.330 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:43:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 27/34 2024-03-24T11:43:54.438 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 28/34 2024-03-24T11:43:56.607 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:43:56.649 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:43:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:43:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:43:57.357 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:43:57.357 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T11:43:57.358 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T11:43:57.358 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:43:57.423 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-03-24T11:44:03.521 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:04.459 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:44:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:44:04.484 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:04.485 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T11:44:04.485 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:44:04.485 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:44:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:44:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:44:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 6/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 8/34 2024-03-24T11:44:05.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 9/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 10/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 11/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 13/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-24T11:44:05.748 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-24T11:44:05.749 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-24T11:44:05.749 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-24T11:44:05.749 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-24T11:44:05.749 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-24T11:44:05.749 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-24T11:44:05.749 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-24T11:44:05.750 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-24T11:44:05.750 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-24T11:44:05.750 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:44:05.750 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-24T11:44:05.750 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/34 2024-03-24T11:44:06.692 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:44:06.692 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-03-24T11:44:06.692 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-03-24T11:44:06.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-03-24T11:44:06.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-03-24T11:44:06.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-03-24T11:44:06.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-03-24T11:44:06.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-03-24T11:44:06.693 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:06.860 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:44:06.860 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:06.861 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-03-24T11:44:06.861 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.861 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.861 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:44:06.861 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T11:44:06.861 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.862 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.862 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.862 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.862 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T11:44:06.862 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T11:44:06.863 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T11:44:06.863 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.863 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T11:44:06.863 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T11:44:06.863 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T11:44:06.863 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.863 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T11:44:06.864 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.864 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T11:44:06.864 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T11:44:06.864 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T11:44:06.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T11:44:06.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:06.866 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T11:44:06.866 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T11:44:06.866 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T11:44:06.866 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T11:44:06.866 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T11:44:06.866 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:06.866 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:44:07.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-03-24T11:44:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:44:07.668 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:44:07.668 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:07.668 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T11:44:07.668 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:44:07.669 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:44:07.669 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:07.670 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:44:07.747 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:44:07.748 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:44:07.767 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:29 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:44:07.921 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:44:07.922 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:07.922 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:44:07.922 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:07.922 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:44:07.922 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 47 M 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 163 k 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:07.923 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-24T11:44:07.924 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:07.924 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 48 M 2024-03-24T11:44:07.924 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 205 M 2024-03-24T11:44:07.924 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:44:08.175 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-24T11:44:08.367 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-24T11:44:08.392 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): libcephsqlite-19.0.0-1667.gdb0330b1.el9. 405 kB/s | 163 kB 00:00 2024-03-24T11:44:08.409 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-24T11:44:08.584 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-24T11:44:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:44:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/34 2024-03-24T11:44:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/34 2024-03-24T11:44:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 6/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 8/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 9/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 10/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 11/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 13/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-24T11:44:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-24T11:44:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-24T11:44:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:44:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-24T11:44:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/34 2024-03-24T11:44:09.762 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T11:44:09.763 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.764 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.765 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T11:44:09.765 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.765 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.765 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:09.765 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T11:44:09.765 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T11:44:09.765 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T11:44:09.765 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T11:44:09.766 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T11:44:09.766 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:09.766 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:10.084 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-test 2024-03-24T11:44:10.642 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:27 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:10.799 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:10.799 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:10.799 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:44:10.799 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:10.799 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 47 M 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 163 k 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-03-24T11:44:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:10.801 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 48 M 2024-03-24T11:44:10.801 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 205 M 2024-03-24T11:44:10.801 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:44:11.084 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-24T11:44:11.134 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-24T11:44:11.167 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-24T11:44:11.193 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): libcephsqlite-19.0.0-1667.gdb0330b1.el9. 619 kB/s | 163 kB 00:00 2024-03-24T11:44:11.618 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 416 kB/s | 187 kB 00:00 2024-03-24T11:44:14.626 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): ceph-test-19.0.0-1667.gdb0330b1.el9.x86_ 13 MB/s | 47 MB 00:03 2024-03-24T11:44:14.629 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:14.630 INFO:teuthology.orchestra.run.smithi193.stdout:Total 13 MB/s | 48 MB 00:03 2024-03-24T11:44:14.644 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:44:14.666 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:44:14.666 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:44:15.048 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:44:15.049 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:44:15.276 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): ceph-test-19.0.0-1667.gdb0330b1.el9.x86_ 6.5 MB/s | 47 MB 00:07 2024-03-24T11:44:15.278 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:15.278 INFO:teuthology.orchestra.run.smithi064.stdout:Total 6.6 MB/s | 48 MB 00:07 2024-03-24T11:44:15.291 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:44:15.312 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:44:15.312 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:44:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:44:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T11:44:15.701 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:44:15.702 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:44:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T11:44:15.755 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:44:15.793 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T11:44:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:44:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:44:16.419 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T11:44:16.459 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T11:44:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:44:16.543 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T11:44:16.570 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:44:20.290 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:44:20.364 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:44:21.238 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:44:21.313 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:44:22.767 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:44:22.767 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/6 2024-03-24T11:44:22.767 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/6 2024-03-24T11:44:22.767 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:44:22.767 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T11:44:22.767 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T11:44:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T11:44:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:23.219 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:44:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: jq-1.6-16.el9.x86_64 2024-03-24T11:44:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T11:44:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T11:44:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T11:44:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:23.220 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:23.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph 2024-03-24T11:44:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:44:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/6 2024-03-24T11:44:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/6 2024-03-24T11:44:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:44:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T11:44:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T11:44:23.952 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T11:44:23.952 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:23.952 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:44:23.952 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:23.952 INFO:teuthology.orchestra.run.smithi064.stdout: jq-1.6-16.el9.x86_64 2024-03-24T11:44:23.952 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:23.952 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T11:44:23.953 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T11:44:23.953 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T11:44:23.953 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:23.953 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:44:24.033 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:41 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:24.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-03-24T11:44:24.194 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:24.196 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 6.6 k 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 2.1 M 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 1.5 M 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 247 k 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 4.7 M 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 17 M 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T11:44:24.197 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T11:44:24.198 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T11:44:24.199 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:24.200 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:44:24.200 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-24T11:44:24.200 INFO:teuthology.orchestra.run.smithi193.stdout:Install 36 Packages 2024-03-24T11:44:24.200 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:24.200 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 30 M 2024-03-24T11:44:24.200 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 107 M 2024-03-24T11:44:24.200 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:44:24.743 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:46 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:44:24.901 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:44:24.904 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-24T11:44:24.904 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:44:24.904 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-24T11:44:24.904 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:44:24.904 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 6.6 k 2024-03-24T11:44:24.904 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:44:24.904 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 2.1 M 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 1.5 M 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 247 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 4.7 M 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 17 M 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T11:44:24.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T11:44:24.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout:Install 36 Packages 2024-03-24T11:44:24.907 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:24.908 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 30 M 2024-03-24T11:44:24.908 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 107 M 2024-03-24T11:44:24.909 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:44:25.048 INFO:teuthology.orchestra.run.smithi193.stdout:(1/36): ceph-19.0.0-1667.gdb0330b1.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-03-24T11:44:25.356 INFO:teuthology.orchestra.run.smithi193.stdout:(2/36): ceph-mgr-19.0.0-1667.gdb0330b1.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2024-03-24T11:44:25.398 INFO:teuthology.orchestra.run.smithi193.stdout:(3/36): ceph-mds-19.0.0-1667.gdb0330b1.el9.x86_ 3.7 MB/s | 2.1 MB 00:00 2024-03-24T11:44:25.455 INFO:teuthology.orchestra.run.smithi064.stdout:(1/36): ceph-19.0.0-1667.gdb0330b1.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-24T11:44:25.498 INFO:teuthology.orchestra.run.smithi193.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1667.gdb03 2.4 MB/s | 247 kB 00:00 2024-03-24T11:44:25.607 INFO:teuthology.orchestra.run.smithi193.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-24T11:44:25.649 INFO:teuthology.orchestra.run.smithi193.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-24T11:44:25.683 INFO:teuthology.orchestra.run.smithi193.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-24T11:44:25.716 INFO:teuthology.orchestra.run.smithi193.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-24T11:44:25.750 INFO:teuthology.orchestra.run.smithi193.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-24T11:44:25.792 INFO:teuthology.orchestra.run.smithi193.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-24T11:44:25.859 INFO:teuthology.orchestra.run.smithi193.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-24T11:44:25.893 INFO:teuthology.orchestra.run.smithi193.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-24T11:44:25.926 INFO:teuthology.orchestra.run.smithi193.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-24T11:44:25.968 INFO:teuthology.orchestra.run.smithi193.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-24T11:44:26.297 INFO:teuthology.orchestra.run.smithi064.stdout:(2/36): ceph-mds-19.0.0-1667.gdb0330b1.el9.x86_ 2.0 MB/s | 2.1 MB 00:01 2024-03-24T11:44:26.344 INFO:teuthology.orchestra.run.smithi193.stdout:(15/36): ceph-osd-19.0.0-1667.gdb0330b1.el9.x86 17 MB/s | 17 MB 00:00 2024-03-24T11:44:26.369 INFO:teuthology.orchestra.run.smithi193.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 312 kB/s | 125 kB 00:00 2024-03-24T11:44:26.444 INFO:teuthology.orchestra.run.smithi193.stdout:(17/36): ceph-mon-19.0.0-1667.gdb0330b1.el9.x86 3.4 MB/s | 4.7 MB 00:01 2024-03-24T11:44:26.495 INFO:teuthology.orchestra.run.smithi193.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2024-03-24T11:44:26.545 INFO:teuthology.orchestra.run.smithi193.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 434 kB/s | 43 kB 00:00 2024-03-24T11:44:26.564 INFO:teuthology.orchestra.run.smithi064.stdout:(3/36): ceph-mgr-19.0.0-1667.gdb0330b1.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-03-24T11:44:26.570 INFO:teuthology.orchestra.run.smithi193.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 752 kB/s | 151 kB 00:00 2024-03-24T11:44:26.612 INFO:teuthology.orchestra.run.smithi193.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 258 kB/s | 11 kB 00:00 2024-03-24T11:44:26.629 INFO:teuthology.orchestra.run.smithi193.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-24T11:44:26.654 INFO:teuthology.orchestra.run.smithi193.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-03-24T11:44:26.672 INFO:teuthology.orchestra.run.smithi064.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1667.gdb03 2.2 MB/s | 247 kB 00:00 2024-03-24T11:44:26.679 INFO:teuthology.orchestra.run.smithi193.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-24T11:44:26.705 INFO:teuthology.orchestra.run.smithi193.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 306 kB/s | 23 kB 00:00 2024-03-24T11:44:26.730 INFO:teuthology.orchestra.run.smithi193.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-24T11:44:26.755 INFO:teuthology.orchestra.run.smithi193.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-24T11:44:26.781 INFO:teuthology.orchestra.run.smithi193.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-03-24T11:44:26.797 INFO:teuthology.orchestra.run.smithi064.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-24T11:44:26.806 INFO:teuthology.orchestra.run.smithi193.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-24T11:44:26.831 INFO:teuthology.orchestra.run.smithi193.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-24T11:44:26.841 INFO:teuthology.orchestra.run.smithi064.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-24T11:44:26.856 INFO:teuthology.orchestra.run.smithi193.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-03-24T11:44:26.873 INFO:teuthology.orchestra.run.smithi064.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-24T11:44:26.881 INFO:teuthology.orchestra.run.smithi193.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-24T11:44:26.907 INFO:teuthology.orchestra.run.smithi064.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-24T11:44:26.907 INFO:teuthology.orchestra.run.smithi193.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-03-24T11:44:26.932 INFO:teuthology.orchestra.run.smithi193.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-24T11:44:26.940 INFO:teuthology.orchestra.run.smithi064.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-24T11:44:26.957 INFO:teuthology.orchestra.run.smithi193.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-24T11:44:26.982 INFO:teuthology.orchestra.run.smithi064.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-24T11:44:27.008 INFO:teuthology.orchestra.run.smithi193.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-24T11:44:27.010 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:27.010 INFO:teuthology.orchestra.run.smithi193.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-24T11:44:27.035 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:44:27.049 INFO:teuthology.orchestra.run.smithi064.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-24T11:44:27.077 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:44:27.077 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:44:27.083 INFO:teuthology.orchestra.run.smithi064.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-24T11:44:27.117 INFO:teuthology.orchestra.run.smithi064.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-24T11:44:27.150 INFO:teuthology.orchestra.run.smithi064.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-24T11:44:27.317 INFO:teuthology.orchestra.run.smithi064.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 750 kB/s | 125 kB 00:00 2024-03-24T11:44:27.384 INFO:teuthology.orchestra.run.smithi064.stdout:(16/36): ceph-mon-19.0.0-1667.gdb0330b1.el9.x86 2.4 MB/s | 4.7 MB 00:01 2024-03-24T11:44:27.498 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:44:27.499 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:44:27.551 INFO:teuthology.orchestra.run.smithi064.stdout:(17/36): ceph-osd-19.0.0-1667.gdb0330b1.el9.x86 13 MB/s | 17 MB 00:01 2024-03-24T11:44:27.635 INFO:teuthology.orchestra.run.smithi064.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 70 kB/s | 22 kB 00:00 2024-03-24T11:44:27.693 INFO:teuthology.orchestra.run.smithi064.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 307 kB/s | 43 kB 00:00 2024-03-24T11:44:27.769 INFO:teuthology.orchestra.run.smithi064.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-24T11:44:27.827 INFO:teuthology.orchestra.run.smithi064.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 182 kB/s | 11 kB 00:00 2024-03-24T11:44:27.853 INFO:teuthology.orchestra.run.smithi064.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-24T11:44:27.878 INFO:teuthology.orchestra.run.smithi064.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 707 kB/s | 172 kB 00:00 2024-03-24T11:44:27.903 INFO:teuthology.orchestra.run.smithi064.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-24T11:44:27.929 INFO:teuthology.orchestra.run.smithi064.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-24T11:44:27.954 INFO:teuthology.orchestra.run.smithi064.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-24T11:44:27.979 INFO:teuthology.orchestra.run.smithi064.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-24T11:44:28.004 INFO:teuthology.orchestra.run.smithi064.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-03-24T11:44:28.030 INFO:teuthology.orchestra.run.smithi064.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-24T11:44:28.063 INFO:teuthology.orchestra.run.smithi064.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-24T11:44:28.088 INFO:teuthology.orchestra.run.smithi064.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-24T11:44:28.114 INFO:teuthology.orchestra.run.smithi064.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-24T11:44:28.139 INFO:teuthology.orchestra.run.smithi064.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-03-24T11:44:28.164 INFO:teuthology.orchestra.run.smithi064.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-24T11:44:28.198 INFO:teuthology.orchestra.run.smithi064.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-24T11:44:28.256 INFO:teuthology.orchestra.run.smithi064.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-24T11:44:28.258 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:28.259 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.9 MB/s | 30 MB 00:03 2024-03-24T11:44:28.283 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:44:28.329 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:44:28.329 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:44:28.750 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:44:28.750 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:44:28.821 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:44:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-24T11:44:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-24T11:44:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-24T11:44:29.024 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-24T11:44:29.070 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-24T11:44:29.103 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-24T11:44:29.194 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-24T11:44:29.272 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-24T11:44:29.314 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-24T11:44:29.385 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-24T11:44:29.448 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-24T11:44:29.487 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-24T11:44:29.535 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-24T11:44:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-24T11:44:29.783 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-24T11:44:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-24T11:44:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-24T11:44:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-24T11:44:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-24T11:44:30.003 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-24T11:44:30.038 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:44:30.072 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-24T11:44:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:44:30.112 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-24T11:44:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-24T11:44:30.147 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-24T11:44:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-24T11:44:30.182 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-24T11:44:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-24T11:44:30.270 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-24T11:44:30.285 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-24T11:44:30.304 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-24T11:44:30.327 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-24T11:44:30.360 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-24T11:44:30.376 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-24T11:44:30.454 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-24T11:44:30.520 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-24T11:44:30.532 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-24T11:44:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-24T11:44:30.646 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:44:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-24T11:44:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-24T11:44:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-24T11:44:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-24T11:44:30.813 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 31/36 2024-03-24T11:44:30.843 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:44:30.867 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-24T11:44:30.869 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:44:30.869 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:30.869 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T11:44:30.869 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:44:30.869 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:44:30.869 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-24T11:44:31.106 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-24T11:44:31.146 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-24T11:44:31.180 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-24T11:44:31.209 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-24T11:44:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-24T11:44:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:44:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-24T11:44:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-24T11:44:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-24T11:44:31.471 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-24T11:44:31.563 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-24T11:44:31.611 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-24T11:44:31.684 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-24T11:44:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-24T11:44:31.966 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:44:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 31/36 2024-03-24T11:44:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:44:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:44:32.191 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:32.191 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T11:44:32.191 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:44:32.191 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:44:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:32.302 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:44:32.328 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:44:32.329 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:32.329 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T11:44:32.329 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:44:32.329 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:44:32.329 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:32.778 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:44:32.801 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:44:32.801 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:32.801 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T11:44:32.802 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:44:32.802 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:44:32.802 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:33.004 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:44:33.027 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:44:33.027 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:33.027 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T11:44:33.027 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:44:33.027 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:44:33.027 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:33.084 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:44:33.716 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:44:33.742 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:44:33.742 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:33.742 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T11:44:33.742 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:44:33.743 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:44:33.743 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:34.208 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:44:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:44:34.233 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:34.233 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T11:44:34.233 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:44:34.233 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:44:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:44:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:44:34.473 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:34.473 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T11:44:34.473 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:44:34.473 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:44:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 6/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-24T11:44:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-24T11:44:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-24T11:44:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-24T11:44:35.711 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-24T11:44:35.711 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:35.711 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:44:35.711 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:35.711 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:35.711 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T11:44:35.712 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T11:44:35.713 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T11:44:35.714 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T11:44:35.714 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T11:44:35.714 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T11:44:35.714 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:35.714 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:36.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base 2024-03-24T11:44:36.324 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:44:36.324 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/36 2024-03-24T11:44:36.324 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 6/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-24T11:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-24T11:44:36.326 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-24T11:44:36.327 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-24T11:44:36.327 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-24T11:44:36.327 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-24T11:44:36.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-24T11:44:36.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:44:36.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-24T11:44:36.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-24T11:44:36.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-24T11:44:36.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-24T11:44:36.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-24T11:44:36.619 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:53 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:36.714 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:44:36.765 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:36.766 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:44:36.766 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:36.838 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install cephadm 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:37.235 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T11:44:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T11:44:37.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T11:44:37.238 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T11:44:37.238 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T11:44:37.238 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T11:44:37.238 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T11:44:37.238 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:37.238 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:44:37.398 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:54 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:37.548 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:37.549 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:37.549 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:44:37.549 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:37.549 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:44:37.549 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 748 k 2024-03-24T11:44:37.549 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:37.550 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:44:37.550 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:37.550 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-24T11:44:37.550 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:37.550 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 748 k 2024-03-24T11:44:37.550 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 754 k 2024-03-24T11:44:37.550 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:44:37.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-03-24T11:44:37.802 INFO:teuthology.orchestra.run.smithi193.stdout:cephadm-19.0.0-1667.gdb0330b1.el9.noarch.rpm 2.9 MB/s | 748 kB 00:00 2024-03-24T11:44:37.802 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:37.802 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.9 MB/s | 748 kB 00:00 2024-03-24T11:44:37.803 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:44:37.808 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:44:37.808 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:44:37.813 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:44:37.813 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:44:37.887 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:44:38.147 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:00 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:44:38.246 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:44:38.284 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:44:38.301 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:44:38.302 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:44:38.302 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:44:38.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-03-24T11:44:38.372 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:44:38.737 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:44:38.938 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:00 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:44:39.024 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:44:39.024 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:39.024 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:44:39.024 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:39.024 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:39.024 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:39.092 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 748 k 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:39.093 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-24T11:44:39.094 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:39.094 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 748 k 2024-03-24T11:44:39.094 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 754 k 2024-03-24T11:44:39.094 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:44:39.232 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T11:44:39.534 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.0.0-1667.gdb0330b1.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-24T11:44:39.535 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:39.535 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-24T11:44:39.535 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:44:39.541 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:44:39.541 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:44:39.547 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:44:39.547 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:44:39.626 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:44:39.793 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:56 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:39.944 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 145 k 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 145 k 2024-03-24T11:44:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 436 k 2024-03-24T11:44:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:44:39.984 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:44:40.074 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:44:40.167 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-immutable-object-cache-19.0.0-1667.gdb0330 656 kB/s | 145 kB 00:00 2024-03-24T11:44:40.168 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:40.168 INFO:teuthology.orchestra.run.smithi193.stdout:Total 652 kB/s | 145 kB 00:00 2024-03-24T11:44:40.168 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:44:40.175 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:44:40.175 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:44:40.226 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:44:40.227 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:44:40.367 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:44:40.435 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:44:40.458 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:44:40.458 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:40.458 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T11:44:40.458 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:44:40.730 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:44:40.731 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:40.731 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:44:40.731 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:40.731 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:40.731 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:44:40.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T11:44:41.434 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:44:41.434 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:41.434 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:44:41.434 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:41.435 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:41.435 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:41.505 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:03 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:44:41.661 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 145 k 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 436 k 2024-03-24T11:44:41.662 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:44:41.672 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr 2024-03-24T11:44:42.016 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.0.0-1667.gdb0330 412 kB/s | 145 kB 00:00 2024-03-24T11:44:42.016 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:42.017 INFO:teuthology.orchestra.run.smithi064.stdout:Total 410 kB/s | 145 kB 00:00 2024-03-24T11:44:42.017 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:44:42.025 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:44:42.025 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:44:42.080 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:44:42.080 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:44:42.228 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:44:42.245 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:59 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:42.291 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:44:42.316 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:44:42.316 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:44:42.316 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T11:44:42.316 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:42.342 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:44:42.394 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:42.395 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:44:42.396 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:42.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T11:44:43.026 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:00 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:43.175 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 1.8 M 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 31 k 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 15 k 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T11:44:43.177 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout:Install 12 Packages 2024-03-24T11:44:43.178 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:43.179 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 6.9 M 2024-03-24T11:44:43.179 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 26 M 2024-03-24T11:44:43.179 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:44:43.250 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:44:43.250 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:43.250 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:44:43.251 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:44:43.251 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:43.251 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:44:43.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-03-24T11:44:43.572 INFO:teuthology.orchestra.run.smithi193.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1667.gdb 156 kB/s | 31 kB 00:00 2024-03-24T11:44:43.672 INFO:teuthology.orchestra.run.smithi193.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1667.gdb0 51 kB/s | 15 kB 00:00 2024-03-24T11:44:43.740 INFO:teuthology.orchestra.run.smithi193.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-24T11:44:43.765 INFO:teuthology.orchestra.run.smithi193.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-24T11:44:43.856 INFO:teuthology.orchestra.run.smithi193.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1667.gdb0330b 3.7 MB/s | 1.8 MB 00:00 2024-03-24T11:44:43.899 INFO:teuthology.orchestra.run.smithi193.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-03-24T11:44:43.966 INFO:teuthology.orchestra.run.smithi193.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.7 MB/s | 548 kB 00:00 2024-03-24T11:44:43.991 INFO:teuthology.orchestra.run.smithi193.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 145 kB/s | 19 kB 00:00 2024-03-24T11:44:44.016 INFO:teuthology.orchestra.run.smithi193.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-24T11:44:44.042 INFO:teuthology.orchestra.run.smithi193.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 609 kB/s | 31 kB 00:00 2024-03-24T11:44:44.067 INFO:teuthology.orchestra.run.smithi193.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.6 MB/s | 188 kB 00:00 2024-03-24T11:44:44.070 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:06 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:44:44.169 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:44:44.223 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:44:44.225 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:44:44.225 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:44:44.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T11:44:44.409 INFO:teuthology.orchestra.run.smithi193.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.0 MB/s | 2.0 MB 00:00 2024-03-24T11:44:44.412 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:44.412 INFO:teuthology.orchestra.run.smithi193.stdout:Total 5.6 MB/s | 6.9 MB 00:01 2024-03-24T11:44:44.431 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:44:44.448 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:44:44.448 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:44:44.610 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:44:44.610 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:44:44.853 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:06 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:44:45.011 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 1.8 M 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 31 k 2024-03-24T11:44:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 15 k 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-24T11:44:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:45.014 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.9 M 2024-03-24T11:44:45.014 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-24T11:44:45.014 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:44:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:44:45.210 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-24T11:44:45.253 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-24T11:44:45.299 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-24T11:44:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-24T11:44:45.426 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-24T11:44:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-24T11:44:45.641 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-24T11:44:45.660 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1667.gdb 158 kB/s | 31 kB 00:00 2024-03-24T11:44:45.685 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-24T11:44:45.686 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1667.gdb0 68 kB/s | 15 kB 00:00 2024-03-24T11:44:45.713 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:44:45.779 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 10/12 2024-03-24T11:44:45.819 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-24T11:44:45.861 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-24T11:44:45.903 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1667.gdb0330b 4.0 MB/s | 1.8 MB 00:00 2024-03-24T11:44:45.954 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-03-24T11:44:46.022 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 11/12 2024-03-24T11:44:46.046 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:44:46.079 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-03-24T11:44:46.105 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.2 MB/s | 548 kB 00:00 2024-03-24T11:44:46.130 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-24T11:44:46.155 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 613 kB/s | 31 kB 00:00 2024-03-24T11:44:46.180 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-24T11:44:46.664 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-03-24T11:44:46.667 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:46.668 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.1 MB/s | 6.9 MB 00:01 2024-03-24T11:44:46.686 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:44:46.704 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:44:46.705 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:44:46.877 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:44:46.878 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:44:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 1/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 2/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 3/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-24T11:44:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-24T11:44:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-24T11:44:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:47.211 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:44:47.475 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T11:44:47.482 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-24T11:44:47.517 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-24T11:44:47.565 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-24T11:44:47.593 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-24T11:44:47.686 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-24T11:44:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-24T11:44:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-24T11:44:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-24T11:44:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:44:48.032 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:05 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 10/12 2024-03-24T11:44:48.182 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:48.182 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-24T11:44:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 7.4 M 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T11:44:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T11:44:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T11:44:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T11:44:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:48.184 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:44:48.184 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-24T11:44:48.184 INFO:teuthology.orchestra.run.smithi193.stdout:Install 12 Packages 2024-03-24T11:44:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:48.184 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 37 M 2024-03-24T11:44:48.185 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 209 M 2024-03-24T11:44:48.185 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:44:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 11/12 2024-03-24T11:44:48.325 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:44:48.517 INFO:teuthology.orchestra.run.smithi193.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 250 kB/s | 32 kB 00:00 2024-03-24T11:44:48.550 INFO:teuthology.orchestra.run.smithi193.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2024-03-24T11:44:48.617 INFO:teuthology.orchestra.run.smithi193.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-24T11:44:48.642 INFO:teuthology.orchestra.run.smithi193.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 381 kB/s | 34 kB 00:00 2024-03-24T11:44:48.801 INFO:teuthology.orchestra.run.smithi193.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-24T11:44:48.860 INFO:teuthology.orchestra.run.smithi193.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-24T11:44:48.885 INFO:teuthology.orchestra.run.smithi193.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-24T11:44:49.156 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 1/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 2/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 3/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-24T11:44:49.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-24T11:44:49.177 INFO:teuthology.orchestra.run.smithi193.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-03-24T11:44:49.236 INFO:teuthology.orchestra.run.smithi193.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-24T11:44:49.386 INFO:teuthology.orchestra.run.smithi193.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 937 kB/s | 195 kB 00:00 2024-03-24T11:44:49.461 INFO:teuthology.orchestra.run.smithi193.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-24T11:44:49.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T11:44:49.506 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T11:44:49.507 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T11:44:49.507 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:49.507 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:44:49.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T11:44:50.153 INFO:teuthology.orchestra.run.smithi193.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.2 MB/s | 7.4 MB 00:01 2024-03-24T11:44:50.156 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:50.156 INFO:teuthology.orchestra.run.smithi193.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-24T11:44:50.198 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:44:50.223 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:44:50.223 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:44:50.324 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:12 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:44:50.481 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 7.4 M 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T11:44:50.482 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-24T11:44:50.483 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:44:50.484 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 37 M 2024-03-24T11:44:50.484 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 209 M 2024-03-24T11:44:50.484 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:44:50.578 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:44:50.578 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:44:50.887 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-24T11:44:50.921 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2024-03-24T11:44:50.979 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 591 kB/s | 34 kB 00:00 2024-03-24T11:44:51.080 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.9 MB/s | 3.0 MB 00:00 2024-03-24T11:44:51.274 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:44:51.351 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T11:44:51.382 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T11:44:51.397 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-16 11 MB/s | 7.4 MB 00:00 2024-03-24T11:44:51.474 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:44:51.510 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T11:44:51.572 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 217 kB 00:00 2024-03-24T11:44:51.664 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 6.9 MB/s | 4.7 MB 00:00 2024-03-24T11:44:51.687 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:44:51.717 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:44:51.731 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.3 MB/s | 4.8 MB 00:00 2024-03-24T11:44:51.801 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T11:44:52.039 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-24T11:44:52.065 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 590 kB/s | 195 kB 00:00 2024-03-24T11:44:52.240 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-24T11:44:52.265 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-24T11:44:52.267 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:52.268 INFO:teuthology.orchestra.run.smithi064.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-24T11:44:52.308 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:44:52.333 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:44:52.333 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:44:52.453 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T11:44:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T11:44:52.707 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:44:52.707 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:44:53.434 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:44:53.497 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T11:44:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T11:44:53.627 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:44:53.673 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T11:44:53.807 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T11:44:53.857 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:44:53.888 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:44:53.976 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T11:44:54.663 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T11:44:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T11:44:54.843 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T11:44:54.877 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:44:56.100 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T11:44:57.165 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T11:44:57.204 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 1/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T11:44:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T11:44:57.901 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T11:44:57.901 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:57.901 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:44:57.901 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T11:44:57.902 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T11:44:57.903 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:57.903 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:44:58.310 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-rook 2024-03-24T11:44:58.882 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:15 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:44:59.034 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:44:59.035 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-24T11:44:59.035 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:44:59.035 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 49 k 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T11:44:59.036 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T11:44:59.037 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T11:44:59.037 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T11:44:59.037 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T11:44:59.037 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:59.037 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:44:59.037 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-24T11:44:59.037 INFO:teuthology.orchestra.run.smithi193.stdout:Install 13 Packages 2024-03-24T11:44:59.037 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:44:59.038 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 2.0 M 2024-03-24T11:44:59.038 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 26 M 2024-03-24T11:44:59.038 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:44:59.513 INFO:teuthology.orchestra.run.smithi193.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 204 kB/s | 24 kB 00:00 2024-03-24T11:44:59.538 INFO:teuthology.orchestra.run.smithi193.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 125 kB/s | 18 kB 00:00 2024-03-24T11:44:59.589 INFO:teuthology.orchestra.run.smithi193.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-24T11:44:59.614 INFO:teuthology.orchestra.run.smithi193.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T11:44:59.639 INFO:teuthology.orchestra.run.smithi193.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-24T11:44:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:44:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 1/12 2024-03-24T11:44:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T11:44:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T11:44:59.665 INFO:teuthology.orchestra.run.smithi193.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-24T11:44:59.690 INFO:teuthology.orchestra.run.smithi193.stdout:(7/13): ceph-mgr-rook-19.0.0-1667.gdb0330b1.el9 166 kB/s | 49 kB 00:00 2024-03-24T11:44:59.782 INFO:teuthology.orchestra.run.smithi193.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-24T11:44:59.807 INFO:teuthology.orchestra.run.smithi193.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-24T11:44:59.924 INFO:teuthology.orchestra.run.smithi193.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-03-24T11:44:59.958 INFO:teuthology.orchestra.run.smithi193.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 757 kB/s | 203 kB 00:00 2024-03-24T11:45:00.000 INFO:teuthology.orchestra.run.smithi193.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-24T11:45:00.075 INFO:teuthology.orchestra.run.smithi193.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2024-03-24T11:45:00.077 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:00.077 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-24T11:45:00.078 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:00.091 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:00.091 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T11:45:00.092 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T11:45:00.093 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T11:45:00.093 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T11:45:00.093 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T11:45:00.093 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T11:45:00.093 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T11:45:00.093 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T11:45:00.093 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:00.093 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:00.183 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:00.183 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:00.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-03-24T11:45:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:00.898 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T11:45:00.951 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T11:45:00.996 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:22 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:01.006 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T11:45:01.044 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T11:45:01.086 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T11:45:01.152 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 49 k 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T11:45:01.154 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-03-24T11:45:01.155 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.0 M 2024-03-24T11:45:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-24T11:45:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:01.179 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T11:45:01.297 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T11:45:01.345 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T11:45:01.414 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 164 kB/s | 18 kB 00:00 2024-03-24T11:45:01.431 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 192 kB/s | 24 kB 00:00 2024-03-24T11:45:01.481 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-24T11:45:01.498 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-24T11:45:01.523 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-24T11:45:01.540 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-24T11:45:01.649 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): ceph-mgr-rook-19.0.0-1667.gdb0330b1.el9 143 kB/s | 49 kB 00:00 2024-03-24T11:45:01.782 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-03-24T11:45:01.799 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-24T11:45:01.950 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-03-24T11:45:02.025 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 539 kB/s | 203 kB 00:00 2024-03-24T11:45:02.067 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 769 kB/s | 90 kB 00:00 2024-03-24T11:45:02.070 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T11:45:02.100 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.2 MB/s | 1.0 MB 00:00 2024-03-24T11:45:02.105 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:02.105 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-24T11:45:02.105 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:02.118 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T11:45:02.119 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:02.119 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:02.152 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T11:45:02.208 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T11:45:02.218 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:02.219 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:02.230 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:45:02.734 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T11:45:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T11:45:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 1/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T11:45:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T11:45:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T11:45:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T11:45:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T11:45:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T11:45:03.120 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T11:45:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T11:45:03.335 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T11:45:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T11:45:03.391 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T11:45:03.392 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T11:45:03.392 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T11:45:03.392 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T11:45:03.392 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T11:45:03.392 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T11:45:03.392 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:03.392 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:03.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T11:45:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T11:45:04.204 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:21 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T11:45:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T11:45:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T11:45:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:45:04.354 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:04.355 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-24T11:45:04.355 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:45:04.355 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-24T11:45:04.355 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:45:04.355 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 138 k 2024-03-24T11:45:04.355 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:45:04.355 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-03-24T11:45:04.356 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:04.357 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 6.8 M 2024-03-24T11:45:04.357 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 33 M 2024-03-24T11:45:04.357 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:45:04.762 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-24T11:45:04.854 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-24T11:45:04.971 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1667.gdb0330b1.e 369 kB/s | 138 kB 00:00 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 1/13 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T11:45:05.122 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T11:45:05.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T11:45:05.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T11:45:05.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T11:45:05.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T11:45:05.354 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 121 kB/s | 58 kB 00:00 2024-03-24T11:45:05.405 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 125 kB/s | 51 kB 00:00 2024-03-24T11:45:05.471 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 775 kB/s | 548 kB 00:00 2024-03-24T11:45:05.474 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:05.474 INFO:teuthology.orchestra.run.smithi193.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T11:45:05.480 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T11:45:05.481 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T11:45:05.481 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T11:45:05.481 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T11:45:05.481 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T11:45:05.481 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T11:45:05.481 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T11:45:05.481 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:05.481 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:05.487 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:05.498 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:05.498 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:05.589 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:05.591 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:05.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T11:45:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:06.072 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T11:45:06.123 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T11:45:06.292 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:28 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:06.448 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:06.448 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 138 k 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:06.449 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:06.450 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-24T11:45:06.450 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-24T11:45:06.450 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:06.451 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.8 M 2024-03-24T11:45:06.451 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-03-24T11:45:06.451 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:06.648 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T11:45:06.735 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T11:45:06.797 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T11:45:06.828 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:45:07.110 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 5.8 MB 00:00 2024-03-24T11:45:07.210 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1667.gdb0330b1.e 400 kB/s | 138 kB 00:00 2024-03-24T11:45:07.386 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 476 kB/s | 248 kB 00:00 2024-03-24T11:45:07.478 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 216 kB/s | 58 kB 00:00 2024-03-24T11:45:07.553 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-24T11:45:07.645 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 197 kB/s | 51 kB 00:00 2024-03-24T11:45:07.648 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:07.648 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-24T11:45:07.661 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:07.672 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:07.672 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:45:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/6 2024-03-24T11:45:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T11:45:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T11:45:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T11:45:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T11:45:07.765 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:07.766 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:08.015 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T11:45:08.015 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:08.015 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:08.015 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:45:08.016 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T11:45:08.016 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T11:45:08.016 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T11:45:08.016 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T11:45:08.016 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T11:45:08.016 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:08.016 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T11:45:08.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-fuse 2024-03-24T11:45:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T11:45:08.813 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:25 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:08.831 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T11:45:08.919 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T11:45:08.964 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 886 k 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 964 k 2024-03-24T11:45:08.965 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.8 M 2024-03-24T11:45:08.966 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:45:08.987 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T11:45:09.017 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:45:09.107 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 557 kB/s | 79 kB 00:00 2024-03-24T11:45:09.434 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-fuse-19.0.0-1667.gdb0330b1.el9.x86_ 1.8 MB/s | 886 kB 00:00 2024-03-24T11:45:09.435 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:09.436 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-03-24T11:45:09.436 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:09.456 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:09.456 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:09.530 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:09.531 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:09.725 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:09.791 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:45:09.791 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/6 2024-03-24T11:45:09.791 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T11:45:09.791 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T11:45:09.791 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T11:45:09.791 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T11:45:09.868 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T11:45:09.942 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T11:45:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T11:45:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:10.089 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:10.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-03-24T11:45:10.648 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:45:10.648 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/2 2024-03-24T11:45:10.893 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:10.938 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T11:45:10.938 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:10.938 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:10.938 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T11:45:10.938 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:10.938 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:11.048 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:11.048 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 886 k 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-24T11:45:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:11.050 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 964 k 2024-03-24T11:45:11.050 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.8 M 2024-03-24T11:45:11.050 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:11.172 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 648 kB/s | 79 kB 00:00 2024-03-24T11:45:11.240 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-volume 2024-03-24T11:45:11.380 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.0.0-1667.gdb0330b1.el9.x86_ 2.6 MB/s | 886 kB 00:00 2024-03-24T11:45:11.381 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:11.382 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.8 MB/s | 964 kB 00:00 2024-03-24T11:45:11.382 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:11.402 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:11.403 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:11.478 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:11.478 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:11.802 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T11:45:11.816 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:28 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:45:11.967 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:11.967 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:11.967 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 264 k 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-03-24T11:45:11.968 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:11.969 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 332 k 2024-03-24T11:45:11.969 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.5 M 2024-03-24T11:45:11.969 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:45:12.090 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 569 kB/s | 69 kB 00:00 2024-03-24T11:45:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:45:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/2 2024-03-24T11:45:12.765 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-volume-19.0.0-1667.gdb0330b1.el9.no 331 kB/s | 264 kB 00:00 2024-03-24T11:45:12.766 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:12.767 INFO:teuthology.orchestra.run.smithi193.stdout:Total 416 kB/s | 332 kB 00:00 2024-03-24T11:45:12.767 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:12.776 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:12.777 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T11:45:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:12.786 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T11:45:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:12.786 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:12.816 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:12.816 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:13.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-03-24T11:45:13.150 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-24T11:45:13.193 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:45:13.209 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:45:13.209 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:45:13.209 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T11:45:13.209 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:13.633 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:13.789 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:13.790 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 264 k 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 332 k 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.5 M 2024-03-24T11:45:13.791 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:13.926 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 511 kB/s | 69 kB 00:00 2024-03-24T11:45:13.947 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 1/2 2024-03-24T11:45:14.076 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-volume-19.0.0-1667.gdb0330b1.el9.no 926 kB/s | 264 kB 00:00 2024-03-24T11:45:14.077 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:14.078 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-24T11:45:14.078 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:14.087 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:14.087 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:14.126 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:14.127 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:14.291 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:14.302 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-24T11:45:14.303 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:14.303 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:14.303 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:45:14.303 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T11:45:14.303 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:14.303 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-24T11:45:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:45:14.511 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:45:14.511 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:45:14.511 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T11:45:14.511 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:14.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados-devel 2024-03-24T11:45:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 1/2 2024-03-24T11:45:15.218 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:15.368 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:15.369 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:15.369 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-24T11:45:15.369 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:15.369 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:45:15.369 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 127 k 2024-03-24T11:45:15.369 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:15.369 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:45:15.370 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:15.370 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-24T11:45:15.370 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:15.370 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 127 k 2024-03-24T11:45:15.370 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 456 k 2024-03-24T11:45:15.370 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:45:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-24T11:45:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:15.544 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:45:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T11:45:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:15.544 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:15.680 INFO:teuthology.orchestra.run.smithi193.stdout:librados-devel-19.0.0-1667.gdb0330b1.el9.x86_64 409 kB/s | 127 kB 00:00 2024-03-24T11:45:15.680 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:15.680 INFO:teuthology.orchestra.run.smithi193.stdout:Total 407 kB/s | 127 kB 00:00 2024-03-24T11:45:15.681 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:15.686 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:15.687 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:15.741 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:15.742 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:15.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-03-24T11:45:15.941 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:16.037 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:16.426 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:16.500 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:16.583 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:16.584 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:16.584 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-24T11:45:16.584 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:16.584 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 127 k 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-03-24T11:45:16.585 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:16.752 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:16.752 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:16.752 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:16.752 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:16.752 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:16.752 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:16.815 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.0.0-1667.gdb0330b1.el9.x86_64 552 kB/s | 127 kB 00:00 2024-03-24T11:45:16.816 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:16.816 INFO:teuthology.orchestra.run.smithi064.stdout:Total 548 kB/s | 127 kB 00:00 2024-03-24T11:45:16.816 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:16.822 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:16.822 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:16.879 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:16.880 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:16.956 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs2 2024-03-24T11:45:17.087 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:17.194 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:17.529 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:17.628 INFO:teuthology.orchestra.run.smithi193.stdout:Package libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:17.681 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:17.682 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:45:17.682 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:17.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs-devel 2024-03-24T11:45:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:17.909 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:17.910 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:18.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-03-24T11:45:18.301 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:18.449 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 31 k 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-24T11:45:18.450 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:18.451 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 31 k 2024-03-24T11:45:18.451 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 140 k 2024-03-24T11:45:18.451 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:45:18.609 INFO:teuthology.orchestra.run.smithi193.stdout:libcephfs-devel-19.0.0-1667.gdb0330b1.el9.x86_6 197 kB/s | 31 kB 00:00 2024-03-24T11:45:18.610 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:18.610 INFO:teuthology.orchestra.run.smithi193.stdout:Total 195 kB/s | 31 kB 00:00 2024-03-24T11:45:18.610 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:18.613 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:18.613 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:18.640 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:18.640 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:18.695 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:18.738 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:18.795 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:18.848 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:18.850 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:18.851 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:45:18.851 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:18.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-03-24T11:45:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:19.468 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:19.468 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:19.468 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:19.468 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:19.468 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:19.468 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:19.480 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:19.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados2 2024-03-24T11:45:19.638 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:19.638 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-24T11:45:19.638 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:19.638 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 31 k 2024-03-24T11:45:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:19.639 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:19.639 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:19.639 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-24T11:45:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:19.639 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 31 k 2024-03-24T11:45:19.639 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 140 k 2024-03-24T11:45:19.639 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:19.887 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.0.0-1667.gdb0330b1.el9.x86_6 126 kB/s | 31 kB 00:00 2024-03-24T11:45:19.887 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:19.888 INFO:teuthology.orchestra.run.smithi064.stdout:Total 125 kB/s | 31 kB 00:00 2024-03-24T11:45:19.888 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:19.891 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:19.891 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:19.922 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:19.923 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:20.194 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:20.291 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:20.342 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:20.343 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:45:20.343 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:20.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librbd1 2024-03-24T11:45:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:20.810 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:20.810 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:20.960 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:20.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-03-24T11:45:21.057 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:21.108 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:21.109 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:45:21.109 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:21.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rados 2024-03-24T11:45:21.538 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:43 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:21.636 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:21.691 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:21.692 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:45:21.692 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:21.731 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:38 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:21.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-03-24T11:45:21.828 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:21.880 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:21.881 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:45:21.881 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:21.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rgw 2024-03-24T11:45:22.322 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:22.421 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:22.475 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:22.476 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:45:22.476 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:22.512 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:22.608 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:22.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-03-24T11:45:22.659 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:22.660 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:45:22.660 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:22.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-cephfs 2024-03-24T11:45:23.190 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:45 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:23.290 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:23.344 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:23.345 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:45:23.346 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:23.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-03-24T11:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:23.503 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rbd 2024-03-24T11:45:23.987 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:45 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:24.064 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:24.087 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:24.141 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:24.142 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:45:24.142 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:24.161 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:24.212 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:24.213 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-24T11:45:24.213 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:24.273 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-fuse 2024-03-24T11:45:24.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-03-24T11:45:24.835 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:24.880 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:46 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:24.980 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:24.982 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 85 k 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-24T11:45:24.983 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:24.984 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 85 k 2024-03-24T11:45:24.984 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 231 k 2024-03-24T11:45:24.984 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:45:25.035 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:25.036 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:45:25.036 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:25.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-03-24T11:45:25.179 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-fuse-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 438 kB/s | 85 kB 00:00 2024-03-24T11:45:25.179 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:25.180 INFO:teuthology.orchestra.run.smithi193.stdout:Total 435 kB/s | 85 kB 00:00 2024-03-24T11:45:25.180 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:25.187 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:25.187 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:25.242 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:25.242 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:25.679 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:47 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:25.778 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:45:25.832 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:25.833 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-24T11:45:25.833 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:25.875 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:25.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-03-24T11:45:26.152 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:26.152 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:26.152 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:26.152 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:26.152 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:26.152 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:26.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-mirror 2024-03-24T11:45:26.460 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:48 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:26.613 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 85 k 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:26.614 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-03-24T11:45:26.615 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-03-24T11:45:26.615 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:26.865 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 341 kB/s | 85 kB 00:00 2024-03-24T11:45:26.865 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:26.866 INFO:teuthology.orchestra.run.smithi064.stdout:Total 339 kB/s | 85 kB 00:00 2024-03-24T11:45:26.866 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:26.873 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:26.874 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:26.931 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:26.931 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:26.960 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:43 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:27.116 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:27.116 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-24T11:45:27.116 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:27.116 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:45:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:45:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:27.117 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:45:27.117 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:27.117 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-24T11:45:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:27.117 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 3.1 M 2024-03-24T11:45:27.117 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 13 M 2024-03-24T11:45:27.118 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:45:27.192 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:27.307 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:27.684 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:27.834 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-mirror-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 4.3 MB/s | 3.1 MB 00:00 2024-03-24T11:45:27.834 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:27.835 INFO:teuthology.orchestra.run.smithi193.stdout:Total 4.3 MB/s | 3.1 MB 00:00 2024-03-24T11:45:27.835 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:27.844 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:27.844 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:27.916 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:27.916 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:28.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-03-24T11:45:28.338 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:28.435 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:45:28.435 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T11:45:28.435 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:45:28.435 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:45:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:28.844 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:50 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:29.000 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:29.001 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-24T11:45:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:29.002 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-03-24T11:45:29.002 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 13 M 2024-03-24T11:45:29.002 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:29.560 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:29.560 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:29.619 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-24T11:45:29.619 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:29.620 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-03-24T11:45:29.620 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:29.630 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:29.630 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:29.707 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:29.707 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:29.812 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-nbd 2024-03-24T11:45:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:30.236 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:30.236 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:45:30.236 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T11:45:30.236 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:45:30.236 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:45:30.236 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:30.383 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:47 ago on Sun 24 Mar 2024 11:43:43 AM UTC. 2024-03-24T11:45:30.533 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 171 k 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-24T11:45:30.534 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-24T11:45:30.535 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:30.535 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 171 k 2024-03-24T11:45:30.535 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 494 k 2024-03-24T11:45:30.535 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T11:45:30.747 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-nbd-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 807 kB/s | 171 kB 00:00 2024-03-24T11:45:30.747 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:30.747 INFO:teuthology.orchestra.run.smithi193.stdout:Total 801 kB/s | 171 kB 00:00 2024-03-24T11:45:30.747 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T11:45:30.755 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T11:45:30.755 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T11:45:30.811 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T11:45:30.812 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T11:45:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T11:45:31.062 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:31.319 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:31.319 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:31.508 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:31.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-03-24T11:45:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:31.818 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T11:45:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:45:31.818 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T11:45:32.058 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:32.098 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:54 ago on Sun 24 Mar 2024 11:43:38 AM UTC. 2024-03-24T11:45:32.251 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-24T11:45:32.252 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-24T11:45:32.252 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:32.252 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-24T11:45:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 171 k 2024-03-24T11:45:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:32.253 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-24T11:45:32.253 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-24T11:45:32.253 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-24T11:45:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:32.253 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 171 k 2024-03-24T11:45:32.253 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 494 k 2024-03-24T11:45:32.253 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-24T11:45:32.881 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 272 kB/s | 171 kB 00:00 2024-03-24T11:45:32.882 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:32.882 INFO:teuthology.orchestra.run.smithi064.stdout:Total 271 kB/s | 171 kB 00:00 2024-03-24T11:45:32.882 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-24T11:45:32.890 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-24T11:45:32.890 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-24T11:45:32.949 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-24T11:45:32.950 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-24T11:45:33.102 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-24T11:45:33.207 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:33.630 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:45:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:33.925 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-24T11:45:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:45:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:45:33.925 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-24T11:45:34.201 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:34.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:34.342 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T11:45:34.362 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-1667.gdb0330b1.el9 2024-03-24T11:45:34.363 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1667.gdb0330b1.el9 2024-03-24T11:45:34.363 INFO:teuthology.task.install:The correct ceph version 19.0.0-1667.gdb0330b1 is installed. 2024-03-24T11:45:34.365 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:34.625 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T11:45:34.651 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-1667.gdb0330b1.el9 2024-03-24T11:45:34.651 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1667.gdb0330b1.el9 2024-03-24T11:45:34.651 INFO:teuthology.task.install:The correct ceph version 19.0.0-1667.gdb0330b1 is installed. 2024-03-24T11:45:34.653 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T11:45:34.653 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:45:34.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T11:45:34.682 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:45:34.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T11:45:34.722 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T11:45:34.723 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:45:34.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T11:45:34.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T11:45:34.817 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:45:34.817 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T11:45:34.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T11:45:34.918 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T11:45:34.918 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:45:34.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T11:45:34.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T11:45:35.011 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:45:35.011 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T11:45:35.040 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T11:45:35.108 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T11:45:35.108 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:45:35.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T11:45:35.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T11:45:35.200 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:45:35.200 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T11:45:35.227 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T11:45:35.294 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T11:45:35.393 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'cephadm_mode': 'root'} 2024-03-24T11:45:35.393 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:35.393 INFO:tasks.cephadm:Cluster fsid is 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:45:35.394 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T11:45:35.394 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.64:6789]', 'mon.c': '[v1:172.21.15.64:6790]', 'mon.b': '[v1:172.21.15.193:6789]'} 2024-03-24T11:45:35.394 INFO:tasks.cephadm:First mon is mon.a on smithi064 2024-03-24T11:45:35.394 INFO:tasks.cephadm:First mgr is y 2024-03-24T11:45:35.394 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T11:45:35.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-03-24T11:45:35.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-03-24T11:45:35.449 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T11:45:35.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:35.704 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77704/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1667-gdb0330b1', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1667.gdb0330b1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-16 11:55:36.797747', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-24T11:45:35.917 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 db0330b1e4e2470d52b750e251e55a522b4f7d69 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:35.918 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm 2024-03-24T11:45:35.918 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:45:36.373 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:45 /home/ubuntu/cephtest/cephadm 2024-03-24T11:45:36.374 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:45:37.048 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:45 /home/ubuntu/cephtest/cephadm 2024-03-24T11:45:37.049 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:45:37.067 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:45:37.096 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on all hosts... 2024-03-24T11:45:37.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:45:37.110 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:45:37.942 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:45:37.966 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:46:18.110 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:46:18.110 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:46:18.111 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:46:18.111 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-03-24T11:46:18.111 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:46:18.111 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:46:18.111 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:46:20.587 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:46:20.588 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:46:20.588 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:46:20.588 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-03-24T11:46:20.588 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:46:20.588 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:46:20.588 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:46:20.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-03-24T11:46:20.648 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-03-24T11:46:20.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-03-24T11:46:20.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-03-24T11:46:20.745 INFO:tasks.cephadm:Writing seed config... 2024-03-24T11:46:20.746 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T11:46:20.746 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T11:46:20.746 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-24T11:46:20.746 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-24T11:46:20.746 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-24T11:46:20.746 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-24T11:46:20.746 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-24T11:46:20.746 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-24T11:46:20.747 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-24T11:46:20.748 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T11:46:20.748 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-24T11:46:20.748 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:46:20.748 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T11:46:20.775 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 06f23456-e9d4-11ee-95cd-87774f69a715 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-24T11:46:20.775 DEBUG:teuthology.orchestra.run.smithi064:mon.a> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T11:46:20.817 DEBUG:teuthology.orchestra.run.smithi064:mgr.y> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T11:46:20.859 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T11:46:20.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 -v bootstrap --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.64:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:46:21.058 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:46:21.058 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69', '-v', 'bootstrap', '--fsid', '06f23456-e9d4-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.64:6789]', '--skip-admin-label'] 2024-03-24T11:46:21.083 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:46:21.084 INFO:teuthology.orchestra.run.smithi064.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T11:46:21.084 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying podman|docker is present... 2024-03-24T11:46:21.109 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:46:21.109 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying lvm2 is present... 2024-03-24T11:46:21.109 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying time synchronization is in place... 2024-03-24T11:46:21.116 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:46:21.116 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:46:21.122 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:46:21.122 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-03-24T11:46:21.128 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-03-24T11:46:21.134 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-03-24T11:46:21.134 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:46:21.134 INFO:teuthology.orchestra.run.smithi064.stdout:Repeating the final host check... 2024-03-24T11:46:21.157 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:46:21.157 INFO:teuthology.orchestra.run.smithi064.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T11:46:21.157 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl is present 2024-03-24T11:46:21.157 INFO:teuthology.orchestra.run.smithi064.stdout:lvcreate is present 2024-03-24T11:46:21.163 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:46:21.163 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:46:21.169 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:46:21.169 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-03-24T11:46:21.175 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-03-24T11:46:21.181 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-03-24T11:46:21.181 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:46:21.181 INFO:teuthology.orchestra.run.smithi064.stdout:Host looks OK 2024-03-24T11:46:21.182 INFO:teuthology.orchestra.run.smithi064.stdout:Cluster fsid: 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:46:21.182 INFO:teuthology.orchestra.run.smithi064.stdout:Acquiring lock 140329681593968 on /run/cephadm/06f23456-e9d4-11ee-95cd-87774f69a715.lock 2024-03-24T11:46:21.182 INFO:teuthology.orchestra.run.smithi064.stdout:Lock 140329681593968 acquired on /run/cephadm/06f23456-e9d4-11ee-95cd-87774f69a715.lock 2024-03-24T11:46:21.182 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 6789 ... 2024-03-24T11:46:21.182 INFO:teuthology.orchestra.run.smithi064.stdout:Base mon IP(s) is [172.21.15.64:6789], mon addrv is [v1:172.21.15.64:6789] 2024-03-24T11:46:21.185 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.64 metric 100 2024-03-24T11:46:21.185 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.64 metric 100 2024-03-24T11:46:21.188 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T11:46:21.188 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T11:46:21.188 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1768sec hoplimit 64 pref medium 2024-03-24T11:46:21.190 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T11:46:21.190 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T11:46:21.190 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:46:21.190 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T11:46:21.190 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8df/64 scope link 2024-03-24T11:46:21.190 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:46:21.191 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-03-24T11:46:21.191 INFO:teuthology.orchestra.run.smithi064.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-24T11:46:21.191 INFO:teuthology.orchestra.run.smithi064.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T11:46:21.192 INFO:teuthology.orchestra.run.smithi064.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:46:21.712 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:46:21.713 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:46:21.713 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T11:46:21.713 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:989de1c398e87dbcb4906791a352373dc0e0415d619bc5b36f1f1823df929cb1 2024-03-24T11:46:21.713 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-24T11:46:21.713 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying config sha256:4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:46:21.713 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T11:46:24.857 INFO:teuthology.orchestra.run.smithi064.stdout:ceph: stdout ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:46:24.857 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph version: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:46:24.857 INFO:teuthology.orchestra.run.smithi064.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T11:46:25.913 INFO:teuthology.orchestra.run.smithi064.stdout:stat: stdout 167 167 2024-03-24T11:46:25.913 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial keys... 2024-03-24T11:46:26.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQCSEgBmwntZEBAAPyl4qstyQmpq2+x8HXqA7w== 2024-03-24T11:46:27.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQCTEgBmzKqyDBAAefz3KAVDq+Y1PcUF28r8Ng== 2024-03-24T11:46:28.846 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQCUEgBmfnemCRAATr5aT1/llTzCgc85ipOfUA== 2024-03-24T11:46:28.847 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial monmap... 2024-03-24T11:46:33.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:46:33.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T11:46:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:46:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:46:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool for a [v1:172.21.15.64:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:46:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-03-24T11:46:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: set fsid to 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:46:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:46:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:46:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mon... 2024-03-24T11:46:39.260 INFO:teuthology.orchestra.run.smithi064.stdout:create mon.a on 2024-03-24T11:46:39.466 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T11:46:39.602 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T11:46:39.772 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-06f23456-e9d4-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-06f23456-e9d4-11ee-95cd-87774f69a715.target. 2024-03-24T11:46:39.772 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-06f23456-e9d4-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-06f23456-e9d4-11ee-95cd-87774f69a715.target. 2024-03-24T11:46:39.994 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:46:39.994 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T11:46:40.159 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-06f23456-e9d4-11ee-95cd-87774f69a715.target.wants/ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-06f23456-e9d4-11ee-95cd-87774f69a715@.service. 2024-03-24T11:46:40.766 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:46:40.767 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-03-24T11:46:40.773 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:46:40.773 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-03-24T11:46:40.773 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-03-24T11:46:40.773 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:46:40.774 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon to start... 2024-03-24T11:46:40.774 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon... 2024-03-24T11:46:42.289 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T11:46:42.289 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout id: 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:46:42.289 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T11:46:42.289 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:42.289 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout services: 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 1.04524s) 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout data: 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:mon is available 2024-03-24T11:46:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.64:6789] 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:46:43.345 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:46:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:46:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:46:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:46:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:46:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:Generating new minimal ceph.conf... 2024-03-24T11:46:44.934 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:44 smithi064 ceph-mon[29750]: from='client.? v1:172.21.15.64:0/191198260' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:44.938 INFO:teuthology.orchestra.run.smithi064.stdout:Restarting the monitor... 2024-03-24T11:46:45.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:44 smithi064 systemd[1]: Stopping Ceph mon.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:46:45.607 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:45 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a[29727]: 2024-03-24T11:46:45.308+0000 7f74ffdc6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:46:45.607 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:45 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a[29727]: 2024-03-24T11:46:45.308+0000 7f74ffdc6700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:46:45.607 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:45 smithi064 podman[30072]: 2024-03-24 11:46:45.329827968 +0000 UTC m=+0.219980722 container died d8a4b230c483fa858b981e4ed335f62c73f6b584fad022b7e60a2f9a186c189b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T11:46:45.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:45 smithi064 podman[30072]: 2024-03-24 11:46:45.689948717 +0000 UTC m=+0.580101478 container cleanup d8a4b230c483fa858b981e4ed335f62c73f6b584fad022b7e60a2f9a186c189b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T11:46:45.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:45 smithi064 bash[30072]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:46:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:45 smithi064 podman[30091]: 2024-03-24 11:46:45.94911288 +0000 UTC m=+0.615964068 container remove d8a4b230c483fa858b981e4ed335f62c73f6b584fad022b7e60a2f9a186c189b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T11:46:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T11:46:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 systemd[1]: Stopped Ceph mon.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:46:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 systemd[1]: Starting Ceph mon.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:46:46.564 INFO:teuthology.orchestra.run.smithi064.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T11:46:46.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 podman[30185]: 2024-03-24 11:46:46.232637072 +0000 UTC m=+0.017680913 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:46.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 podman[30185]: 2024-03-24 11:46:46.336252124 +0000 UTC m=+0.121295946 container create 14686e0dd6a64be73fc7f01107e418099640a63202422466e0d2b74ea16fc72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T11:46:46.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 podman[30185]: 2024-03-24 11:46:46.502272511 +0000 UTC m=+0.287316346 container init 14686e0dd6a64be73fc7f01107e418099640a63202422466e0d2b74ea16fc72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 podman[30185]: 2024-03-24 11:46:46.50804283 +0000 UTC m=+0.293086652 container start 14686e0dd6a64be73fc7f01107e418099640a63202422466e0d2b74ea16fc72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: pidfile_write: ignore empty --pid-file 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: load: jerasure load: lrc 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Git sha 0 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: DB SUMMARY 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: DB Session ID: B607LACU2VII0LBADSLI 2024-03-24T11:46:46.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:46:46.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:46:46.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T11:46:46.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T11:46:46.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 89833 ; 2024-03-24T11:46:46.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:46:46.622 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.env: 0x5562421b7ae0 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.info_log: 0x556242a6b900 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.statistics: (nil) 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.use_fsync: 0 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:46:46.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.db_log_dir: 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.wal_dir: 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:46:46.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.write_buffer_manager: 0x556242a3f400 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:46:46.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.unordered_write: 0 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.row_cache: None 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.wal_filter: None 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.wal_compression: 0 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:46:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:46:46.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_open_files: -1 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Compression algorithms supported: 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: kZSTD supported: 0 2024-03-24T11:46:46.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.merge_operator: 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_filter: None 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:46:46.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556242a6ba00) 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_type: 0 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_index_type: 0 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_shortening: 1 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: checksum: 4 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: no_block_cache: 0 2024-03-24T11:46:46.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache: 0x556242a6f090 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_options: 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: capacity : 536870912 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_shard_bits : 4 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: strict_capacity_limit : 0 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_compressed: (nil) 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: persistent_cache: (nil) 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size: 4096 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size_deviation: 10 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_restart_interval: 16 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_block_restart_interval: 1 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: metadata_block_size: 4096 2024-03-24T11:46:46.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: partition_filters: 0 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: use_delta_encoding: 1 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: filter_policy: bloomfilter 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: whole_key_filtering: 1 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: verify_compression: 0 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: format_version: 5 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_index_compression: 1 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_align: 0 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: max_auto_readahead_size: 262144 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: prepopulate_block_cache: 0 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:46:46.632 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression: NoCompression 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.num_levels: 7 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:46:46.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:46:46.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:46:46.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:46:46.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.ttl: 2592000 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:46:46.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5731a7df-c0ce-4626-8304-b9f915e23181 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280806536735, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T11:46:46.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280806537655, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 85633, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 286, "table_properties": {"data_size": 83735, "index_size": 244, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11721, "raw_average_key_size": 48, "raw_value_size": 77259, "raw_average_value_size": 320, "num_data_blocks": 11, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280806, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5731a7df-c0ce-4626-8304-b9f915e23181", "db_session_id": "B607LACU2VII0LBADSLI", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280806537736, "job": 1, "event": "recovery_finished"} 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556242b60000 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: DB pointer 0x556242b4a000 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** DB Stats ** 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:46:46.639 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 2/0 85.43 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 85.43 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:46:46.640 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 27.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 27.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache BinnedLRUCache@0x556242a6f090#7 capacity: 512.00 MB usage: 1.27 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:46:46.641 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: starting mon.a rank 0 at public addrs v1:172.21.15.64:6789/0 at bind addrs v1:172.21.15.64:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a@-1(???) e1 preinit fsid 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a@-1(???).mds e1 new map 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a@-1(???).mds e1 print_map 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout: e1 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout: legacy client fscid: -1 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout: No filesystems configured 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:46:46.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 bash[30185]: 14686e0dd6a64be73fc7f01107e418099640a63202422466e0d2b74ea16fc72f 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 systemd[1]: Started Ceph mon.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: monmap e1: 1 mons at {a=v1:172.21.15.64:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: fsmap 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:46:46.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:46 smithi064 ceph-mon[30223]: mgrmap e1: no daemons active 2024-03-24T11:46:47.637 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T11:46:47.658 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:46:47.658 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mgr... 2024-03-24T11:46:47.659 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T11:46:47.659 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T11:46:47.904 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:46:47.905 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-24T11:46:48.050 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-06f23456-e9d4-11ee-95cd-87774f69a715.target.wants/ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-06f23456-e9d4-11ee-95cd-87774f69a715@.service. 2024-03-24T11:46:48.206 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:48 smithi064 systemd[1]: Starting Ceph mgr.y for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:46:48.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:48 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2514084063' entity='client.admin' 2024-03-24T11:46:48.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:48 smithi064 podman[30470]: 2024-03-24 11:46:48.186351177 +0000 UTC m=+0.016861578 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:48.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:48 smithi064 podman[30470]: 2024-03-24 11:46:48.344385713 +0000 UTC m=+0.174896099 container create 561c09f8ef41fd5d9946247d3cca2ce2a51bde0757b8ece3744f062ce3b2a597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:46:48.677 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:46:48.677 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-03-24T11:46:48.684 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:46:48.684 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-03-24T11:46:48.684 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-03-24T11:46:48.684 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:46:48.692 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:46:48.692 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-03-24T11:46:48.699 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:46:48.700 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-03-24T11:46:48.700 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-03-24T11:46:48.700 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T11:46:48.700 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr to start... 2024-03-24T11:46:48.700 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr... 2024-03-24T11:46:48.828 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:48 smithi064 podman[30470]: 2024-03-24 11:46:48.554993287 +0000 UTC m=+0.385503686 container init 561c09f8ef41fd5d9946247d3cca2ce2a51bde0757b8ece3744f062ce3b2a597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:46:48.828 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:48 smithi064 podman[30470]: 2024-03-24 11:46:48.560399743 +0000 UTC m=+0.390910143 container start 561c09f8ef41fd5d9946247d3cca2ce2a51bde0757b8ece3744f062ce3b2a597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:46:48.828 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:48 smithi064 bash[30470]: 561c09f8ef41fd5d9946247d3cca2ce2a51bde0757b8ece3744f062ce3b2a597 2024-03-24T11:46:48.828 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:48 smithi064 systemd[1]: Started Ceph mgr.y for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:46:49.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:49 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1941269985' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:46:49.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:49.128+0000 7f612c50c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:46:49.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:49.227+0000 7f612c50c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:46:49.878 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:49.878 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:49.878 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T11:46:49.879 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:46:49.880 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:46:40.657527+0000", 2024-03-24T11:46:49.882 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:49.883 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:49.883 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:46:49.883 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:49.883 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (1/15)... 2024-03-24T11:46:50.147 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:49.908+0000 7f612c50c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:46:50.147 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:50.003+0000 7f612c50c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:46:50.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:50.146+0000 7f612c50c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:46:50.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:50.243+0000 7f612c50c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:46:50.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:50.327+0000 7f612c50c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:46:50.895 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:50.597+0000 7f612c50c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:46:50.895 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:50.710+0000 7f612c50c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:46:50.895 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:50.798+0000 7f612c50c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:46:50.895 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:50.894+0000 7f612c50c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:46:51.302 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:51.071+0000 7f612c50c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:46:52.297 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:52.074+0000 7f612c50c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:46:52.298 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:52.213+0000 7f612c50c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:46:52.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/10921346' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:46:52.848 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:52.554+0000 7f612c50c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:46:52.848 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:52.646+0000 7f612c50c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:46:52.848 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:52.847+0000 7f612c50c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:46:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T11:46:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:46:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:46:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:53.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:46:53.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:53.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:46:53.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:46:53.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:46:53.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:46:53.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:46:53.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:46:53.053 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:53.053 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:46:53.053 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:46:53.053 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:46:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:46:40.657527+0000", 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:53.055 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (2/15)... 2024-03-24T11:46:53.302 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:53.035+0000 7f612c50c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:46:53.587 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:53.326+0000 7f612c50c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:46:53.588 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:53.587+0000 7f612c50c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:46:54.052 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:53.827+0000 7f612c50c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:46:55.052 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:54 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:54.649+0000 7f612c50c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:46:55.437 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:55.270+0000 7f612c50c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:46:55.437 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:55.388+0000 7f612c50c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:46:55.802 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:46:55.615+0000 7f612c50c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3452774622' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: Activating manager daemon y 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: mgrmap e2: y(active, starting, since 0.00301276s) 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:46:55.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: Manager daemon y is now available 2024-03-24T11:46:55.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:46:55.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:46:55.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:55 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:46:56.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:56.131 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:46:56.131 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:46:56.131 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:46:56.131 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:46:56.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:46:56.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:46:56.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:56.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:46:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:46:40.657527+0000", 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:46:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (3/15)... 2024-03-24T11:46:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:56 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' 2024-03-24T11:46:57.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:56 smithi064 ceph-mon[30223]: from='mgr.14100 v1:172.21.15.64:0/1090363908' entity='mgr.y' 2024-03-24T11:46:57.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:56 smithi064 ceph-mon[30223]: mgrmap e3: y(active, since 1.0073s) 2024-03-24T11:46:58.875 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:58 smithi064 ceph-mon[30223]: mgrmap e4: y(active, since 2s) 2024-03-24T11:46:59.493 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:46:59.494 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:46:59.495 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:46:59.498 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:59.498 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:46:59.499 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:46:40.657527+0000", 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:mgr is available 2024-03-24T11:46:59.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:46:59 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2926890074' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:47:00.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:47:00.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:47:00.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:47:00.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:47:00.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:47:00.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:47:00.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:47:00.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:47:00.809 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:47:00.809 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:47:00.809 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-03-24T11:47:00.809 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:47:00.809 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:47:00.809 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:47:00.809 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling cephadm module... 2024-03-24T11:47:00.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:00 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/116970126' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:47:02.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:01 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2280214450' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T11:47:02.053 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:01 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ignoring --setuser ceph since I am not root 2024-03-24T11:47:02.053 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:01 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ignoring --setgroup ceph since I am not root 2024-03-24T11:47:02.513 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:02 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:02.255+0000 7ff6e2665200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:47:02.513 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:02 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:02.351+0000 7ff6e2665200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:47:02.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2280214450' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T11:47:02.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:02 smithi064 ceph-mon[30223]: mgrmap e5: y(active, since 6s) 2024-03-24T11:47:03.216 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:03.026+0000 7ff6e2665200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:47:03.216 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:03.122+0000 7ff6e2665200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:47:03.461 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:47:03.461 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T11:47:03.461 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:47:03.461 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:47:03.461 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:47:03.461 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:47:03.461 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-03-24T11:47:03.462 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 5... 2024-03-24T11:47:03.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:03.265+0000 7ff6e2665200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:47:03.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:03.362+0000 7ff6e2665200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:47:03.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:03.448+0000 7ff6e2665200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:47:03.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/801366215' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:47:03.803 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:03.731+0000 7ff6e2665200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:47:04.209 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:03.845+0000 7ff6e2665200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:47:04.209 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:03 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:03.935+0000 7ff6e2665200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:47:04.209 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:04 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:04.031+0000 7ff6e2665200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:47:04.209 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:04 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:04.208+0000 7ff6e2665200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:47:05.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:05 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:05.189+0000 7ff6e2665200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:47:05.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:05 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:05.321+0000 7ff6e2665200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:47:05.949 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:05 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:05.656+0000 7ff6e2665200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:47:05.949 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:05 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:05.748+0000 7ff6e2665200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:47:05.949 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:05 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:05.948+0000 7ff6e2665200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:47:06.302 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:06 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:06.137+0000 7ff6e2665200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:47:06.691 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:06 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:06.430+0000 7ff6e2665200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:47:06.691 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:06 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:06.690+0000 7ff6e2665200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:47:07.052 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:06 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:06.930+0000 7ff6e2665200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:47:08.052 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:07.749+0000 7ff6e2665200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:47:08.649 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:08.329+0000 7ff6e2665200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:47:08.649 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:08.428+0000 7ff6e2665200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:47:08.649 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:08.648+0000 7ff6e2665200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:47:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: Active manager daemon y restarted 2024-03-24T11:47:09.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: Activating manager daemon y 2024-03-24T11:47:09.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T11:47:09.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: mgrmap e6: y(active, starting, since 0.00313096s) 2024-03-24T11:47:09.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:47:09.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:47:09.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:47:09.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:47:09.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:47:09.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: Manager daemon y is now available 2024-03-24T11:47:09.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:09.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:09.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:09.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:08 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:10.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:09 smithi064 ceph-mon[30223]: Found migration_current of "None". Setting to last migration. 2024-03-24T11:47:10.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:09 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:10.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:09 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:47:10.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:09 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:47:10.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:09 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:10.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:09 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:10.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:09 smithi064 ceph-mon[30223]: mgrmap e7: y(active, since 1.00534s) 2024-03-24T11:47:10.267 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:47:10.267 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T11:47:10.267 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:47:10.268 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:47:10.268 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 5 is available 2024-03-24T11:47:10.268 INFO:teuthology.orchestra.run.smithi064.stdout:Setting orchestrator backend to cephadm... 2024-03-24T11:47:10.850 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:10 smithi064 ceph-mon[30223]: from='client.14124 v1:172.21.15.64:0/3081832633' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:47:10.850 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:10 smithi064 ceph-mon[30223]: from='client.14124 v1:172.21.15.64:0/3081832633' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:47:11.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:11 smithi064 ceph-mon[30223]: [24/Mar/2024:11:47:10] ENGINE Bus STARTING 2024-03-24T11:47:11.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:11 smithi064 ceph-mon[30223]: [24/Mar/2024:11:47:10] ENGINE Serving on https://172.21.15.64:7150 2024-03-24T11:47:11.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:11 smithi064 ceph-mon[30223]: mgrmap e8: y(active, since 2s) 2024-03-24T11:47:11.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:11 smithi064 ceph-mon[30223]: [24/Mar/2024:11:47:10] ENGINE Serving on http://172.21.15.64:8765 2024-03-24T11:47:11.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:11 smithi064 ceph-mon[30223]: [24/Mar/2024:11:47:10] ENGINE Bus STARTED 2024-03-24T11:47:11.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:11 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:11.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:11 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:11.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:11 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:12.776 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:12 smithi064 ceph-mon[30223]: from='client.14132 v1:172.21.15.64:0/3237233215' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:12.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T11:47:12.779 INFO:teuthology.orchestra.run.smithi064.stdout:Generating ssh key... 2024-03-24T11:47:14.013 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:13 smithi064 ceph-mon[30223]: from='client.14134 v1:172.21.15.64:0/308639062' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:14.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: Generating public/private rsa key pair. 2024-03-24T11:47:14.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: Your identification has been saved in /tmp/tmprfsw4w4c/key. 2024-03-24T11:47:14.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: Your public key has been saved in /tmp/tmprfsw4w4c/key.pub. 2024-03-24T11:47:14.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: The key fingerprint is: 2024-03-24T11:47:14.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: SHA256:CLalfyfnY8dPCmIElIssahK58KbRX+5+pznlpclsXTs ceph-06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:47:14.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: The key's randomart image is: 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: +---[RSA 3072]----+ 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: | .. | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: | .. | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: | . .o.o. | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: |+ ..o=.o | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: |.* .o . S | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: |=.+ ... . . . | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: |o+ . o. ==+* ... | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: |. . .ooB@.+oE | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: | oo.o*.o.... | 2024-03-24T11:47:14.304 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: +----[SHA256]-----+ 2024-03-24T11:47:14.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-mon[30223]: from='client.14136 v1:172.21.15.64:0/2595636719' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:14.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-mon[30223]: Generating ssh key... 2024-03-24T11:47:14.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:14.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:14 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:15.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:15 smithi064 ceph-mon[30223]: from='client.14138 v1:172.21.15.64:0/3414314037' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:15.808 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0SqYLu3e1coRSiCRrZpl005rKeb3lOiFACUXZPGfbYOzHLeQbI2jsJ2/zEKVc6pbqYTn7PYqO+XCnDeqWD2HHor2gvkFImuQB1jL9Ss1Bi50JG2ucyS8WEHmFF80PCZcKLF9UrUFNagpFg9gVG5Vo2f/nB7h/Ww5fVp5MRroS9/2zrb5yby0QdWK6zKY+6I6mQSAEEsCQN0P6hK3vXj5dxUwGw7Ehihayy4sbIVTzOWVg9yVlB//9jGkr+piOE3uEshUV4hhWCsRjU110DswmmlUn1wZsDmYRYwDz2kuLv7hWkhxG6YJTY6Oy566uUFmHzycGRyHOgqQAa9YmZpkrCMHHwqIvY4q0I+Wi0qhhkwGReXSS5Q0sZasX7hbMQseNKQJAhHlxFp6o8wBSVHUPVLdz130U4rJciCSj5wtudpqQKAkgy54XOZ5SufwURWHTDP7et0q8w24FNx4XEYRPLjigorESAk7Kt9puoEWlh6PRzoaghX0eY45VwPTvtpU= ceph-06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:47:15.808 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:47:15.808 INFO:teuthology.orchestra.run.smithi064.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T11:47:15.809 INFO:teuthology.orchestra.run.smithi064.stdout:Adding host smithi064... 2024-03-24T11:47:17.037 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:16 smithi064 ceph-mon[30223]: from='client.14140 v1:172.21.15.64:0/520213141' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:18.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:17 smithi064 ceph-mon[30223]: Deploying cephadm binary to smithi064 2024-03-24T11:47:21.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Added host 'smithi064' with addr '172.21.15.64' 2024-03-24T11:47:21.341 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mon service... 2024-03-24T11:47:22.037 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:21 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:22.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:21 smithi064 ceph-mon[30223]: Added host smithi064 2024-03-24T11:47:22.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:21 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:22.810 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T11:47:22.810 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mgr service... 2024-03-24T11:47:23.062 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:23 smithi064 ceph-mon[30223]: from='client.14142 v1:172.21.15.64:0/3548321910' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:23.062 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:23 smithi064 ceph-mon[30223]: Saving service mon spec with placement count:5 2024-03-24T11:47:23.062 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:23 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:24.854 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T11:47:25.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:24 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:25.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:24 smithi064 ceph-mon[30223]: from='client.14144 v1:172.21.15.64:0/120047894' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:25.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:24 smithi064 ceph-mon[30223]: Saving service mgr spec with placement count:2 2024-03-24T11:47:25.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:24 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:25.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:24 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:26.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:26 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/994497973' entity='client.admin' 2024-03-24T11:47:27.579 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling the dashboard module... 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2482458162' entity='client.admin' 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:27 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:29.298 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3931856868' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T11:47:29.299 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:28 smithi064 ceph-mon[30223]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:29.299 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:28 smithi064 ceph-mon[30223]: from='mgr.14120 v1:172.21.15.64:0/1004914477' entity='mgr.y' 2024-03-24T11:47:29.299 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:29 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ignoring --setuser ceph since I am not root 2024-03-24T11:47:29.299 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:29 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ignoring --setgroup ceph since I am not root 2024-03-24T11:47:29.817 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:29 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:29.562+0000 7fb185c7f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:47:29.817 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:29 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:29.662+0000 7fb185c7f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:47:30.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:29 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3931856868' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T11:47:30.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:29 smithi064 ceph-mon[30223]: mgrmap e9: y(active, since 20s) 2024-03-24T11:47:30.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:30 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:30.340+0000 7fb185c7f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:47:30.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:30 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:30.445+0000 7fb185c7f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:47:30.978 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:30 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:30.589+0000 7fb185c7f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:47:30.978 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:30 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:30.690+0000 7fb185c7f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:47:30.978 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:30 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:30.774+0000 7fb185c7f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:47:31.249 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:30 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2400054465' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:47:31.250 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:31 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:31.049+0000 7fb185c7f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:47:31.250 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:31 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:31.161+0000 7fb185c7f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:47:31.250 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:31 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:31.248+0000 7fb185c7f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:47:31.525 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:31 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:31.346+0000 7fb185c7f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:47:31.526 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:31 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:31.524+0000 7fb185c7f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:47:31.907 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:47:31.907 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T11:47:31.907 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:47:31.907 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:47:31.907 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:47:31.907 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:47:31.907 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-03-24T11:47:31.907 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 9... 2024-03-24T11:47:32.802 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:32 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:32.542+0000 7fb185c7f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:47:32.803 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:32 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:32.677+0000 7fb185c7f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:47:33.302 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:33.016+0000 7fb185c7f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:47:33.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:33.108+0000 7fb185c7f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:47:33.793 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:33.309+0000 7fb185c7f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:47:33.793 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:33.499+0000 7fb185c7f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:47:34.052 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:33.792+0000 7fb185c7f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:47:34.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:34 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:34.053+0000 7fb185c7f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:47:34.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:34 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:34.295+0000 7fb185c7f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:47:35.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:35 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:35.119+0000 7fb185c7f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:47:36.022 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:35 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:35.700+0000 7fb185c7f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:47:36.022 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:35 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:35.800+0000 7fb185c7f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:47:36.022 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:47:36.021+0000 7fb185c7f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:47:36.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: Active manager daemon y restarted 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: Activating manager daemon y 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: mgrmap e10: y(active, starting, since 0.00301237s) 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: Manager daemon y is now available 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:36.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:37.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:47:37.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:47:37.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:37 smithi064 ceph-mon[30223]: mgrmap e11: y(active, since 1.00653s) 2024-03-24T11:47:38.075 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:47:38.075 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T11:47:38.076 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:47:38.076 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:47:38.076 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 9 is available 2024-03-24T11:47:38.076 INFO:teuthology.orchestra.run.smithi064.stdout:Generating a dashboard self-signed certificate... 2024-03-24T11:47:38.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:38 smithi064 ceph-mon[30223]: [24/Mar/2024:11:47:37] ENGINE Bus STARTING 2024-03-24T11:47:38.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:38 smithi064 ceph-mon[30223]: [24/Mar/2024:11:47:37] ENGINE Serving on http://172.21.15.64:8765 2024-03-24T11:47:38.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:38 smithi064 ceph-mon[30223]: [24/Mar/2024:11:47:37] ENGINE Serving on https://172.21.15.64:7150 2024-03-24T11:47:38.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:38 smithi064 ceph-mon[30223]: [24/Mar/2024:11:47:37] ENGINE Bus STARTED 2024-03-24T11:47:38.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:39.497 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T11:47:39.497 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial admin user... 2024-03-24T11:47:39.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:39 smithi064 ceph-mon[30223]: mgrmap e12: y(active, since 2s) 2024-03-24T11:47:39.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:39 smithi064 ceph-mon[30223]: from='client.14164 v1:172.21.15.64:0/2547761084' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:39.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:39.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:39.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:41.574 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:41 smithi064 ceph-mon[30223]: from='client.14166 v1:172.21.15.64:0/3500482808' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:41.574 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$rW1DdoAv7o1fRVCgxKIfTOLTQCCf6ksFN0fQaD.ls5uG.k3S0jCGu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711280860, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T11:47:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:Fetching dashboard port number... 2024-03-24T11:47:42.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2059412926' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T11:47:42.826 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T11:47:42.835 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:47:42.835 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-03-24T11:47:42.842 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:47:42.842 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-03-24T11:47:42.842 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-03-24T11:47:42.842 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T11:47:42.843 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph Dashboard is now available at: 2024-03-24T11:47:42.843 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:42.843 INFO:teuthology.orchestra.run.smithi064.stdout: URL: https://smithi064.front.sepia.ceph.com:8443/ 2024-03-24T11:47:42.844 INFO:teuthology.orchestra.run.smithi064.stdout: User: admin 2024-03-24T11:47:42.844 INFO:teuthology.orchestra.run.smithi064.stdout: Password: g1q44pvmzz 2024-03-24T11:47:42.844 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:42.844 INFO:teuthology.orchestra.run.smithi064.stdout:Saving cluster configuration to /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config directory 2024-03-24T11:47:42.844 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling autotune for osd_memory_target 2024-03-24T11:47:43.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:43 smithi064 ceph-mon[30223]: mgrmap e13: y(active, since 6s) 2024-03-24T11:47:45.521 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: ceph telemetry on 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:45.522 INFO:teuthology.orchestra.run.smithi064.stdout:For more information see: 2024-03-24T11:47:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T11:47:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:47:45.523 INFO:teuthology.orchestra.run.smithi064.stdout:Bootstrap complete. 2024-03-24T11:47:45.563 INFO:tasks.cephadm:Fetching config... 2024-03-24T11:47:45.563 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:47:45.563 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T11:47:45.578 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T11:47:45.579 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:47:45.579 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T11:47:45.634 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T11:47:45.634 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:47:45.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T11:47:45.700 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T11:47:45.700 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:47:45.700 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T11:47:45.755 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T11:47:45.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0SqYLu3e1coRSiCRrZpl005rKeb3lOiFACUXZPGfbYOzHLeQbI2jsJ2/zEKVc6pbqYTn7PYqO+XCnDeqWD2HHor2gvkFImuQB1jL9Ss1Bi50JG2ucyS8WEHmFF80PCZcKLF9UrUFNagpFg9gVG5Vo2f/nB7h/Ww5fVp5MRroS9/2zrb5yby0QdWK6zKY+6I6mQSAEEsCQN0P6hK3vXj5dxUwGw7Ehihayy4sbIVTzOWVg9yVlB//9jGkr+piOE3uEshUV4hhWCsRjU110DswmmlUn1wZsDmYRYwDz2kuLv7hWkhxG6YJTY6Oy566uUFmHzycGRyHOgqQAa9YmZpkrCMHHwqIvY4q0I+Wi0qhhkwGReXSS5Q0sZasX7hbMQseNKQJAhHlxFp6o8wBSVHUPVLdz130U4rJciCSj5wtudpqQKAkgy54XOZ5SufwURWHTDP7et0q8w24FNx4XEYRPLjigorESAk7Kt9puoEWlh6PRzoaghX0eY45VwPTvtpU= ceph-06f23456-e9d4-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:47:45.833 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0SqYLu3e1coRSiCRrZpl005rKeb3lOiFACUXZPGfbYOzHLeQbI2jsJ2/zEKVc6pbqYTn7PYqO+XCnDeqWD2HHor2gvkFImuQB1jL9Ss1Bi50JG2ucyS8WEHmFF80PCZcKLF9UrUFNagpFg9gVG5Vo2f/nB7h/Ww5fVp5MRroS9/2zrb5yby0QdWK6zKY+6I6mQSAEEsCQN0P6hK3vXj5dxUwGw7Ehihayy4sbIVTzOWVg9yVlB//9jGkr+piOE3uEshUV4hhWCsRjU110DswmmlUn1wZsDmYRYwDz2kuLv7hWkhxG6YJTY6Oy566uUFmHzycGRyHOgqQAa9YmZpkrCMHHwqIvY4q0I+Wi0qhhkwGReXSS5Q0sZasX7hbMQseNKQJAhHlxFp6o8wBSVHUPVLdz130U4rJciCSj5wtudpqQKAkgy54XOZ5SufwURWHTDP7et0q8w24FNx4XEYRPLjigorESAk7Kt9puoEWlh6PRzoaghX0eY45VwPTvtpU= ceph-06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:47:45.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0SqYLu3e1coRSiCRrZpl005rKeb3lOiFACUXZPGfbYOzHLeQbI2jsJ2/zEKVc6pbqYTn7PYqO+XCnDeqWD2HHor2gvkFImuQB1jL9Ss1Bi50JG2ucyS8WEHmFF80PCZcKLF9UrUFNagpFg9gVG5Vo2f/nB7h/Ww5fVp5MRroS9/2zrb5yby0QdWK6zKY+6I6mQSAEEsCQN0P6hK3vXj5dxUwGw7Ehihayy4sbIVTzOWVg9yVlB//9jGkr+piOE3uEshUV4hhWCsRjU110DswmmlUn1wZsDmYRYwDz2kuLv7hWkhxG6YJTY6Oy566uUFmHzycGRyHOgqQAa9YmZpkrCMHHwqIvY4q0I+Wi0qhhkwGReXSS5Q0sZasX7hbMQseNKQJAhHlxFp6o8wBSVHUPVLdz130U4rJciCSj5wtudpqQKAkgy54XOZ5SufwURWHTDP7et0q8w24FNx4XEYRPLjigorESAk7Kt9puoEWlh6PRzoaghX0eY45VwPTvtpU= ceph-06f23456-e9d4-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:47:45.892 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0SqYLu3e1coRSiCRrZpl005rKeb3lOiFACUXZPGfbYOzHLeQbI2jsJ2/zEKVc6pbqYTn7PYqO+XCnDeqWD2HHor2gvkFImuQB1jL9Ss1Bi50JG2ucyS8WEHmFF80PCZcKLF9UrUFNagpFg9gVG5Vo2f/nB7h/Ww5fVp5MRroS9/2zrb5yby0QdWK6zKY+6I6mQSAEEsCQN0P6hK3vXj5dxUwGw7Ehihayy4sbIVTzOWVg9yVlB//9jGkr+piOE3uEshUV4hhWCsRjU110DswmmlUn1wZsDmYRYwDz2kuLv7hWkhxG6YJTY6Oy566uUFmHzycGRyHOgqQAa9YmZpkrCMHHwqIvY4q0I+Wi0qhhkwGReXSS5Q0sZasX7hbMQseNKQJAhHlxFp6o8wBSVHUPVLdz130U4rJciCSj5wtudpqQKAkgy54XOZ5SufwURWHTDP7et0q8w24FNx4XEYRPLjigorESAk7Kt9puoEWlh6PRzoaghX0eY45VwPTvtpU= ceph-06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:47:45.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T11:47:46.047 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3639969228' entity='client.admin' 2024-03-24T11:47:46.209 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:49.524 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T11:47:49.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T11:47:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:49 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4205481782' entity='client.admin' 2024-03-24T11:47:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:51.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:51.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:51.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:47:51.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:51.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:51.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:52.807 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-03-24T11:47:52.808 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:47:52.808 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:47:52.827 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:47:52.828 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:52.885 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-03-24T11:47:52.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch host add smithi193 2024-03-24T11:47:53.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:53 smithi064 ceph-mon[30223]: from='client.14176 v1:172.21.15.64:0/172274561' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:53.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:53.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:53.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:53.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:53.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:53.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:53.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:47:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:54.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:54 smithi064 ceph-mon[30223]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-24T11:47:54.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:54 smithi064 ceph-mon[30223]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:54.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:54 smithi064 ceph-mon[30223]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:54.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:54 smithi064 ceph-mon[30223]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:47:55.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:55 smithi064 ceph-mon[30223]: from='client.14178 v1:172.21.15.64:0/632733500' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:56.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:56 smithi064 ceph-mon[30223]: Deploying cephadm binary to smithi193 2024-03-24T11:47:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:57 smithi064 ceph-mon[30223]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:59.254 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-03-24T11:47:59.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:47:59 smithi064 ceph-mon[30223]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:59.857 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:48:00.101 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:48:00.501 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:00.501 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:00 smithi064 ceph-mon[30223]: Added host smithi193 2024-03-24T11:48:00.501 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:00.501 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:01.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:01 smithi064 ceph-mon[30223]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:01.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:48:01.969 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-03-24T11:48:02.564 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T11:48:02.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T11:48:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:48:03.180 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:02 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:03.180 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:02 smithi064 ceph-mon[30223]: from='client.14180 v1:172.21.15.64:0/1837876198' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:48:04.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:03 smithi064 ceph-mon[30223]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:04.833 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-03-24T11:48:05.206 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1398433670' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:48:05.483 INFO:tasks.cephadm:Adding mon.a on smithi064 2024-03-24T11:48:05.483 INFO:tasks.cephadm:Adding mon.c on smithi064 2024-03-24T11:48:05.483 INFO:tasks.cephadm:Adding mon.b on smithi193 2024-03-24T11:48:05.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi064:[v1:172.21.15.64:6789]=a;smithi064:[v1:172.21.15.64:6790]=c;smithi193:[v1:172.21.15.193:6789]=b' 2024-03-24T11:48:05.721 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1398433670' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:06.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:06.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:48:06.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:06.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:06.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:06 smithi064 ceph-mon[30223]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:06 smithi064 ceph-mon[30223]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:48:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:06 smithi064 ceph-mon[30223]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:48:07.380 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-03-24T11:48:07.949 DEBUG:teuthology.orchestra.run.smithi064:mon.c> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T11:48:07.951 DEBUG:teuthology.orchestra.run.smithi193:mon.b> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T11:48:07.954 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:48:07.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:48:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:07 smithi064 ceph-mon[30223]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:08.951 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:08 smithi064 ceph-mon[30223]: from='client.14184 v1:172.21.15.193:0/1332431082' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:[v1:172.21.15.64:6789]=a;smithi064:[v1:172.21.15.64:6790]=c;smithi193:[v1:172.21.15.193:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:08 smithi064 ceph-mon[30223]: Saving service mon spec with placement smithi064:[v1:172.21.15.64:6789]=a;smithi064:[v1:172.21.15.64:6790]=c;smithi193:[v1:172.21.15.193:6789]=b;count:3 2024-03-24T11:48:09.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:08 smithi064 ceph-mon[30223]: Deploying daemon mon.b on smithi193 2024-03-24T11:48:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:09 smithi064 ceph-mon[30223]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:11.739 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:48:11.740 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"06f23456-e9d4-11ee-95cd-87774f69a715","modified":"2024-03-24T11:46:29.205766Z","created":"2024-03-24T11:46:29.205766Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:48:11.741 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-24T11:48:11.856 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: load: jerasure load: lrc 2024-03-24T11:48:12.133 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:48:12.133 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Git sha 0 2024-03-24T11:48:12.133 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:48:12.133 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: DB SUMMARY 2024-03-24T11:48:12.133 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: DB Session ID: M7VBM3SBP69WLPBT6OY1 2024-03-24T11:48:12.133 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:48:12.133 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:48:12.134 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.env: 0x5619d12d3ae0 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.info_log: 0x5619d252d920 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.statistics: (nil) 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.use_fsync: 0 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:48:12.135 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.db_log_dir: 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.wal_dir: 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:48:12.136 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.write_buffer_manager: 0x5619d2501400 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:48:12.137 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.unordered_write: 0 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:48:12.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.row_cache: None 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.wal_filter: None 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.wal_compression: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:48:12.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:48:12.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:48:12.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:48:12.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:48:12.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:48:12.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:48:12.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:48:12.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:48:12.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:48:12.141 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:48:12.141 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_open_files: -1 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:48:12.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Compression algorithms supported: 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: kZSTD supported: 0 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:48:12.143 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.merge_operator: 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_filter: None 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5619d252da20) 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:48:12.144 INFO:journalctl@ceph.mon.b.smithi193.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: index_type: 0 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: data_block_index_type: 0 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: index_shortening: 1 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: checksum: 4 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: no_block_cache: 0 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: block_cache: 0x5619d2531090 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: block_cache_options: 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: capacity : 536870912 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: num_shard_bits : 4 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: strict_capacity_limit : 0 2024-03-24T11:48:12.145 INFO:journalctl@ceph.mon.b.smithi193.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: block_cache_compressed: (nil) 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: persistent_cache: (nil) 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: block_size: 4096 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: block_size_deviation: 10 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: block_restart_interval: 16 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: index_block_restart_interval: 1 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: metadata_block_size: 4096 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: partition_filters: 0 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: use_delta_encoding: 1 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: filter_policy: bloomfilter 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: whole_key_filtering: 1 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: verify_compression: 0 2024-03-24T11:48:12.146 INFO:journalctl@ceph.mon.b.smithi193.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout: format_version: 5 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout: enable_index_compression: 1 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout: block_align: 0 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout: max_auto_readahead_size: 262144 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout: prepopulate_block_cache: 0 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression: NoCompression 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:48:12.147 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.num_levels: 7 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:48:12.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:48:12.149 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:48:12.150 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:48:12.151 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.ttl: 2592000 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:48:12.152 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:48:12.153 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:48:12.154 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:48:12.154 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T11:48:12.154 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a3c4cef4-0b0e-45f5-84de-f72b98253ea5 2024-03-24T11:48:12.154 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280891857863, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T11:48:12.154 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T11:48:12.154 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280891858274, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280891, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a3c4cef4-0b0e-45f5-84de-f72b98253ea5", "db_session_id": "M7VBM3SBP69WLPBT6OY1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:48:12.154 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280891858354, "job": 1, "event": "recovery_finished"} 2024-03-24T11:48:12.155 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T11:48:12.155 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:48:12.155 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5619d2622000 2024-03-24T11:48:12.155 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: DB pointer 0x5619d260c000 2024-03-24T11:48:12.155 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:48:12.155 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:48:12.155 INFO:journalctl@ceph.mon.b.smithi193.stdout: ** DB Stats ** 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: ** Compaction Stats [default] ** 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: L0 1/0 1.55 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:48:12.156 INFO:journalctl@ceph.mon.b.smithi193.stdout: Sum 1/0 1.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: ** Compaction Stats [default] ** 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:48:12.157 INFO:journalctl@ceph.mon.b.smithi193.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: Block cache BinnedLRUCache@0x5619d2531090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: using public_addrv v1:172.21.15.193:6789/0 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: starting mon.b rank -1 at public addrs v1:172.21.15.193:6789/0 at bind addrs v1:172.21.15.193:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:48:12.158 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(???) e0 preinit fsid 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:48:12.159 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).mds e1 new map 2024-03-24T11:48:12.159 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-24T11:48:12.159 INFO:journalctl@ceph.mon.b.smithi193.stdout: e1 2024-03-24T11:48:12.159 INFO:journalctl@ceph.mon.b.smithi193.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:48:12.159 INFO:journalctl@ceph.mon.b.smithi193.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:48:12.159 INFO:journalctl@ceph.mon.b.smithi193.stdout: legacy client fscid: -1 2024-03-24T11:48:12.159 INFO:journalctl@ceph.mon.b.smithi193.stdout: 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout: No filesystems configured 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.160 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: Added host smithi193 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='client.14180 v1:172.21.15.64:0/1837876198' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1398433670' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1398433670' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:12.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:48:12.162 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='client.14184 v1:172.21.15.193:0/1332431082' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:[v1:172.21.15.64:6789]=a;smithi064:[v1:172.21.15.64:6790]=c;smithi193:[v1:172.21.15.193:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: Saving service mon spec with placement smithi064:[v1:172.21.15.64:6789]=a;smithi064:[v1:172.21.15.64:6790]=c;smithi193:[v1:172.21.15.193:6789]=b;count:3 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: Deploying daemon mon.b on smithi193 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.163 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/1700071790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:12.164 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 ceph-mon[33593]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T11:48:12.164 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 bash[33541]: b9bde26803e0620d57cac292d3fee13e72c40e6a9cbcb079cedb37fd91aba3f2 2024-03-24T11:48:12.164 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:11 smithi193 systemd[1]: Started Ceph mon.b for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:48:12.233 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:11 smithi064 ceph-mon[30223]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.233 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:11 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/1700071790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:13.510 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:48:13.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:48:13.765 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:48:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:48:17.225 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"06f23456-e9d4-11ee-95cd-87774f69a715","modified":"2024-03-24T11:48:11.878370Z","created":"2024-03-24T11:46:29.205766Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:48:17.226 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-03-24T11:48:17.227 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:48:17.227 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: mon.a calling monitor election 2024-03-24T11:48:17.227 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.227 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:17.227 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.227 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.227 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: mon.b calling monitor election 2024-03-24T11:48:17.227 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: monmap e2: 2 mons at {a=v1:172.21.15.64:6789/0,b=v1:172.21.15.193:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: fsmap 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: mgrmap e13: y(active, since 40s) 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: overall HEALTH_OK 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:17.228 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: mon.a calling monitor election 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: mon.b calling monitor election 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:48:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: monmap e2: 2 mons at {a=v1:172.21.15.64:6789/0,b=v1:172.21.15.193:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: fsmap 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: mgrmap e13: y(active, since 40s) 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: overall HEALTH_OK 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:17.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:17 smithi193 ceph-mon[33593]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:17 smithi193 ceph-mon[33593]: Deploying daemon mon.c on smithi064 2024-03-24T11:48:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/1447635715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:17 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:18.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:17 smithi064 ceph-mon[30223]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:18.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:17 smithi064 ceph-mon[30223]: Deploying daemon mon.c on smithi064 2024-03-24T11:48:18.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/1447635715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:18.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:17 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:18.814 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:48:18.815 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:48:19.059 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:48:20.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:19 smithi193 ceph-mon[33593]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:19 smithi064 ceph-mon[30223]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 podman[42204]: 2024-03-24 11:48:20.12083885 +0000 UTC m=+0.018262257 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:48:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 podman[42204]: 2024-03-24 11:48:20.398263995 +0000 UTC m=+0.295687402 container init 1616be2bab01d62ce834526346f1820d4353d8f61be752ff6b38e678b3992e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T11:48:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 podman[42204]: 2024-03-24 11:48:20.403761035 +0000 UTC m=+0.301184434 container start 1616be2bab01d62ce834526346f1820d4353d8f61be752ff6b38e678b3992e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T11:48:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:48:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:48:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pidfile_write: ignore empty --pid-file 2024-03-24T11:48:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: load: jerasure load: lrc 2024-03-24T11:48:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:48:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Git sha 0 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: DB SUMMARY 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: DB Session ID: 36AFEDXFC5BPT8L0E3DU 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:48:20.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:48:20.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:48:20.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.env: 0x55d597a28ae0 2024-03-24T11:48:20.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:48:20.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.info_log: 0x55d598af18e0 2024-03-24T11:48:20.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.statistics: (nil) 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.use_fsync: 0 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:48:20.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.db_log_dir: 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.wal_dir: 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:48:20.808 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.write_buffer_manager: 0x55d598ac5400 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.unordered_write: 0 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:48:20.809 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.row_cache: None 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.wal_filter: None 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.wal_compression: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:48:20.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:48:20.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_open_files: -1 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Compression algorithms supported: 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: kZSTD supported: 0 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:48:20.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.merge_operator: 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_filter: None 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:48:20.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d598af19e0) 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_type: 0 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_index_type: 0 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_shortening: 1 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: checksum: 4 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: no_block_cache: 0 2024-03-24T11:48:20.814 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache: 0x55d598af5090 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_options: 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: capacity : 536870912 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_shard_bits : 4 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: strict_capacity_limit : 0 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_compressed: (nil) 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: persistent_cache: (nil) 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size: 4096 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size_deviation: 10 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_restart_interval: 16 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_block_restart_interval: 1 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: metadata_block_size: 4096 2024-03-24T11:48:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout: partition_filters: 0 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: use_delta_encoding: 1 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: filter_policy: bloomfilter 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: whole_key_filtering: 1 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: verify_compression: 0 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: format_version: 5 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_index_compression: 1 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_align: 0 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: max_auto_readahead_size: 262144 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: prepopulate_block_cache: 0 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression: NoCompression 2024-03-24T11:48:20.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.num_levels: 7 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:48:20.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:48:20.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:48:20.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:48:20.820 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.ttl: 2592000 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:48:20.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d0c024b7-0541-40ce-834c-8ff4f081d194 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280900433184, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280900433571, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280900, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d0c024b7-0541-40ce-834c-8ff4f081d194", "db_session_id": "36AFEDXFC5BPT8L0E3DU", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280900433652, "job": 1, "event": "recovery_finished"} 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d598be6000 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: DB pointer 0x55d598bd0000 2024-03-24T11:48:20.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** DB Stats ** 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:48:20.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: L0 1/0 1.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Sum 1/0 1.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:48:20.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: Block cache BinnedLRUCache@0x55d598af5090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: using public_addrv v1:172.21.15.64:6790/0 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: starting mon.c rank -1 at public addrs v1:172.21.15.64:6790/0 at bind addrs v1:172.21.15.64:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(???) e0 preinit fsid 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T11:48:20.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: e1 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: legacy client fscid: -1 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: No filesystems configured 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:48:20.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: Added host smithi193 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='client.14180 v1:172.21.15.64:0/1837876198' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1398433670' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1398433670' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:48:20.827 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.828 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='client.14184 v1:172.21.15.193:0/1332431082' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:[v1:172.21.15.64:6789]=a;smithi064:[v1:172.21.15.64:6790]=c;smithi193:[v1:172.21.15.193:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: Saving service mon spec with placement smithi064:[v1:172.21.15.64:6789]=a;smithi064:[v1:172.21.15.64:6790]=c;smithi193:[v1:172.21.15.193:6789]=b;count:3 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: Deploying daemon mon.b on smithi193 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/1700071790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.a calling monitor election 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:20.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.b calling monitor election 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: monmap e2: 2 mons at {a=v1:172.21.15.64:6789/0,b=v1:172.21.15.193:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: fsmap 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mgrmap e13: y(active, since 40s) 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: overall HEALTH_OK 2024-03-24T11:48:20.830 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: Deploying daemon mon.c on smithi064 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/1447635715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 ceph-mon[42243]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 bash[42204]: 1616be2bab01d62ce834526346f1820d4353d8f61be752ff6b38e678b3992e13 2024-03-24T11:48:20.831 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:20 smithi064 systemd[1]: Started Ceph mon.c for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:48:25.797 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T11:48:25.797 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"06f23456-e9d4-11ee-95cd-87774f69a715","modified":"2024-03-24T11:48:20.452713Z","created":"2024-03-24T11:46:29.205766Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-24T11:48:25.799 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-03-24T11:48:25.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: mon.a calling monitor election 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: mon.b calling monitor election 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: mon.c calling monitor election 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: monmap e3: 3 mons at {a=v1:172.21.15.64:6789/0,b=v1:172.21.15.193:6789/0,c=v1:172.21.15.64:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: fsmap 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: mgrmap e13: y(active, since 49s) 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: overall HEALTH_OK 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: mon.a calling monitor election 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: mon.b calling monitor election 2024-03-24T11:48:25.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: mon.c calling monitor election 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: monmap e3: 3 mons at {a=v1:172.21.15.64:6789/0,b=v1:172.21.15.193:6789/0,c=v1:172.21.15.64:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: fsmap 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: mgrmap e13: y(active, since 49s) 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: overall HEALTH_OK 2024-03-24T11:48:25.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:25.806 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:25.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: mon.a calling monitor election 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: mon.b calling monitor election 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: mon.c calling monitor election 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:25.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:48:25.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: monmap e3: 3 mons at {a=v1:172.21.15.64:6789/0,b=v1:172.21.15.193:6789/0,c=v1:172.21.15.64:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:48:25.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: fsmap 2024-03-24T11:48:25.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:48:25.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: mgrmap e13: y(active, since 49s) 2024-03-24T11:48:25.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: overall HEALTH_OK 2024-03-24T11:48:25.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:25.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.358 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T11:48:26.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T11:48:26.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:26.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/1603140804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:26.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.669 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:26.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:26.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/1603140804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:26.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:26 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:26.718 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:26 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.718 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:26 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.718 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:26 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:26.718 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:26 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/1603140804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:48:26.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:26 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:26.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:26 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:48:27.038 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:48:27.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[30223]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[42243]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:27.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:27.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:27 smithi193 ceph-mon[33593]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:27.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:27.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:27.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:28.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-24T11:48:28.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:48:28.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:28.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:28.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:28.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[30223]: Reconfiguring daemon mon.a on smithi064 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:48:28.793 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:28.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:28.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:28.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:28 smithi064 ceph-mon[42243]: Reconfiguring daemon mon.a on smithi064 2024-03-24T11:48:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:29.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:48:29.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:29.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:29.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:29.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:28 smithi193 ceph-mon[33593]: Reconfiguring daemon mon.a on smithi064 2024-03-24T11:48:29.230 INFO:teuthology.orchestra.run.smithi064.stdout:# minimal ceph.conf for 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:48:29.230 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-03-24T11:48:29.230 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:48:29.230 INFO:teuthology.orchestra.run.smithi064.stdout: mon_host = 172.21.15.64:6789/0 172.21.15.193:6789/0 v1:172.21.15.64:6790/0 2024-03-24T11:48:29.996 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:29 smithi064 ceph-mon[30223]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:29.996 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:29 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3428697891' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:29.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:29 smithi064 ceph-mon[42243]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:29.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:29 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3428697891' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:30.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:29 smithi193 ceph-mon[33593]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:30.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:29 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3428697891' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:30.293 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T11:48:30.293 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:48:30.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:48:30.322 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:48:30.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:48:30.391 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:48:30.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:48:30.426 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:48:30.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:48:30.497 INFO:tasks.cephadm:Adding mgr.y on smithi064 2024-03-24T11:48:30.497 INFO:tasks.cephadm:Adding mgr.x on smithi193 2024-03-24T11:48:30.498 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi064=y;smithi193=x' 2024-03-24T11:48:30.768 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:48:31.145 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:31.210 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:31.211 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:30 smithi193 ceph-mon[33593]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:48:31.211 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:31.211 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:31.211 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:31.211 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:30 smithi193 ceph-mon[33593]: Reconfiguring daemon mon.c on smithi064 2024-03-24T11:48:31.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:31.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:31.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[30223]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:48:31.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:31.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:31.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:31.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[30223]: Reconfiguring daemon mon.c on smithi064 2024-03-24T11:48:31.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:31.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:31.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[42243]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:48:31.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:31.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:31.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:31.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:30 smithi064 ceph-mon[42243]: Reconfiguring daemon mon.c on smithi064 2024-03-24T11:48:32.115 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:31 smithi193 ceph-mon[33593]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:32.115 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:32.115 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:32.115 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:32.115 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:32.115 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:32.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[42243]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[30223]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:48:32.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:48:32.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:32.572 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-03-24T11:48:33.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:32 smithi064 ceph-mon[42243]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:48:33.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:32 smithi064 ceph-mon[42243]: Reconfiguring daemon mon.b on smithi193 2024-03-24T11:48:33.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:32 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:33.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:32 smithi064 ceph-mon[30223]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:48:33.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:32 smithi064 ceph-mon[30223]: Reconfiguring daemon mon.b on smithi193 2024-03-24T11:48:33.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:32 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:33.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:32 smithi193 ceph-mon[33593]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:48:33.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:32 smithi193 ceph-mon[33593]: Reconfiguring daemon mon.b on smithi193 2024-03-24T11:48:33.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:32 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:33.623 DEBUG:teuthology.orchestra.run.smithi193:mgr.x> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T11:48:33.626 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T11:48:33.626 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:48:33.626 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:48:33.645 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:48:33.645 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-03-24T11:48:33.703 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:48:33.704 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:48:33.704 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T11:48:33.704 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:48:33.704 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:48:33.704 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-24 11:48:19.945833423 +0000 2024-03-24T11:48:33.704 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-24 11:42:40.517239785 +0000 2024-03-24T11:48:33.704 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-24 11:42:40.517239785 +0000 2024-03-24T11:48:33.704 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-24 11:42:40.517239785 +0000 2024-03-24T11:48:33.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:48:33.775 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-24T11:48:33.775 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-24T11:48:33.775 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000281378 s, 1.8 MB/s 2024-03-24T11:48:33.776 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:48:33.835 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-24 11:48:19.945833423 +0000 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-24 11:42:40.881232513 +0000 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-24 11:42:40.881232513 +0000 2024-03-24T11:48:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-24 11:42:40.881232513 +0000 2024-03-24T11:48:33.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:48:33.960 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-24T11:48:33.960 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-24T11:48:33.960 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000208457 s, 2.5 MB/s 2024-03-24T11:48:33.961 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:48:34.018 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-03-24T11:48:34.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:33 smithi064 ceph-mon[42243]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:34.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:33 smithi064 ceph-mon[42243]: from='client.24103 v1:172.21.15.193:0/4035812555' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi193=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:34.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:33 smithi064 ceph-mon[42243]: Saving service mgr spec with placement smithi064=y;smithi193=x;count:2 2024-03-24T11:48:34.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:33 smithi064 ceph-mon[30223]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:34.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:33 smithi064 ceph-mon[30223]: from='client.24103 v1:172.21.15.193:0/4035812555' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi193=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:34.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:33 smithi064 ceph-mon[30223]: Saving service mgr spec with placement smithi064=y;smithi193=x;count:2 2024-03-24T11:48:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:48:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:48:34.068 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-24T11:48:34.068 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:48:34.068 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:48:34.069 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-24 11:48:19.946833404 +0000 2024-03-24T11:48:34.069 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-24 11:42:41.213225881 +0000 2024-03-24T11:48:34.069 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-24 11:42:41.214225861 +0000 2024-03-24T11:48:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-24 11:42:41.213225881 +0000 2024-03-24T11:48:34.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:48:34.134 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-24T11:48:34.135 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-24T11:48:34.135 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000207064 s, 2.5 MB/s 2024-03-24T11:48:34.136 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:48:34.192 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-03-24T11:48:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:48:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:48:34.247 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T11:48:34.247 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:48:34.247 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:48:34.247 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-24 11:48:19.946833404 +0000 2024-03-24T11:48:34.248 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-24 11:42:41.546219226 +0000 2024-03-24T11:48:34.248 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-24 11:42:41.546219226 +0000 2024-03-24T11:48:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-24 11:42:41.546219226 +0000 2024-03-24T11:48:34.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:48:34.313 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-24T11:48:34.313 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-24T11:48:34.313 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000228288 s, 2.2 MB/s 2024-03-24T11:48:34.314 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:48:34.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:33 smithi193 ceph-mon[33593]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:34.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:33 smithi193 ceph-mon[33593]: from='client.24103 v1:172.21.15.193:0/4035812555' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi193=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:34.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:33 smithi193 ceph-mon[33593]: Saving service mgr spec with placement smithi064=y;smithi193=x;count:2 2024-03-24T11:48:34.369 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:48:34.370 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:48:34.386 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:48:34.386 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-24 11:48:11.017369273 +0000 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-24 11:42:38.050066919 +0000 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-24 11:42:38.050066919 +0000 2024-03-24T11:48:34.442 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-24 11:42:38.050066919 +0000 2024-03-24T11:48:34.443 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:48:34.510 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-24T11:48:34.510 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-24T11:48:34.510 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000294939 s, 1.7 MB/s 2024-03-24T11:48:34.511 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:48:34.567 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-24 11:48:11.018369258 +0000 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-24 11:42:38.378061687 +0000 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-24 11:42:38.378061687 +0000 2024-03-24T11:48:34.622 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-24 11:42:38.378061687 +0000 2024-03-24T11:48:34.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:48:34.688 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-24T11:48:34.688 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-24T11:48:34.688 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000215973 s, 2.4 MB/s 2024-03-24T11:48:34.689 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:48:34.744 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-03-24T11:48:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:48:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:48:34.799 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-24T11:48:34.799 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:48:34.800 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:48:34.800 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-24 11:48:11.018369258 +0000 2024-03-24T11:48:34.800 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-24 11:42:38.724056167 +0000 2024-03-24T11:48:34.800 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-24 11:42:38.724056167 +0000 2024-03-24T11:48:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-24 11:42:38.724056167 +0000 2024-03-24T11:48:34.800 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:48:34.865 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-24T11:48:34.866 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-24T11:48:34.866 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000189205 s, 2.7 MB/s 2024-03-24T11:48:34.867 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:48:34.923 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-03-24T11:48:34.978 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:48:34.978 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:48:34.978 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T11:48:34.978 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:48:34.978 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:48:34.979 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-24 11:48:11.018369258 +0000 2024-03-24T11:48:34.979 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-24 11:42:39.081050473 +0000 2024-03-24T11:48:34.979 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-24 11:42:39.081050473 +0000 2024-03-24T11:48:34.979 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-24 11:42:39.081050473 +0000 2024-03-24T11:48:34.979 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:48:35.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:35.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.043 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:35.043 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.043 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:35.043 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:48:35.043 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:35.043 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.043 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:34 smithi193 ceph-mon[33593]: Deploying daemon mgr.x on smithi193 2024-03-24T11:48:35.044 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-24T11:48:35.044 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-24T11:48:35.044 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000183571 s, 2.8 MB/s 2024-03-24T11:48:35.045 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:48:35.101 INFO:tasks.cephadm:Deploying osd.0 on smithi064 with /dev/vg_nvme/lv_4... 2024-03-24T11:48:35.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:48:35.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[42243]: Deploying daemon mgr.x on smithi193 2024-03-24T11:48:35.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:34 smithi064 ceph-mon[30223]: Deploying daemon mgr.x on smithi193 2024-03-24T11:48:35.356 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:48:36.011 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:36 smithi193 systemd[1]: Starting Ceph mgr.x for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:48:36.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:36 smithi064 ceph-mon[30223]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:36.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:36 smithi064 ceph-mon[42243]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:36.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:36 smithi193 ceph-mon[33593]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:36.336 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:36 smithi193 podman[35658]: 2024-03-24 11:48:36.128958022 +0000 UTC m=+0.017393510 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:48:36.337 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:36 smithi193 podman[35658]: 2024-03-24 11:48:36.231621124 +0000 UTC m=+0.120056658 container create 2e068a588fac7a63c588868dbe3bdf974a8f98e24f194cc5ed690fae4cb95f9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T11:48:36.837 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:36 smithi193 podman[35658]: 2024-03-24 11:48:36.442388525 +0000 UTC m=+0.330824019 container init 2e068a588fac7a63c588868dbe3bdf974a8f98e24f194cc5ed690fae4cb95f9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b) 2024-03-24T11:48:36.837 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:36 smithi193 podman[35658]: 2024-03-24 11:48:36.447873423 +0000 UTC m=+0.336308918 container start 2e068a588fac7a63c588868dbe3bdf974a8f98e24f194cc5ed690fae4cb95f9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-24T11:48:36.837 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:36 smithi193 bash[35658]: 2e068a588fac7a63c588868dbe3bdf974a8f98e24f194cc5ed690fae4cb95f9b 2024-03-24T11:48:36.837 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:36 smithi193 systemd[1]: Started Ceph mgr.x for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:48:37.337 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:37.008+0000 7f3bdcbc5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:48:37.337 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:37.103+0000 7f3bdcbc5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:48:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:48:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:48:37.575 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-03-24T11:48:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-03-24T11:48:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012272 s, 854 MB/s 2024-03-24T11:48:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[30223]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[42243]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.875 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:37.887 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:37.792+0000 7f3bdcbc5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:48:37.888 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-mon[33593]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:37.888 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.888 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.888 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.888 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:37.888 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:38.216 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:37.885+0000 7f3bdcbc5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:48:38.216 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:38.031+0000 7f3bdcbc5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:48:38.216 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:38.129+0000 7f3bdcbc5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:48:38.216 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:38.214+0000 7f3bdcbc5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:48:38.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_4 2024-03-24T11:48:38.488 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:48:38.757 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:38.473+0000 7f3bdcbc5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:48:38.757 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:38.580+0000 7f3bdcbc5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:48:38.758 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:38.664+0000 7f3bdcbc5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:48:38.758 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:38.755+0000 7f3bdcbc5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:48:39.066 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:38.930+0000 7f3bdcbc5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[30223]: Reconfiguring daemon mgr.y on smithi064 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:39.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:39.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:39 smithi064 ceph-mon[42243]: Reconfiguring daemon mgr.y on smithi064 2024-03-24T11:48:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:39.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:39.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-mon[33593]: Reconfiguring daemon mgr.y on smithi064 2024-03-24T11:48:40.336 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:39.883+0000 7f3bdcbc5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:48:40.337 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:40.010+0000 7f3bdcbc5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:48:40.619 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:40.339+0000 7f3bdcbc5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:48:40.619 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:40.427+0000 7f3bdcbc5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:48:40.619 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:40.617+0000 7f3bdcbc5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:48:41.075 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:40.797+0000 7f3bdcbc5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:48:41.337 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:41.073+0000 7f3bdcbc5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:48:41.836 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:41.336+0000 7f3bdcbc5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:48:41.836 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:41.567+0000 7f3bdcbc5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:41.910 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:41.910 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:41.911 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:41.912 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:42.772 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:42 smithi064 ceph-mon[30223]: from='client.14220 v1:172.21.15.64:0/1988557555' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:42.772 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:42 smithi064 ceph-mon[42243]: from='client.14220 v1:172.21.15.64:0/1988557555' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:42 smithi193 ceph-mon[33593]: from='client.14220 v1:172.21.15.64:0/1988557555' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:42.837 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:42.348+0000 7f3bdcbc5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:48:43.217 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:42.910+0000 7f3bdcbc5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:48:43.218 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:43.006+0000 7f3bdcbc5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:48:43.218 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:48:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:48:43.215+0000 7f3bdcbc5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:48:44.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[30223]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:44.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[30223]: Standby manager daemon x started 2024-03-24T11:48:44.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[30223]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:48:44.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[30223]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:44.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[30223]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:48:44.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[30223]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[42243]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:44.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[42243]: Standby manager daemon x started 2024-03-24T11:48:44.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[42243]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:48:44.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[42243]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:44.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[42243]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:48:44.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:43 smithi064 ceph-mon[42243]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:43 smithi193 ceph-mon[33593]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:43 smithi193 ceph-mon[33593]: Standby manager daemon x started 2024-03-24T11:48:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:43 smithi193 ceph-mon[33593]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:48:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:43 smithi193 ceph-mon[33593]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:44.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:43 smithi193 ceph-mon[33593]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:48:44.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:43 smithi193 ceph-mon[33593]: from='mgr.? v1:172.21.15.193:0/1715601974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:45.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[30223]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4271012828' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]: dispatch 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[30223]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]: dispatch 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[30223]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]': finished 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[30223]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1939194708' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[42243]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:48:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4271012828' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]: dispatch 2024-03-24T11:48:45.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[42243]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]: dispatch 2024-03-24T11:48:45.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[42243]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]': finished 2024-03-24T11:48:45.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[42243]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:48:45.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:45.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1939194708' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:44 smithi193 ceph-mon[33593]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-24T11:48:45.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:44 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:48:45.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4271012828' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]: dispatch 2024-03-24T11:48:45.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:44 smithi193 ceph-mon[33593]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]: dispatch 2024-03-24T11:48:45.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:44 smithi193 ceph-mon[33593]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d"}]': finished 2024-03-24T11:48:45.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:44 smithi193 ceph-mon[33593]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:48:45.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:44 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:45.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1939194708' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:45 smithi064 ceph-mon[30223]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:46.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:45 smithi064 ceph-mon[42243]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:45 smithi193 ceph-mon[33593]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:48.184 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:47 smithi064 ceph-mon[30223]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:48.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:47 smithi064 ceph-mon[42243]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:47 smithi193 ceph-mon[33593]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:50.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:49 smithi064 ceph-mon[30223]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:50.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:49 smithi064 ceph-mon[42243]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:50.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:49 smithi193 ceph-mon[33593]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:51.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:50 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:48:51.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:50 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:50 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:48:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:50 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:50 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:48:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:50 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:52.151 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:51 smithi064 ceph-mon[30223]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:52.151 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:51 smithi064 ceph-mon[30223]: Deploying daemon osd.0 on smithi064 2024-03-24T11:48:52.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:51 smithi064 ceph-mon[42243]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:52.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:51 smithi064 ceph-mon[42243]: Deploying daemon osd.0 on smithi064 2024-03-24T11:48:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:51 smithi193 ceph-mon[33593]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:52.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:51 smithi193 ceph-mon[33593]: Deploying daemon osd.0 on smithi064 2024-03-24T11:48:54.209 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:53 smithi064 ceph-mon[30223]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:54.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:53 smithi064 ceph-mon[42243]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:53 smithi193 ceph-mon[33593]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:55 smithi064 ceph-mon[30223]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:55 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:56.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:55 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:56.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:55 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:55 smithi064 ceph-mon[42243]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:55 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:55 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:55 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:55 smithi193 ceph-mon[33593]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:55 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:55 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:55 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:57 smithi064 ceph-mon[30223]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:57 smithi064 ceph-mon[42243]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:57 smithi193 ceph-mon[33593]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:59.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[30223]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:59.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:59.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:59.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[30223]: from='osd.0 v1:172.21.15.64:6801/2675992571' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:48:59.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[30223]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:48:59.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[42243]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:59.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:59.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:59.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[42243]: from='osd.0 v1:172.21.15.64:6801/2675992571' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:48:59.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:48:59 smithi064 ceph-mon[42243]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:48:59.590 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 0 on host 'smithi064' 2024-03-24T11:48:59.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:59 smithi193 ceph-mon[33593]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:59 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:59 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:59 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:59 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:59 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:48:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:59 smithi193 ceph-mon[33593]: from='osd.0 v1:172.21.15.64:6801/2675992571' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:48:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:48:59 smithi193 ceph-mon[33593]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:49:00.218 DEBUG:teuthology.orchestra.run.smithi064:osd.0> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T11:49:00.220 INFO:tasks.cephadm:Deploying osd.1 on smithi064 with /dev/vg_nvme/lv_3... 2024-03-24T11:49:00.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:49:00.721 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[42243]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:49:00.722 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[42243]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:49:00.722 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:00.722 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[42243]: from='osd.0 v1:172.21.15.64:6801/2675992571' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:00.722 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[42243]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:00.722 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:00.722 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:00.722 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:00.722 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[30223]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:49:00.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[30223]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:49:00.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:00.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[30223]: from='osd.0 v1:172.21.15.64:6801/2675992571' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:00.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[30223]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:00.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:00.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:00.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:00.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:00 smithi193 ceph-mon[33593]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:49:00.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:00 smithi193 ceph-mon[33593]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:49:00.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:00 smithi193 ceph-mon[33593]: from='osd.0 v1:172.21.15.64:6801/2675992571' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:00 smithi193 ceph-mon[33593]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:01.052 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0[46656]: 2024-03-24T11:49:00.950+0000 7ff3ca993700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T11:49:01.053 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:00 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0[46656]: 2024-03-24T11:49:00.953+0000 7ff3c277e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:49:01.065 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[42243]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[42243]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[42243]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[42243]: from='osd.0 ' entity='osd.0' 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[30223]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[30223]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:01.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[30223]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:49:01.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:01.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:01.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:01 smithi064 ceph-mon[30223]: from='osd.0 ' entity='osd.0' 2024-03-24T11:49:01.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:01 smithi193 ceph-mon[33593]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:49:01.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:01 smithi193 ceph-mon[33593]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:01.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:01 smithi193 ceph-mon[33593]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:49:01.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:01.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:01.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:01 smithi193 ceph-mon[33593]: from='osd.0 ' entity='osd.0' 2024-03-24T11:49:02.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[30223]: osd.0 v1:172.21.15.64:6801/2675992571 boot 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[30223]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[42243]: osd.0 v1:172.21.15.64:6801/2675992571 boot 2024-03-24T11:49:02.635 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[42243]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:49:02.636 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:02 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:02 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:49:02.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:02 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:49:02.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:02 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:02.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:02 smithi193 ceph-mon[33593]: osd.0 v1:172.21.15.64:6801/2675992571 boot 2024-03-24T11:49:02.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:02 smithi193 ceph-mon[33593]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:49:02.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:02 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:04.127 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:49:04.127 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:49:04.127 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-03-24T11:49:04.127 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-03-24T11:49:04.127 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121983 s, 860 MB/s 2024-03-24T11:49:04.127 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-03-24T11:49:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:03 smithi064 ceph-mon[30223]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:49:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:03 smithi064 ceph-mon[30223]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:49:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:03 smithi064 ceph-mon[42243]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:49:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:03 smithi064 ceph-mon[42243]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:49:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:03 smithi193 ceph-mon[33593]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:49:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:03 smithi193 ceph-mon[33593]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:49:04.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_3 2024-03-24T11:49:04.996 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: Detected new or changed devices on smithi064 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi064 to 16064M 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:05.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: Detected new or changed devices on smithi064 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi064 to 16064M 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:05.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: Detected new or changed devices on smithi064 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi064 to 16064M 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:07.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[30223]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:07.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[30223]: from='client.34139 v1:172.21.15.64:0/3241172000' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:07.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:07.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:07.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:07.618 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[42243]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:07.619 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[42243]: from='client.34139 v1:172.21.15.64:0/3241172000' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:07.619 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:07.619 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:07.619 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:07 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:07.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:07 smithi193 ceph-mon[33593]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:07.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:07 smithi193 ceph-mon[33593]: from='client.34139 v1:172.21.15.64:0/3241172000' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:07.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:07 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:07.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:07 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:07.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:07 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:09.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:09 smithi193 ceph-mon[33593]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:09.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:09 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3379850927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]: dispatch 2024-03-24T11:49:09.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:09 smithi193 ceph-mon[33593]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]: dispatch 2024-03-24T11:49:09.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:09 smithi193 ceph-mon[33593]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]': finished 2024-03-24T11:49:09.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:09 smithi193 ceph-mon[33593]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:49:09.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:09 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:10.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[30223]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3379850927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]: dispatch 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[30223]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]: dispatch 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[30223]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]': finished 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[30223]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[42243]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3379850927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]: dispatch 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[42243]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]: dispatch 2024-03-24T11:49:10.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[42243]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9"}]': finished 2024-03-24T11:49:10.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[42243]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:49:10.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:09 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:10.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1661643916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:49:11.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1661643916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:49:11.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1661643916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:49:11.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:11 smithi193 ceph-mon[33593]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:11 smithi064 ceph-mon[30223]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:12.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:11 smithi064 ceph-mon[42243]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:13.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:13 smithi193 ceph-mon[33593]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:13.905 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:13 smithi064 ceph-mon[30223]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:13.905 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:13 smithi064 ceph-mon[42243]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:15.666 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:15 smithi064 ceph-mon[30223]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:15.666 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:15 smithi064 ceph-mon[42243]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:15.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:15 smithi193 ceph-mon[33593]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:16.927 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:16 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:49:16.928 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:16 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:16.928 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:16 smithi064 ceph-mon[42243]: Deploying daemon osd.1 on smithi064 2024-03-24T11:49:16.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:49:16.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:16.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:16 smithi064 ceph-mon[30223]: Deploying daemon osd.1 on smithi064 2024-03-24T11:49:17.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:49:17.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:17.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:16 smithi193 ceph-mon[33593]: Deploying daemon osd.1 on smithi064 2024-03-24T11:49:17.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:17 smithi064 ceph-mon[30223]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:17.681 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:17 smithi064 ceph-mon[42243]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:18.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:17 smithi193 ceph-mon[33593]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:19.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:19 smithi064 ceph-mon[30223]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:19.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:19 smithi064 ceph-mon[42243]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:19.888 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:19 smithi193 ceph-mon[33593]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:21 smithi064 ceph-mon[30223]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:21.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:21 smithi064 ceph-mon[42243]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:21 smithi193 ceph-mon[33593]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:22.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:22 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:22.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:22 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:22.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:22 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:22.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:22 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:22.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:22 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:22.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:22 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:23.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:22 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:23.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:22 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:23.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:22 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:23.916 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:23 smithi064 ceph-mon[30223]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:23.917 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:23 smithi064 ceph-mon[42243]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:24.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:23 smithi193 ceph-mon[33593]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:25.437 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 1 on host 'smithi064' 2024-03-24T11:49:25.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='osd.1 v1:172.21.15.64:6805/841667271' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='osd.1 v1:172.21.15.64:6805/841667271' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:25.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:26.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:26.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='osd.1 v1:172.21.15.64:6805/841667271' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:49:26.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:49:26.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:26.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:26.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:26.204 DEBUG:teuthology.orchestra.run.smithi064:osd.1> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T11:49:26.207 INFO:tasks.cephadm:Deploying osd.2 on smithi064 with /dev/vg_nvme/lv_2... 2024-03-24T11:49:26.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:49:26.772 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[42243]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:49:26.772 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[42243]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:49:26.772 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:26.772 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[42243]: from='osd.1 v1:172.21.15.64:6805/841667271' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:26.772 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[42243]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:26.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[30223]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:49:26.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[30223]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:49:26.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:26.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[30223]: from='osd.1 v1:172.21.15.64:6805/841667271' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:26.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:26 smithi064 ceph-mon[30223]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:27.028 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:26 smithi193 ceph-mon[33593]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:49:27.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:26 smithi193 ceph-mon[33593]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:49:27.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:26 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:27.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:26 smithi193 ceph-mon[33593]: from='osd.1 v1:172.21.15.64:6805/841667271' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:27.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:26 smithi193 ceph-mon[33593]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:27.408 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1[52921]: 2024-03-24T11:49:27.134+0000 7f3954927700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T11:49:27.409 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1[52921]: 2024-03-24T11:49:27.136+0000 7f394c712700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:49:27.948 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[42243]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:27.948 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[42243]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:27.948 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[42243]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:49:27.948 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:27.948 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:27.948 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[42243]: from='osd.1 ' entity='osd.1' 2024-03-24T11:49:27.949 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:27.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[30223]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:27.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[30223]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:27.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[30223]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:49:27.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:27.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:27.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[30223]: from='osd.1 ' entity='osd.1' 2024-03-24T11:49:27.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:27 smithi193 ceph-mon[33593]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:28.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:27 smithi193 ceph-mon[33593]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:28.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:27 smithi193 ceph-mon[33593]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:49:28.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:28.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:28.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:27 smithi193 ceph-mon[33593]: from='osd.1 ' entity='osd.1' 2024-03-24T11:49:28.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:28.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:49:28.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:49:28.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[30223]: osd.1 v1:172.21.15.64:6805/841667271 boot 2024-03-24T11:49:28.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[30223]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:49:28.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:28.902 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:49:28.902 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:49:28.903 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[42243]: osd.1 v1:172.21.15.64:6805/841667271 boot 2024-03-24T11:49:28.903 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[42243]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:49:28.903 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:28 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:28 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:49:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:28 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:49:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:28 smithi193 ceph-mon[33593]: osd.1 v1:172.21.15.64:6805/841667271 boot 2024-03-24T11:49:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:28 smithi193 ceph-mon[33593]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:49:29.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:28 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:49:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:49:29.936 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-03-24T11:49:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-03-24T11:49:29.936 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149801 s, 700 MB/s 2024-03-24T11:49:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-03-24T11:49:29.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:29 smithi064 ceph-mon[30223]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:29.937 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:29 smithi064 ceph-mon[42243]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:30.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:29 smithi193 ceph-mon[33593]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:49:30.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_2 2024-03-24T11:49:30.862 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:30.985 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:49:30.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:30.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:30.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:30.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:30.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:30.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:30.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:30.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:49:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:31.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:31 smithi064 ceph-mon[30223]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:31.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:31 smithi064 ceph-mon[30223]: Detected new or changed devices on smithi064 2024-03-24T11:49:31.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:31 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi064 to 8032M 2024-03-24T11:49:31.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:31 smithi064 ceph-mon[42243]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:31.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:31 smithi064 ceph-mon[42243]: Detected new or changed devices on smithi064 2024-03-24T11:49:31.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:31 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi064 to 8032M 2024-03-24T11:49:32.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:31 smithi193 ceph-mon[33593]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:31 smithi193 ceph-mon[33593]: Detected new or changed devices on smithi064 2024-03-24T11:49:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:31 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi064 to 8032M 2024-03-24T11:49:33.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:32 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:33.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:32 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:33.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:32 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:33.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:32 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:33.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:32 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:33.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:32 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:32 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:32 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:32 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:33.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:33 smithi064 ceph-mon[30223]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:33.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:33 smithi064 ceph-mon[30223]: from='client.34166 v1:172.21.15.64:0/293300173' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:33.995 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:33 smithi064 ceph-mon[42243]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:33.995 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:33 smithi064 ceph-mon[42243]: from='client.34166 v1:172.21.15.64:0/293300173' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:33 smithi193 ceph-mon[33593]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:34.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:33 smithi193 ceph-mon[33593]: from='client.34166 v1:172.21.15.64:0/293300173' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:36.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[30223]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/780247240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]: dispatch 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[30223]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]: dispatch 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[30223]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]': finished 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[30223]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[42243]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/780247240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]: dispatch 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[42243]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]: dispatch 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[42243]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]': finished 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[42243]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:49:36.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:35 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:35 smithi193 ceph-mon[33593]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:36.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/780247240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]: dispatch 2024-03-24T11:49:36.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:35 smithi193 ceph-mon[33593]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]: dispatch 2024-03-24T11:49:36.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:35 smithi193 ceph-mon[33593]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b"}]': finished 2024-03-24T11:49:36.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:35 smithi193 ceph-mon[33593]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:49:36.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:35 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:36 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/85654831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:49:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:36 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/85654831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:49:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:36 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/85654831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:49:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:37 smithi064 ceph-mon[30223]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:37 smithi064 ceph-mon[42243]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:38.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:37 smithi193 ceph-mon[33593]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:39.994 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:39 smithi064 ceph-mon[30223]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:39.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:39 smithi064 ceph-mon[42243]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:40.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:39 smithi193 ceph-mon[33593]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:41.955 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:41 smithi064 ceph-mon[30223]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:41.956 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:41 smithi064 ceph-mon[42243]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:41 smithi193 ceph-mon[33593]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:43.039 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:42 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:49:43.039 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:42 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:43.039 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:42 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:49:43.039 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:42 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:42 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:49:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:42 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:43 smithi064 ceph-mon[30223]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:43 smithi064 ceph-mon[30223]: Deploying daemon osd.2 on smithi064 2024-03-24T11:49:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:43 smithi064 ceph-mon[42243]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:43 smithi064 ceph-mon[42243]: Deploying daemon osd.2 on smithi064 2024-03-24T11:49:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:43 smithi193 ceph-mon[33593]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:43 smithi193 ceph-mon[33593]: Deploying daemon osd.2 on smithi064 2024-03-24T11:49:46.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:45 smithi064 ceph-mon[30223]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:46.048 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:45 smithi064 ceph-mon[42243]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:46.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:45 smithi193 ceph-mon[33593]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:48.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:47 smithi064 ceph-mon[30223]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:47 smithi064 ceph-mon[42243]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:47 smithi193 ceph-mon[33593]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:49.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:48 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:48 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:48 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:49.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:48 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:49.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:48 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:49.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:48 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:49.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:48 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:49.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:48 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:49.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:48 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:50.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:49 smithi064 ceph-mon[42243]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:49 smithi064 ceph-mon[30223]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:50.256 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:49 smithi193 ceph-mon[33593]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:51.533 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 2 on host 'smithi064' 2024-03-24T11:49:52.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[30223]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:52.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[30223]: from='osd.2 v1:172.21.15.64:6809/435147390' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[30223]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[42243]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[42243]: from='osd.2 v1:172.21.15.64:6809/435147390' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[42243]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:51 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:51 smithi193 ceph-mon[33593]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:52.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:51 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:51 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:51 smithi193 ceph-mon[33593]: from='osd.2 v1:172.21.15.64:6809/435147390' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:52.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:51 smithi193 ceph-mon[33593]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:52.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:51 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:51 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:52.163 DEBUG:teuthology.orchestra.run.smithi064:osd.2> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T11:49:52.166 INFO:tasks.cephadm:Deploying osd.3 on smithi064 with /dev/vg_nvme/lv_1... 2024-03-24T11:49:52.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:49:52.466 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:53.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[30223]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:49:53.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[30223]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:49:53.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:53.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[30223]: from='osd.2 v1:172.21.15.64:6809/435147390' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:53.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[30223]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:53.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[42243]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:49:53.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[42243]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:49:53.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:53.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[42243]: from='osd.2 v1:172.21.15.64:6809/435147390' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:53.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:52 smithi064 ceph-mon[42243]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:52 smithi193 ceph-mon[33593]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:49:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:52 smithi193 ceph-mon[33593]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:49:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:52 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:52 smithi193 ceph-mon[33593]: from='osd.2 v1:172.21.15.64:6809/435147390' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:52 smithi193 ceph-mon[33593]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:49:53.483 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2[58928]: 2024-03-24T11:49:53.289+0000 7fdf2ae9d700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T11:49:53.483 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2[58928]: 2024-03-24T11:49:53.292+0000 7fdf22c88700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:49:54.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[30223]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:54.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[30223]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:54.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[30223]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:49:54.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[30223]: from='osd.2 ' entity='osd.2' 2024-03-24T11:49:54.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.018 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[42243]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:54.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[42243]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:54.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[42243]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:49:54.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[42243]: from='osd.2 ' entity='osd.2' 2024-03-24T11:49:54.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:53 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:53 smithi193 ceph-mon[33593]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:53 smithi193 ceph-mon[33593]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:49:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:53 smithi193 ceph-mon[33593]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:49:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:53 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:53 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:53 smithi193 ceph-mon[33593]: from='osd.2 ' entity='osd.2' 2024-03-24T11:49:54.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:53 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:54.551 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:49:54.552 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:49:54.552 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-03-24T11:49:54.552 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-03-24T11:49:54.552 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124898 s, 840 MB/s 2024-03-24T11:49:54.552 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-03-24T11:49:55.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[30223]: osd.2 v1:172.21.15.64:6809/435147390 boot 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[30223]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[42243]: osd.2 v1:172.21.15.64:6809/435147390 boot 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[42243]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:49:55.153 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:54 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:55.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_1 2024-03-24T11:49:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:54 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:49:55.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:54 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:49:55.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:54 smithi193 ceph-mon[33593]: osd.2 v1:172.21.15.64:6809/435147390 boot 2024-03-24T11:49:55.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:54 smithi193 ceph-mon[33593]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:49:55.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:54 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:55.868 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:56.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:55 smithi064 ceph-mon[30223]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:55 smithi064 ceph-mon[42243]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:55 smithi193 ceph-mon[33593]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:57.141 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:56 smithi064 ceph-mon[30223]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:49:57.141 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:56 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:57.142 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:56 smithi064 ceph-mon[42243]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:49:57.142 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:56 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:56 smithi193 ceph-mon[33593]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:49:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:56 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:58.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: pgmap v79: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: pgmap v79: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.206 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.206 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:58.206 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:58.206 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.206 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:58.206 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:58.206 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:58.206 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: pgmap v79: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:58.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:58.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:58.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:58.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:58.783 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:58 smithi064 sudo[63420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:49:58.783 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:58 smithi064 sudo[63420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:58.783 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:58 smithi064 sudo[63420]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:58.783 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:58 smithi064 sudo[63420]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:59.052 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:58 smithi064 sudo[63490]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:49:59.052 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:58 smithi064 sudo[63490]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:59.052 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:58 smithi064 sudo[63490]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:59.052 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 11:49:58 smithi064 sudo[63490]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[30223]: Detected new or changed devices on smithi064 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi064 to 5354M 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[30223]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:59.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[42243]: Detected new or changed devices on smithi064 2024-03-24T11:49:59.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi064 to 5354M 2024-03-24T11:49:59.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:59.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[42243]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:49:59.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:59.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:59.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:59.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:49:58 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:58 smithi193 ceph-mon[33593]: Detected new or changed devices on smithi064 2024-03-24T11:49:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:58 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi064 to 5354M 2024-03-24T11:49:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:58 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:58 smithi193 ceph-mon[33593]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:49:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:58 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:49:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:58 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:49:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:58 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:49:58 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:49:59.337 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63493]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:49:59.337 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63493]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:59.337 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63493]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:59.337 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63493]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:59.596 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63551]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:49:59.597 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:59.597 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63551]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:59.597 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63551]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:59.865 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63593]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:49:59.865 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63593]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:59.865 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63593]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:59.865 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63593]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:50:00.125 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63596]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:50:00.125 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63596]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:50:00.125 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63596]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:50:00.125 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 11:49:59 smithi064 sudo[63596]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:50:00.387 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 sudo[63607]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:50:00.387 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 sudo[63607]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:50:00.387 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 sudo[63607]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:50:00.387 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 sudo[63607]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:50:00.639 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[42243]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[42243]: from='client.14307 v1:172.21.15.64:0/2840580256' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[42243]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[42243]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 sudo[63633]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 sudo[63633]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 sudo[63633]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 sudo[63633]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[30223]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:00.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[30223]: from='client.14307 v1:172.21.15.64:0/2840580256' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:00.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[30223]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:50:00.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[30223]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:50:00.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:00.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:00.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 ceph-mon[33593]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 ceph-mon[33593]: from='client.14307 v1:172.21.15.64:0/2840580256' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 ceph-mon[33593]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 ceph-mon[33593]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 sudo[36420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 sudo[36420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 sudo[36420]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:50:00.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 sudo[36420]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:50:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 sudo[36426]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:50:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 sudo[36426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:50:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 sudo[36426]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:50:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:00 smithi193 sudo[36426]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:50:01.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 sudo[63716]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:50:01.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 sudo[63716]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:50:01.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 sudo[63716]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:50:01.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 sudo[63716]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:50:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: overall HEALTH_OK 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:01 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3047936733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479"}]: dispatch 2024-03-24T11:50:01.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:01.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:01.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: overall HEALTH_OK 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:01.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3047936733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479"}]: dispatch 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 sudo[63721]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 sudo[63721]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 sudo[63721]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 sudo[63721]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: overall HEALTH_OK 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:01.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:50:01.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:50:01.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:50:01.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:01 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3047936733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479"}]: dispatch 2024-03-24T11:50:02.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[30223]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[30223]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3047936733' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479"}]': finished 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[30223]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1591942845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[42243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[42243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3047936733' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479"}]': finished 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[42243]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:02.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:02 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1591942845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:02 smithi193 ceph-mon[33593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:50:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:02 smithi193 ceph-mon[33593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:50:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:02 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3047936733' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479"}]': finished 2024-03-24T11:50:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:02 smithi193 ceph-mon[33593]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:50:02.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:02 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:02.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:02 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1591942845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:03.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:03 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:50:03.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:03 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:50:03.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:03 smithi193 ceph-mon[33593]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:03.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:03 smithi193 ceph-mon[33593]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:50:03.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:03 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:50:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:50:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[30223]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[30223]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:50:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:50:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:50:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[42243]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[42243]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:50:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:03 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:05.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:05 smithi193 ceph-mon[33593]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 254 KiB/s wr, 17 op/s 2024-03-24T11:50:05.891 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:05 smithi064 ceph-mon[30223]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 254 KiB/s wr, 17 op/s 2024-03-24T11:50:05.892 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:05 smithi064 ceph-mon[42243]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 254 KiB/s wr, 17 op/s 2024-03-24T11:50:07.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:07 smithi064 ceph-mon[30223]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T11:50:07.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:07 smithi064 ceph-mon[42243]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T11:50:07.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:07 smithi193 ceph-mon[33593]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T11:50:08.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:08 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:50:08.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:08 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:08.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:08 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:50:08.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:08 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:08.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:08 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:50:08.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:08 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:09.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:09 smithi193 ceph-mon[33593]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-24T11:50:09.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:09 smithi193 ceph-mon[33593]: Deploying daemon osd.3 on smithi064 2024-03-24T11:50:09.993 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:09 smithi064 ceph-mon[30223]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-24T11:50:09.993 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:09 smithi064 ceph-mon[30223]: Deploying daemon osd.3 on smithi064 2024-03-24T11:50:09.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:09 smithi064 ceph-mon[42243]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-24T11:50:09.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:09 smithi064 ceph-mon[42243]: Deploying daemon osd.3 on smithi064 2024-03-24T11:50:11.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:11 smithi064 ceph-mon[30223]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-24T11:50:11.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:11 smithi064 ceph-mon[42243]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-24T11:50:12.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:11 smithi193 ceph-mon[33593]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-24T11:50:13.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:13 smithi064 ceph-mon[30223]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-24T11:50:13.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:13 smithi064 ceph-mon[42243]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-24T11:50:14.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:13 smithi193 ceph-mon[33593]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-24T11:50:14.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:14 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:14.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:14 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:14.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:14 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:14.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:14 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:14.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:14 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:14.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:14 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:14 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:14 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:14 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:15 smithi064 ceph-mon[30223]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:15 smithi064 ceph-mon[42243]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-03-24T11:50:16.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:15 smithi193 ceph-mon[33593]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-03-24T11:50:17.060 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 3 on host 'smithi064' 2024-03-24T11:50:17.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[42243]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:17.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[42243]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:50:17.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:17.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:17.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:17.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:17.730 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[30223]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:17.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[30223]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:50:17.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:17.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:17.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:17.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:17 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:17.757 DEBUG:teuthology.orchestra.run.smithi064:osd.3> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T11:50:17.760 INFO:tasks.cephadm:Deploying osd.4 on smithi193 with /dev/vg_nvme/lv_4... 2024-03-24T11:50:17.760 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:50:18.008 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:50:18.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:17 smithi193 ceph-mon[33593]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:18.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:17 smithi193 ceph-mon[33593]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:50:18.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:17 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:18.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:17 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:17 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:17 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:18.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:18 smithi193 ceph-mon[33593]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:50:18.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:18 smithi193 ceph-mon[33593]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:50:18.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:18 smithi193 ceph-mon[33593]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:50:18.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:18 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:18.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-mon[30223]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:50:18.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-mon[30223]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:50:18.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-mon[30223]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:50:18.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:18.970 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-mon[42243]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:50:18.970 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-mon[42243]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:50:18.970 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-mon[42243]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-24T11:50:18.970 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:18.971 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3[65507]: 2024-03-24T11:50:18.646+0000 7f7762eb6700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T11:50:18.971 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 11:50:18 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3[65507]: 2024-03-24T11:50:18.651+0000 7f775eca9700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:50:20.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:19 smithi193 ceph-mon[33593]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:20.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:19 smithi193 ceph-mon[33593]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:50:20.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:19 smithi193 ceph-mon[33593]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T11:50:20.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:19 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:20.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:19 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:20.009 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:50:20.009 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:50:20.009 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-24T11:50:20.009 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-24T11:50:20.010 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115024 s, 912 MB/s 2024-03-24T11:50:20.010 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 2024-03-24T11:50:20.010 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-24T11:50:20.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[30223]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:20.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[30223]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:50:20.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[30223]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T11:50:20.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:20.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:20.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[42243]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:20.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[42243]: from='osd.3 v1:172.21.15.64:6813/3355318038' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-24T11:50:20.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[42243]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T11:50:20.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:20.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:19 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:20.659 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-03-24T11:50:20.727 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[30223]: osd.3 v1:172.21.15.64:6813/3355318038 boot 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[30223]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[42243]: osd.3 v1:172.21.15.64:6813/3355318038 boot 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[42243]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:50:20.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:20.894 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:50:21.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:20 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:50:21.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:20 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:50:21.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:20 smithi193 ceph-mon[33593]: osd.3 v1:172.21.15.64:6813/3355318038 boot 2024-03-24T11:50:21.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:20 smithi193 ceph-mon[33593]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:50:21.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:20 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:22.014 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:22.014 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T11:50:22.014 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: Detected new or changed devices on smithi064 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi064 to 4016M 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:22.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:22.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:21 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:22.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T11:50:22.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: Detected new or changed devices on smithi064 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi064 to 4016M 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:22.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: Detected new or changed devices on smithi064 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:22.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi064 to 4016M 2024-03-24T11:50:22.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:22.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:22.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:22.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:22.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:22.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:22.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:21 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:24.006 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:23 smithi193 ceph-mon[33593]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:24.006 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:23 smithi193 ceph-mon[33593]: from='client.24229 v1:172.21.15.193:0/1873511698' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:24.007 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:50:24.007 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:50:24.007 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:24.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[30223]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:24.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[30223]: from='client.24229 v1:172.21.15.193:0/1873511698' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:24.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:50:24.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:50:24.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:24.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[42243]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:24.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[42243]: from='client.24229 v1:172.21.15.193:0/1873511698' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:24.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:50:24.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:50:24.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[30223]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/3643403366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]: dispatch 2024-03-24T11:50:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[30223]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]: dispatch 2024-03-24T11:50:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[30223]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]': finished 2024-03-24T11:50:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[30223]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:50:26.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[42243]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/3643403366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]: dispatch 2024-03-24T11:50:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[42243]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]: dispatch 2024-03-24T11:50:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[42243]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]': finished 2024-03-24T11:50:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[42243]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:50:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:25 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:25 smithi193 ceph-mon[33593]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:25 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/3643403366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]: dispatch 2024-03-24T11:50:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:25 smithi193 ceph-mon[33593]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]: dispatch 2024-03-24T11:50:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:25 smithi193 ceph-mon[33593]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b"}]': finished 2024-03-24T11:50:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:25 smithi193 ceph-mon[33593]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:50:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:25 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:27.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:26 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/298197191' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:27.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:26 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/298197191' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:26 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/298197191' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:27 smithi064 ceph-mon[30223]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:27 smithi064 ceph-mon[42243]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:27 smithi193 ceph-mon[33593]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:29.907 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:29 smithi193 ceph-mon[33593]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:29 smithi064 ceph-mon[30223]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:30.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:29 smithi064 ceph-mon[42243]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:31.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:31 smithi193 ceph-mon[33593]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:32.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:31 smithi064 ceph-mon[30223]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:32.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:31 smithi064 ceph-mon[42243]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:33.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:32 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:50:33.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:32 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:33.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:32 smithi064 ceph-mon[30223]: Deploying daemon osd.4 on smithi193 2024-03-24T11:50:33.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:32 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:50:33.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:32 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:33.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:32 smithi064 ceph-mon[42243]: Deploying daemon osd.4 on smithi193 2024-03-24T11:50:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:32 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:50:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:32 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:33.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:32 smithi193 ceph-mon[33593]: Deploying daemon osd.4 on smithi193 2024-03-24T11:50:34.006 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:33 smithi193 ceph-mon[33593]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:34.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:33 smithi064 ceph-mon[30223]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:34.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:33 smithi064 ceph-mon[42243]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:35.948 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:35 smithi193 ceph-mon[33593]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:36.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:35 smithi064 ceph-mon[30223]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:36.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:35 smithi064 ceph-mon[42243]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:37.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:37 smithi193 ceph-mon[33593]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:37.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:37.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:37.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:37 smithi064 ceph-mon[30223]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:37 smithi064 ceph-mon[42243]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:38.783 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:38 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:50:38.783 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:38 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:50:38.783 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:38 smithi193 ceph-mon[33593]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:50:38.783 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:39.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:38 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:50:39.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:38 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:50:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:38 smithi064 ceph-mon[42243]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:50:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:39.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:38 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:50:39.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:38 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:50:39.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:38 smithi064 ceph-mon[30223]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:50:39.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:39 smithi193 ceph-mon[33593]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:39 smithi064 ceph-mon[30223]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:40.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:39 smithi064 ceph-mon[42243]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:40.542 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 4 on host 'smithi193' 2024-03-24T11:50:40.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:40 smithi193 ceph-mon[33593]: from='osd.4 v1:172.21.15.193:6800/3851339176' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:50:40.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:40 smithi193 ceph-mon[33593]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:50:40.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:40 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:40.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:40 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:40.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:40 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:40.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:40 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[30223]: from='osd.4 v1:172.21.15.193:6800/3851339176' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:50:41.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[30223]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:50:41.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[42243]: from='osd.4 v1:172.21.15.193:6800/3851339176' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:50:41.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[42243]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:50:41.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:40 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:41.159 DEBUG:teuthology.orchestra.run.smithi193:osd.4> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T11:50:41.162 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-03-24T11:50:41.162 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:50:41.442 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:50:41.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:41 smithi193 ceph-mon[33593]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:41.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:41 smithi193 ceph-mon[33593]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:50:41.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:41 smithi193 ceph-mon[33593]: from='osd.4 v1:172.21.15.193:6800/3851339176' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:50:41.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:41 smithi193 ceph-mon[33593]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T11:50:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:41.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:41 smithi193 ceph-mon[33593]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:50:42.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[30223]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:42.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[30223]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:50:42.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[30223]: from='osd.4 v1:172.21.15.193:6800/3851339176' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:50:42.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[30223]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T11:50:42.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:42.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[30223]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:50:42.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[42243]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:42.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[42243]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:50:42.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[42243]: from='osd.4 v1:172.21.15.193:6800/3851339176' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:50:42.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[42243]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T11:50:42.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:42.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:41 smithi064 ceph-mon[42243]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:50:42.253 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 11:50:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T11:50:42.149+0000 7fc417bea700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T11:50:42.253 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 11:50:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T11:50:42.152+0000 7fc40f9d5700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:50:42.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:42 smithi193 ceph-mon[33593]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:50:42.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:42 smithi193 ceph-mon[33593]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T11:50:42.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:42 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:42 smithi193 ceph-mon[33593]: from='osd.4 ' entity='osd.4' 2024-03-24T11:50:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:42 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[30223]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:50:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[30223]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T11:50:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:43.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[30223]: from='osd.4 ' entity='osd.4' 2024-03-24T11:50:43.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[42243]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:50:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[42243]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T11:50:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[42243]: from='osd.4 ' entity='osd.4' 2024-03-24T11:50:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:42 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:43.831 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:43 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:50:43.831 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:43 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:50:43.831 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:43 smithi193 ceph-mon[33593]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:43.831 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:43 smithi193 ceph-mon[33593]: osd.4 v1:172.21.15.193:6800/3851339176 boot 2024-03-24T11:50:43.831 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:43 smithi193 ceph-mon[33593]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:50:43.831 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:43 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:44.032 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:50:44.032 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:50:44.032 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-24T11:50:44.032 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-24T11:50:44.032 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124005 s, 846 MB/s 2024-03-24T11:50:44.032 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-24T11:50:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:50:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:50:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[30223]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[30223]: osd.4 v1:172.21.15.193:6800/3851339176 boot 2024-03-24T11:50:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[30223]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:50:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:50:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:50:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[42243]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T11:50:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[42243]: osd.4 v1:172.21.15.193:6800/3851339176 boot 2024-03-24T11:50:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[42243]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:50:44.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:43 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:44.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-03-24T11:50:44.997 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:44 smithi193 ceph-mon[33593]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:50:44.997 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:44 smithi193 ceph-mon[33593]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:50:45.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:44 smithi064 ceph-mon[30223]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:50:45.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:44 smithi064 ceph-mon[30223]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:50:45.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:44 smithi064 ceph-mon[42243]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:50:45.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:44 smithi064 ceph-mon[42243]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:50:45.241 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:50:45.887 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:45 smithi193 ceph-mon[33593]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:46.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:45 smithi064 ceph-mon[30223]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:46.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:45 smithi064 ceph-mon[42243]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:46.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T11:50:46.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:46.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T11:50:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T11:50:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:50:48.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[30223]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:48.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[30223]: Detected new or changed devices on smithi193 2024-03-24T11:50:48.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi193 to 17088M 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[30223]: osdmap e37: 5 total, 5 up, 5 in 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[42243]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[42243]: Detected new or changed devices on smithi193 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi193 to 17088M 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[42243]: osdmap e37: 5 total, 5 up, 5 in 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:50:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:47 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:47 smithi193 ceph-mon[33593]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:47 smithi193 ceph-mon[33593]: Detected new or changed devices on smithi193 2024-03-24T11:50:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:47 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi193 to 17088M 2024-03-24T11:50:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:47 smithi193 ceph-mon[33593]: osdmap e37: 5 total, 5 up, 5 in 2024-03-24T11:50:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:47 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:50:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:47 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:50:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:47 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:49.015 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:48 smithi193 ceph-mon[33593]: from='client.24256 v1:172.21.15.193:0/3159059522' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:48 smithi064 ceph-mon[30223]: from='client.24256 v1:172.21.15.193:0/3159059522' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:49.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:48 smithi064 ceph-mon[42243]: from='client.24256 v1:172.21.15.193:0/3159059522' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:49 smithi064 ceph-mon[30223]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:50.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:49 smithi064 ceph-mon[42243]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:49 smithi193 ceph-mon[33593]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:50:50.899 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:50 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/1675519862' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]: dispatch 2024-03-24T11:50:50.899 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:50 smithi193 ceph-mon[33593]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]: dispatch 2024-03-24T11:50:50.899 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:50 smithi193 ceph-mon[33593]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]': finished 2024-03-24T11:50:50.899 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:50 smithi193 ceph-mon[33593]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:50:50.899 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:50 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:50:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/1675519862' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]: dispatch 2024-03-24T11:50:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[30223]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]: dispatch 2024-03-24T11:50:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[30223]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]': finished 2024-03-24T11:50:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[30223]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:50:51.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:50:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/1675519862' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]: dispatch 2024-03-24T11:50:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[42243]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]: dispatch 2024-03-24T11:50:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[42243]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6"}]': finished 2024-03-24T11:50:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[42243]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:50:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:50 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:50:52.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:51 smithi064 ceph-mon[42243]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:50:52.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:51 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/1326962960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:52.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:51 smithi064 ceph-mon[30223]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:50:52.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:51 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/1326962960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:52.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:51 smithi193 ceph-mon[33593]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:50:52.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:51 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/1326962960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:50:53.975 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:53 smithi193 ceph-mon[33593]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T11:50:54.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:53 smithi064 ceph-mon[30223]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T11:50:54.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:53 smithi064 ceph-mon[42243]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T11:50:55.890 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:55 smithi193 ceph-mon[33593]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:50:56.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:55 smithi064 ceph-mon[30223]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:50:56.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:55 smithi064 ceph-mon[42243]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:50:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:57 smithi193 ceph-mon[33593]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T11:50:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:50:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:57 smithi193 ceph-mon[33593]: Deploying daemon osd.5 on smithi193 2024-03-24T11:50:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:57 smithi064 ceph-mon[30223]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T11:50:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:50:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:57 smithi064 ceph-mon[30223]: Deploying daemon osd.5 on smithi193 2024-03-24T11:50:58.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:57 smithi064 ceph-mon[42243]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T11:50:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:50:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:57 smithi064 ceph-mon[42243]: Deploying daemon osd.5 on smithi193 2024-03-24T11:50:59.891 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:50:59 smithi193 ceph-mon[33593]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:51:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:50:59 smithi064 ceph-mon[42243]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:51:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:50:59 smithi064 ceph-mon[30223]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:51:01.929 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:01 smithi193 ceph-mon[33593]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:02.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:01 smithi064 ceph-mon[30223]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:02.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:01 smithi064 ceph-mon[42243]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:03.933 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:03 smithi193 ceph-mon[33593]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:03.934 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:03 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:03.934 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:03 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:03.934 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:03 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:03 smithi064 ceph-mon[30223]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:03 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:03 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:03 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:03 smithi064 ceph-mon[42243]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:03 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:03 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:03 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[30223]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T11:51:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T11:51:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[42243]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:06.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T11:51:06.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T11:51:06.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:06.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:06.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:05 smithi193 ceph-mon[33593]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:05 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T11:51:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:05 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T11:51:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.483 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-03-24T11:51:06.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:06 smithi193 ceph-mon[33593]: from='osd.5 v1:172.21.15.193:6804/1625306337' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:51:06.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:06 smithi193 ceph-mon[33593]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:51:06.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:06 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:06 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:06.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:06 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:07.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[30223]: from='osd.5 v1:172.21.15.193:6804/1625306337' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:51:07.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[30223]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:51:07.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:07.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:07.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:07.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[42243]: from='osd.5 v1:172.21.15.193:6804/1625306337' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:51:07.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[42243]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:51:07.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:07.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:07.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:06 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:07.154 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T11:51:07.196 INFO:tasks.cephadm:Deploying osd.6 on smithi193 with /dev/vg_nvme/lv_2... 2024-03-24T11:51:07.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:51:07.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:07 smithi193 ceph-mon[33593]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:07.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:07 smithi193 ceph-mon[33593]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:51:07.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:07 smithi193 ceph-mon[33593]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T11:51:07.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:07 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:07.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:07 smithi193 ceph-mon[33593]: from='osd.5 v1:172.21.15.193:6804/1625306337' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:07.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:07 smithi193 ceph-mon[33593]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:07.734 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 11:51:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[45137]: 2024-03-24T11:51:07.708+0000 7f9d07e09700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T11:51:07.734 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 11:51:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[45137]: 2024-03-24T11:51:07.712+0000 7f9d03bfc700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:51:07.940 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:51:08.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[30223]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:08.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[30223]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:51:08.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[30223]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[30223]: from='osd.5 v1:172.21.15.193:6804/1625306337' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[30223]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[42243]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[42243]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[42243]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[42243]: from='osd.5 v1:172.21.15.193:6804/1625306337' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:08.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:07 smithi064 ceph-mon[42243]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:08.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:08 smithi193 ceph-mon[33593]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:51:08.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:08 smithi193 ceph-mon[33593]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T11:51:08.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:08 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:08.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:08 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:08.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:08 smithi193 ceph-mon[33593]: osd.5 v1:172.21.15.193:6804/1625306337 boot 2024-03-24T11:51:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[30223]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:51:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[30223]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T11:51:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[30223]: osd.5 v1:172.21.15.193:6804/1625306337 boot 2024-03-24T11:51:09.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[42243]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:51:09.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[42243]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T11:51:09.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:09.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:09.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:08 smithi064 ceph-mon[42243]: osd.5 v1:172.21.15.193:6804/1625306337 boot 2024-03-24T11:51:09.827 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:09 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:51:09.828 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:09 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:51:09.828 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:09 smithi193 ceph-mon[33593]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:09.828 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:09 smithi193 ceph-mon[33593]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:51:09.828 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:09 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:09.828 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:09 smithi193 ceph-mon[33593]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:51:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:51:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:51:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[42243]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[42243]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:51:10.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:10.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[42243]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:51:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:51:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:51:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[30223]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:51:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[30223]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:51:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:10.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:09 smithi064 ceph-mon[30223]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:51:10.805 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:51:10.806 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:51:10.806 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-24T11:51:10.806 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-24T11:51:10.806 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115016 s, 912 MB/s 2024-03-24T11:51:10.806 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-24T11:51:11.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:11.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:11.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:11 smithi193 ceph-mon[33593]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T11:51:11.882 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-03-24T11:51:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:51:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[30223]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T11:51:12.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:12.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:11 smithi064 ceph-mon[42243]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T11:51:12.117 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:51:12.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:12 smithi193 ceph-mon[33593]: Detected new or changed devices on smithi193 2024-03-24T11:51:12.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:12 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi193 to 8544M 2024-03-24T11:51:12.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:12 smithi193 ceph-mon[33593]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T11:51:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:12 smithi064 ceph-mon[30223]: Detected new or changed devices on smithi193 2024-03-24T11:51:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:12 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi193 to 8544M 2024-03-24T11:51:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:12 smithi064 ceph-mon[30223]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T11:51:13.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:12 smithi064 ceph-mon[42243]: Detected new or changed devices on smithi193 2024-03-24T11:51:13.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:12 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi193 to 8544M 2024-03-24T11:51:13.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:12 smithi064 ceph-mon[42243]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T11:51:13.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:13 smithi193 ceph-mon[33593]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:14.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:13 smithi064 ceph-mon[30223]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:14.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:13 smithi064 ceph-mon[42243]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:14 smithi193 ceph-mon[33593]: from='client.24283 v1:172.21.15.193:0/799276087' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:14 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:51:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:14 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:51:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:14 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:14 smithi064 ceph-mon[30223]: from='client.24283 v1:172.21.15.193:0/799276087' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:14 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:51:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:14 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:51:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:14 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:15.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:14 smithi064 ceph-mon[42243]: from='client.24283 v1:172.21.15.193:0/799276087' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:14 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:51:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:14 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:51:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:14 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:15.941 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:15 smithi193 ceph-mon[33593]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:16.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:15 smithi064 ceph-mon[30223]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:16.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:15 smithi064 ceph-mon[42243]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:18.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[42243]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:18.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/285826585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]: dispatch 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[42243]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]: dispatch 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[42243]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]': finished 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[42243]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/3918273234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[30223]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/285826585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]: dispatch 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[30223]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]: dispatch 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[30223]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]': finished 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[30223]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:18.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/3918273234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:51:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:17 smithi193 ceph-mon[33593]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/285826585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]: dispatch 2024-03-24T11:51:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:17 smithi193 ceph-mon[33593]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]: dispatch 2024-03-24T11:51:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:17 smithi193 ceph-mon[33593]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996"}]': finished 2024-03-24T11:51:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:17 smithi193 ceph-mon[33593]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:51:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:17 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:18.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/3918273234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:51:19.889 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:19 smithi193 ceph-mon[33593]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:20.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:19 smithi064 ceph-mon[30223]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:20.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:19 smithi064 ceph-mon[42243]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:21.014 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:20 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:51:21.014 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:20 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:51:21.014 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:20 smithi193 ceph-mon[33593]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T11:51:21.014 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:20 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:21.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:20 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:51:21.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:20 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:51:21.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:20 smithi064 ceph-mon[30223]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T11:51:21.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:20 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:21.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:20 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:51:21.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:20 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:51:21.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:20 smithi064 ceph-mon[42243]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T11:51:21.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:20 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:21.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:21 smithi193 ceph-mon[33593]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:22.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:21 smithi064 ceph-mon[30223]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:22.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:21 smithi064 ceph-mon[42243]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:24.032 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:23 smithi193 ceph-mon[33593]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:24.032 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:51:24.032 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:24.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:23 smithi064 ceph-mon[30223]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:24.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:51:24.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:24.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:23 smithi064 ceph-mon[42243]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:24.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:51:24.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:25.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:24 smithi064 ceph-mon[30223]: Deploying daemon osd.6 on smithi193 2024-03-24T11:51:25.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:24 smithi064 ceph-mon[42243]: Deploying daemon osd.6 on smithi193 2024-03-24T11:51:25.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:24 smithi193 ceph-mon[33593]: Deploying daemon osd.6 on smithi193 2024-03-24T11:51:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:25 smithi193 ceph-mon[33593]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:25 smithi064 ceph-mon[30223]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:26.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:25 smithi064 ceph-mon[42243]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:27.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:27 smithi193 ceph-mon[33593]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:27 smithi064 ceph-mon[30223]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:27 smithi064 ceph-mon[42243]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:29 smithi193 ceph-mon[33593]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:29 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:29 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:29 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:29 smithi064 ceph-mon[30223]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:29 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:29 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:29 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:30.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:29 smithi064 ceph-mon[42243]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:30.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:29 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:30.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:29 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:30.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:29 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:31.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:30 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:51:31.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:30 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:51:31.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:30 smithi064 ceph-mon[30223]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T11:51:31.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:30 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:31.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:30 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:51:31.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:30 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:51:31.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:30 smithi064 ceph-mon[42243]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T11:51:31.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:30 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:30 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:51:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:30 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:51:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:30 smithi193 ceph-mon[33593]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T11:51:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:30 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:31.960 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:31 smithi193 ceph-mon[33593]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:32.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:31 smithi064 ceph-mon[30223]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:32.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:31 smithi064 ceph-mon[42243]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:33.134 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 6 on host 'smithi193' 2024-03-24T11:51:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:33.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:33.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='osd.6 v1:172.21.15.193:6808/2408237278' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:51:33.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:51:33.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:33.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:33 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:33.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:33.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='osd.6 v1:172.21.15.193:6808/2408237278' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:33.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='osd.6 v1:172.21.15.193:6808/2408237278' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:33 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:33.827 DEBUG:teuthology.orchestra.run.smithi193:osd.6> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T11:51:33.830 INFO:tasks.cephadm:Deploying osd.7 on smithi193 with /dev/vg_nvme/lv_1... 2024-03-24T11:51:33.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:51:34.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:34 smithi193 ceph-mon[33593]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:51:34.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:34 smithi193 ceph-mon[33593]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T11:51:34.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:34 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:34.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:34 smithi193 ceph-mon[33593]: from='osd.6 v1:172.21.15.193:6808/2408237278' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:34.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:34 smithi193 ceph-mon[33593]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:34.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[42243]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:51:34.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[42243]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T11:51:34.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:34.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[42243]: from='osd.6 v1:172.21.15.193:6808/2408237278' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:34.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[42243]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:34.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[30223]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:51:34.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[30223]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T11:51:34.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:34.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[30223]: from='osd.6 v1:172.21.15.193:6808/2408237278' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:34.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:34 smithi064 ceph-mon[30223]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:51:34.697 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:51:34.914 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 11:51:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T11:51:34.627+0000 7f9f982e2700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T11:51:34.915 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 11:51:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T11:51:34.634+0000 7f9f8f8cc700 -1 osd.6 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:51:35.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:35 smithi193 ceph-mon[33593]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:35.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:35 smithi193 ceph-mon[33593]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:51:35.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:35 smithi193 ceph-mon[33593]: osdmap e50: 7 total, 6 up, 7 in 2024-03-24T11:51:35.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:35 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:35.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:35 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:35.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:35 smithi193 ceph-mon[33593]: from='osd.6 ' entity='osd.6' 2024-03-24T11:51:35.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[30223]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:35.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[30223]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:51:35.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[30223]: osdmap e50: 7 total, 6 up, 7 in 2024-03-24T11:51:35.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:35.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:35.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[30223]: from='osd.6 ' entity='osd.6' 2024-03-24T11:51:35.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[42243]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:35.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[42243]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:51:35.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[42243]: osdmap e50: 7 total, 6 up, 7 in 2024-03-24T11:51:35.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:35.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:35.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:35 smithi064 ceph-mon[42243]: from='osd.6 ' entity='osd.6' 2024-03-24T11:51:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:51:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:51:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[30223]: osd.6 v1:172.21.15.193:6808/2408237278 boot 2024-03-24T11:51:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[30223]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:51:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:36.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:51:36.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:51:36.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:36.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[42243]: osd.6 v1:172.21.15.193:6808/2408237278 boot 2024-03-24T11:51:36.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[42243]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:51:36.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:36 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:36 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:51:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:36 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:51:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:36 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:36 smithi193 ceph-mon[33593]: osd.6 v1:172.21.15.193:6808/2408237278 boot 2024-03-24T11:51:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:36 smithi193 ceph-mon[33593]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:51:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:36 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:37.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:37 smithi193 ceph-mon[33593]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:37.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:37 smithi193 ceph-mon[33593]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T11:51:37.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:37 smithi064 ceph-mon[30223]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:37.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:37 smithi064 ceph-mon[30223]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T11:51:37.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:37 smithi064 ceph-mon[42243]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:51:37.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:37 smithi064 ceph-mon[42243]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T11:51:37.750 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:51:37.750 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:51:37.750 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-24T11:51:37.750 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-24T11:51:37.750 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116286 s, 902 MB/s 2024-03-24T11:51:37.750 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-24T11:51:38.504 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T11:51:38.504 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.504 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.504 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.504 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.504 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.504 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.504 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:38.505 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:38.505 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:38 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T11:51:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:38.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:38 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:38.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-03-24T11:51:39.061 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:51:39.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:39 smithi193 ceph-mon[33593]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T11:51:39.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:39 smithi193 ceph-mon[33593]: Detected new or changed devices on smithi193 2024-03-24T11:51:39.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:39 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi193 to 5696M 2024-03-24T11:51:39.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:39 smithi193 ceph-mon[33593]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T11:51:39.525 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:39 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:51:39.525 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:39 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:51:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[30223]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T11:51:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[30223]: Detected new or changed devices on smithi193 2024-03-24T11:51:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi193 to 5696M 2024-03-24T11:51:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[30223]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T11:51:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:51:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:51:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[42243]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T11:51:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[42243]: Detected new or changed devices on smithi193 2024-03-24T11:51:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi193 to 5696M 2024-03-24T11:51:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[42243]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T11:51:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:51:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:39 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:51:40.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:40 smithi193 ceph-mon[33593]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T11:51:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:40 smithi064 ceph-mon[30223]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T11:51:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:40 smithi064 ceph-mon[42243]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T11:51:41.539 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:41 smithi193 ceph-mon[33593]: pgmap v167: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T11:51:41.540 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:41 smithi193 ceph-mon[33593]: from='client.24310 v1:172.21.15.193:0/12399419' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:41.540 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:51:41.540 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:51:41.540 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:41 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:41.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[30223]: pgmap v167: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T11:51:41.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[30223]: from='client.24310 v1:172.21.15.193:0/12399419' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:41.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:51:41.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:51:41.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[42243]: pgmap v167: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T11:51:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[42243]: from='client.24310 v1:172.21.15.193:0/12399419' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:41.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:51:41.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:51:41.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:41 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:43 smithi064 ceph-mon[30223]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T11:51:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:43 smithi064 ceph-mon[42243]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T11:51:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:43 smithi193 ceph-mon[33593]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T11:51:45.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/1024915305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]: dispatch 2024-03-24T11:51:45.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[30223]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]: dispatch 2024-03-24T11:51:45.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[30223]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]': finished 2024-03-24T11:51:45.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[30223]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T11:51:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:45.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/753018298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:51:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/1024915305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]: dispatch 2024-03-24T11:51:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[42243]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]: dispatch 2024-03-24T11:51:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[42243]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]': finished 2024-03-24T11:51:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[42243]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T11:51:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:45.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/753018298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:51:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/1024915305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]: dispatch 2024-03-24T11:51:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:44 smithi193 ceph-mon[33593]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]: dispatch 2024-03-24T11:51:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:44 smithi193 ceph-mon[33593]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015"}]': finished 2024-03-24T11:51:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:44 smithi193 ceph-mon[33593]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T11:51:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:44 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/753018298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:51:46.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:45 smithi064 ceph-mon[30223]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:46.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:45 smithi064 ceph-mon[42243]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:46.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:45 smithi193 ceph-mon[33593]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:48.000 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:47 smithi193 ceph-mon[33593]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:48.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:47 smithi064 ceph-mon[30223]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:48.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:47 smithi064 ceph-mon[42243]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:49 smithi064 ceph-mon[30223]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:50.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:49 smithi064 ceph-mon[42243]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:50.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:49 smithi193 ceph-mon[33593]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:50 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:51:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:50 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:50 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:51:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:50 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:51.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:50 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:51:51.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:50 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:52.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:51 smithi064 ceph-mon[30223]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:52.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:51 smithi064 ceph-mon[30223]: Deploying daemon osd.7 on smithi193 2024-03-24T11:51:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:51 smithi064 ceph-mon[42243]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:52.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:51 smithi064 ceph-mon[42243]: Deploying daemon osd.7 on smithi193 2024-03-24T11:51:52.082 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:51 smithi193 ceph-mon[33593]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:52.082 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:51 smithi193 ceph-mon[33593]: Deploying daemon osd.7 on smithi193 2024-03-24T11:51:52.753 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:52 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:51:52.753 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:52 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:51:53.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:52 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:51:53.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:52 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:51:53.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:52 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:51:53.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:52 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:51:53.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:53 smithi193 ceph-mon[33593]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:53.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:53 smithi193 ceph-mon[33593]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T11:51:53.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:53 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:54.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:53 smithi064 ceph-mon[30223]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:54.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:53 smithi064 ceph-mon[30223]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T11:51:54.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:53 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:54.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:53 smithi064 ceph-mon[42243]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:54.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:53 smithi064 ceph-mon[42243]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T11:51:54.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:53 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:55.002 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:54 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:51:55.003 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:54 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:51:55.003 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:54 smithi193 ceph-mon[33593]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T11:51:55.003 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:54 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:55.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:54 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:51:55.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:54 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:51:55.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:54 smithi064 ceph-mon[30223]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T11:51:55.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:54 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:55.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:54 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:51:55.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:54 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:51:55.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:54 smithi064 ceph-mon[42243]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T11:51:55.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:54 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:56.028 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:55 smithi193 ceph-mon[33593]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:56.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:55 smithi064 ceph-mon[30223]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:56.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:55 smithi064 ceph-mon[42243]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:57 smithi064 ceph-mon[30223]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:57 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:57.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:57 smithi064 ceph-mon[42243]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:57.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:57.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:57.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:57 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:57 smithi193 ceph-mon[33593]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:57 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:51:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:51:59 smithi064 ceph-mon[30223]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:59.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:51:59 smithi064 ceph-mon[42243]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:51:59 smithi193 ceph-mon[33593]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:51:59.859 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 7 on host 'smithi193' 2024-03-24T11:52:00.453 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:00 smithi193 ceph-mon[33593]: from='osd.7 v1:172.21.15.193:6812/2649989609' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:52:00.454 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:00 smithi193 ceph-mon[33593]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:52:00.454 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:00.454 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:00 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:00.484 DEBUG:teuthology.orchestra.run.smithi193:osd.7> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T11:52:00.486 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-24T11:52:00.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:52:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:00 smithi064 ceph-mon[30223]: from='osd.7 v1:172.21.15.193:6812/2649989609' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:52:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:00 smithi064 ceph-mon[30223]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:52:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:00 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:00.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:00 smithi064 ceph-mon[42243]: from='osd.7 v1:172.21.15.193:6812/2649989609' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:52:00.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:00 smithi064 ceph-mon[42243]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:52:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:00 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:00.746 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[42243]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:52:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[42243]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:52:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[42243]: osdmap e59: 8 total, 7 up, 8 in 2024-03-24T11:52:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[42243]: from='osd.7 v1:172.21.15.193:6812/2649989609' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:52:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[42243]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:52:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:01.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[30223]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:52:01.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[30223]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:52:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[30223]: osdmap e59: 8 total, 7 up, 8 in 2024-03-24T11:52:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[30223]: from='osd.7 v1:172.21.15.193:6812/2649989609' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:52:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[30223]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:52:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:01 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-mon[33593]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:52:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-mon[33593]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:52:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-mon[33593]: osdmap e59: 8 total, 7 up, 8 in 2024-03-24T11:52:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-mon[33593]: from='osd.7 v1:172.21.15.193:6812/2649989609' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:52:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-mon[33593]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T11:52:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:01.587 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T11:52:01.292+0000 7fda862bc700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T11:52:01.587 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 11:52:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T11:52:01.302+0000 7fda820af700 -1 osd.7 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:52:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[42243]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:52:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[42243]: osdmap e60: 8 total, 7 up, 8 in 2024-03-24T11:52:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[42243]: osd.7 v1:172.21.15.193:6812/2649989609 boot 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[42243]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[30223]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[30223]: osdmap e60: 8 total, 7 up, 8 in 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[30223]: osd.7 v1:172.21.15.193:6812/2649989609 boot 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[30223]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:52:02.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:02 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:02 smithi193 ceph-mon[33593]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T11:52:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:02 smithi193 ceph-mon[33593]: osdmap e60: 8 total, 7 up, 8 in 2024-03-24T11:52:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:02 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:02 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:02 smithi193 ceph-mon[33593]: osd.7 v1:172.21.15.193:6812/2649989609 boot 2024-03-24T11:52:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:02 smithi193 ceph-mon[33593]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:52:02.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:02 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:02.705 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:52:03.402 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":62,"num_osds":8,"num_up_osds":8,"osd_up_since":1711281121,"num_in_osds":8,"osd_in_since":1711281103,"num_remapped_pgs":1} 2024-03-24T11:52:03.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:52:03.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:52:03.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:52:03.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[42243]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:52:03.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[42243]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:52:03.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3313696845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:52:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:52:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:52:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[30223]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:52:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[30223]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:52:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:03 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3313696845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:52:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:03 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:52:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:03 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:52:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:03 smithi193 ceph-mon[33593]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:52:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:03 smithi193 ceph-mon[33593]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:52:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:03 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3313696845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:52:03.650 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:04.650 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:04 smithi193 ceph-mon[33593]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:52:04.733 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:04 smithi064 ceph-mon[30223]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:52:04.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:04 smithi064 ceph-mon[42243]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:52:05.466 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:52:05.466 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":64,"fsid":"06f23456-e9d4-11ee-95cd-87774f69a715","created":"2024-03-24T11:46:40.657065+0000","modified":"2024-03-24T11:52:04.602234+0000","last_up_change":"2024-03-24T11:52:01.590830+0000","last_in_change":"2024-03-24T11:51:43.667453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:49:56.369674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc14a64f-9e6c-41f1-901e-c49fabffb53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2675992571}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2675992571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2675992571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2675992571}]},"public_addr":"172.21.15.64:6801/2675992571","cluster_addr":"172.21.15.64:6802/2675992571","heartbeat_back_addr":"172.21.15.64:6804/2675992571","heartbeat_front_addr":"172.21.15.64:6803/2675992571","state":["exists","up"]},{"osd":1,"uuid":"74587332-f0b2-4905-9888-7bdb604cc0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":841667271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":841667271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":841667271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":841667271}]},"public_addr":"172.21.15.64:6805/841667271","cluster_addr":"172.21.15.64:6806/841667271","heartbeat_back_addr":"172.21.15.64:6808/841667271","heartbeat_front_addr":"172.21.15.64:6807/841667271","state":["exists","up"]},{"osd":2,"uuid":"a0c0ed85-9f33-4198-a1f2-447eb17c364b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":435147390}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":435147390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":435147390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":435147390}]},"public_addr":"172.21.15.64:6809/435147390","cluster_addr":"172.21.15.64:6810/435147390","heartbeat_back_addr":"172.21.15.64:6812/435147390","heartbeat_front_addr":"172.21.15.64:6811/435147390","state":["exists","up"]},{"osd":3,"uuid":"1396f66c-3e0c-4a01-b0a7-6f763989a479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3355318038}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3355318038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3355318038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3355318038}]},"public_addr":"172.21.15.64:6813/3355318038","cluster_addr":"172.21.15.64:6814/3355318038","heartbeat_back_addr":"172.21.15.64:6816/3355318038","heartbeat_front_addr":"172.21.15.64:6815/3355318038","state":["exists","up"]},{"osd":4,"uuid":"50d65d8b-3e90-46a4-b19d-50c3015af40b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3851339176}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3851339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3851339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3851339176}]},"public_addr":"172.21.15.193:6800/3851339176","cluster_addr":"172.21.15.193:6801/3851339176","heartbeat_back_addr":"172.21.15.193:6803/3851339176","heartbeat_front_addr":"172.21.15.193:6802/3851339176","state":["exists","up"]},{"osd":5,"uuid":"50c5d50e-7e68-487c-9f68-1abb8c927ff6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":1625306337}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":1625306337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":1625306337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":1625306337}]},"public_addr":"172.21.15.193:6804/1625306337","cluster_addr":"172.21.15.193:6805/1625306337","heartbeat_back_addr":"172.21.15.193:6807/1625306337","heartbeat_front_addr":"172.21.15.193:6806/1625306337","state":["exists","up"]},{"osd":6,"uuid":"7e74fe39-5707-4c90-ad38-1083cd7a6996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":2408237278}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":2408237278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":2408237278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":2408237278}]},"public_addr":"172.21.15.193:6808/2408237278","cluster_addr":"172.21.15.193:6809/2408237278","heartbeat_back_addr":"172.21.15.193:6811/2408237278","heartbeat_front_addr":"172.21.15.193:6810/2408237278","state":["exists","up"]},{"osd":7,"uuid":"e568df64-cc9a-43f8-a432-8628d1da1015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":2649989609}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":2649989609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":2649989609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":2649989609}]},"public_addr":"172.21.15.193:6812/2649989609","cluster_addr":"172.21.15.193:6813/2649989609","heartbeat_back_addr":"172.21.15.193:6815/2649989609","heartbeat_front_addr":"172.21.15.193:6814/2649989609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:29.543497+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:53.392293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:18.954680+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:38.505736+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:52.085478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:06.740500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:33.233120+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:00.443086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/842059688":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/1610565583":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/2730170969":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/3257265142":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/629947599":"2024-03-25T11:47:08.661692+0000","172.21.15.64:6800/520335082":"2024-03-25T11:47:08.661692+0000","172.21.15.64:6800/4027721305":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/4028269665":"2024-03-25T11:47:36.035658+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:52:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:52:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: Detected new or changed devices on smithi193 2024-03-24T11:52:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi193 to 4272M 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3317740546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: Detected new or changed devices on smithi193 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi193 to 4272M 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:05.805 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:05.805 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3317740546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:05.805 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:05.805 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:05.805 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:05 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:52:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: Detected new or changed devices on smithi193 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi193 to 4272M 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3317740546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:06.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:06.088 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:05 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:06.120 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:49:56.369674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T11:52:06.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T11:52:06.361 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:06.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:06 smithi064 ceph-mon[30223]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:52:06.761 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:06 smithi064 ceph-mon[42243]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:52:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:06 smithi193 ceph-mon[33593]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:52:07.660 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:07 smithi064 ceph-mon[42243]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:07.660 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:07 smithi064 ceph-mon[30223]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:08.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:07 smithi193 ceph-mon[33593]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:08.091 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-03-24T11:52:08.721 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:08 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1985060268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:52:08.722 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:08 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1985060268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:52:08.752 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi064 2024-03-24T11:52:08.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi064=foo.a' 2024-03-24T11:52:09.001 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:08 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1985060268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:52:10.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:09 smithi064 ceph-mon[30223]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T11:52:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:09 smithi064 ceph-mon[42243]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T11:52:10.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:09 smithi193 ceph-mon[33593]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T11:52:10.837 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled rgw.foo.a update... 2024-03-24T11:52:11.560 DEBUG:teuthology.orchestra.run.smithi064:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-24T11:52:11.562 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi193 2024-03-24T11:52:11.563 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-24T11:52:11.800 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='client.24355 v1:172.21.15.193:0/1669380997' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi064=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:52:12.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.201 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:12.201 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:11 smithi193 ceph-mon[33593]: Deploying daemon rgw.foo.a on smithi064 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='client.24355 v1:172.21.15.193:0/1669380997' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi064=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[30223]: Deploying daemon rgw.foo.a on smithi064 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='client.24355 v1:172.21.15.193:0/1669380997' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi064=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:12.220 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:11 smithi064 ceph-mon[42243]: Deploying daemon rgw.foo.a on smithi064 2024-03-24T11:52:13.302 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 11:52:13 smithi064 systemd[1]: Starting Ceph rgw.foo.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:52:13.583 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 11:52:13 smithi064 podman[70582]: 2024-03-24 11:52:13.228886124 +0000 UTC m=+0.017929474 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:52:13.583 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 11:52:13 smithi064 podman[70582]: 2024-03-24 11:52:13.347822362 +0000 UTC m=+0.136865711 container create a18f861b4e00de151e1f3e658101d5414e4ede25b6f542b1403dbf18e0ca195b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T11:52:13.583 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 11:52:13 smithi064 podman[70582]: 2024-03-24 11:52:13.583646412 +0000 UTC m=+0.372689763 container init a18f861b4e00de151e1f3e658101d5414e4ede25b6f542b1403dbf18e0ca195b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b) 2024-03-24T11:52:14.052 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 11:52:13 smithi064 podman[70582]: 2024-03-24 11:52:13.588886898 +0000 UTC m=+0.377930251 container start a18f861b4e00de151e1f3e658101d5414e4ede25b6f542b1403dbf18e0ca195b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T11:52:14.052 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 11:52:13 smithi064 bash[70582]: a18f861b4e00de151e1f3e658101d5414e4ede25b6f542b1403dbf18e0ca195b 2024-03-24T11:52:14.052 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 11:52:13 smithi064 systemd[1]: Started Ceph rgw.foo.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:52:14.130 INFO:teuthology.orchestra.run.smithi193.stderr:pool 'datapool' created 2024-03-24T11:52:14.433 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:14 smithi193 ceph-mon[33593]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:52:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:14 smithi064 ceph-mon[30223]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:52:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:14 smithi064 ceph-mon[42243]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:52:14.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- rbd pool init datapool 2024-03-24T11:52:14.968 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/3872915115' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:52:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:52:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-24T11:52:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:52:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:52:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3344403864' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:15 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/3872915115' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:52:15.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3344403864' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/3872915115' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:52:15.338 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3344403864' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:15.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:15 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:16.325 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:16 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3344403864' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:52:16.326 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:16 smithi193 ceph-mon[33593]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:52:16.326 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:16.326 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:16.326 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:16.326 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:16 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:16 smithi193 ceph-mon[33593]: Checking dashboard <-> RGW credentials 2024-03-24T11:52:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3344403864' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:52:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[30223]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:52:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[30223]: Checking dashboard <-> RGW credentials 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3344403864' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[42243]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:16.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:16 smithi064 ceph-mon[42243]: Checking dashboard <-> RGW credentials 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: pgmap v197: 36 pgs: 4 creating+peering, 5 active+clean, 27 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 255 B/s wr, 0 op/s 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/1020317389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: from='client.24415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:52:17.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[30223]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: pgmap v197: 36 pgs: 4 creating+peering, 5 active+clean, 27 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 255 B/s wr, 0 op/s 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/1020317389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: from='client.24415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:52:17.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:17 smithi064 ceph-mon[42243]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: pgmap v197: 36 pgs: 4 creating+peering, 5 active+clean, 27 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 255 B/s wr, 0 op/s 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/1020317389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: from='client.24415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:52:17.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:17 smithi193 ceph-mon[33593]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:18.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:52:18.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:18 smithi064 ceph-mon[30223]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:52:18.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:18 smithi064 ceph-mon[30223]: from='client.24415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:52:18.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:18 smithi064 ceph-mon[30223]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:52:18.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:52:18.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:18 smithi064 ceph-mon[42243]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:52:18.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:18 smithi064 ceph-mon[42243]: from='client.24415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:52:18.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:18 smithi064 ceph-mon[42243]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:52:18.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:52:18.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:18 smithi193 ceph-mon[33593]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:52:18.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:18 smithi193 ceph-mon[33593]: from='client.24415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:52:18.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:18 smithi193 ceph-mon[33593]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:52:19.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:19 smithi193 ceph-mon[33593]: pgmap v200: 68 pgs: 4 creating+peering, 24 active+clean, 40 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:52:19.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:19 smithi193 ceph-mon[33593]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:52:19.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:19 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T11:52:19.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:19 smithi193 ceph-mon[33593]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:52:19.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:19 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:19.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:19 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:19.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:19 smithi193 ceph-mon[33593]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:19.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[30223]: pgmap v200: 68 pgs: 4 creating+peering, 24 active+clean, 40 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:52:19.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[30223]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:52:19.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T11:52:19.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[30223]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:52:19.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:19.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:19.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[30223]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:19.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[42243]: pgmap v200: 68 pgs: 4 creating+peering, 24 active+clean, 40 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:52:19.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[42243]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:52:19.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T11:52:19.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[42243]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:52:19.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:19.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:19.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:19 smithi064 ceph-mon[42243]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:52:20.172 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.193 --placement '1;smithi193=iscsi.a' 2024-03-24T11:52:20.421 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:20.543 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:20 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:52:20.543 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:20 smithi193 ceph-mon[33593]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:52:20.543 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:20 smithi193 ceph-mon[33593]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:52:20.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:20 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:52:20.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:20 smithi064 ceph-mon[30223]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:52:20.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:20 smithi064 ceph-mon[30223]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:52:20.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:52:20.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:20 smithi064 ceph-mon[42243]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:52:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:20 smithi064 ceph-mon[42243]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: 1.0 deep-scrub starts 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: 1.0 deep-scrub ok 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: pgmap v203: 100 pgs: 13 creating+peering, 63 active+clean, 24 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:21.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: 1.0 deep-scrub starts 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: 1.0 deep-scrub ok 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: pgmap v203: 100 pgs: 13 creating+peering, 63 active+clean, 24 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:21.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: 1.0 deep-scrub starts 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: 1.0 deep-scrub ok 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: pgmap v203: 100 pgs: 13 creating+peering, 63 active+clean, 24 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:52:22.391 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled iscsi.datapool update... 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:52:22.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:22 smithi193 ceph-mon[33593]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[42243]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:52:22.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3099653627' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: from='client.24379 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: from='client.24379 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2984224680' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:22 smithi064 ceph-mon[30223]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:52:23.080 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-24T11:52:23.080 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:52:23.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-24T11:52:23.111 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:52:23.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-24T11:52:23.144 DEBUG:teuthology.orchestra.run.smithi193:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@iscsi.iscsi.a.service 2024-03-24T11:52:23.186 INFO:tasks.cephadm:Adding prometheus.a on smithi193 2024-03-24T11:52:23.186 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi193=a' 2024-03-24T11:52:23.657 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: pgmap v206: 132 pgs: 20 creating+peering, 80 active+clean, 32 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:52:23.657 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: from='client.24421 v1:172.21.15.193:0/2403062526' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.193", "placement": "1;smithi193=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:23.657 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: Saving service iscsi.datapool spec with placement smithi193=iscsi.a;count:1 2024-03-24T11:52:23.657 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.657 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.658 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.658 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.658 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:52:23.658 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:52:23.658 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:23.658 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:23 smithi193 ceph-mon[33593]: Deploying daemon iscsi.iscsi.a on smithi193 2024-03-24T11:52:23.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: pgmap v206: 132 pgs: 20 creating+peering, 80 active+clean, 32 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:52:23.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: from='client.24421 v1:172.21.15.193:0/2403062526' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.193", "placement": "1;smithi193=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:23.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: Saving service iscsi.datapool spec with placement smithi193=iscsi.a;count:1 2024-03-24T11:52:23.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[30223]: Deploying daemon iscsi.iscsi.a on smithi193 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: pgmap v206: 132 pgs: 20 creating+peering, 80 active+clean, 32 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: from='client.24421 v1:172.21.15.193:0/2403062526' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.193", "placement": "1;smithi193=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: Saving service iscsi.datapool spec with placement smithi193=iscsi.a;count:1 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:23.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:52:23.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:52:23.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:23.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:23 smithi064 ceph-mon[42243]: Deploying daemon iscsi.iscsi.a on smithi193 2024-03-24T11:52:23.891 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:25.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:25 smithi193 ceph-mon[33593]: pgmap v208: 132 pgs: 7 creating+peering, 123 active+clean, 2 unknown; 581 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.9 KiB/s wr, 140 op/s 2024-03-24T11:52:25.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:25 smithi064 ceph-mon[30223]: pgmap v208: 132 pgs: 7 creating+peering, 123 active+clean, 2 unknown; 581 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.9 KiB/s wr, 140 op/s 2024-03-24T11:52:25.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:25 smithi064 ceph-mon[42243]: pgmap v208: 132 pgs: 7 creating+peering, 123 active+clean, 2 unknown; 581 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.9 KiB/s wr, 140 op/s 2024-03-24T11:52:26.077 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:25 smithi193 podman[62862]: 2024-03-24 11:52:25.851245171 +0000 UTC m=+0.018971626 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:52:26.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:26 smithi193 podman[62862]: 2024-03-24 11:52:26.407287635 +0000 UTC m=+0.575014079 container init 16a9c3f9bafb55e64fc964ee88b2122a85d3345d49face013590a19e4bfdf202 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T11:52:26.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:26 smithi193 podman[62862]: 2024-03-24 11:52:26.413041015 +0000 UTC m=+0.580767453 container start 16a9c3f9bafb55e64fc964ee88b2122a85d3345d49face013590a19e4bfdf202 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T11:52:26.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:26 smithi193 bash[62862]: 16a9c3f9bafb55e64fc964ee88b2122a85d3345d49face013590a19e4bfdf202 2024-03-24T11:52:26.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:26 smithi193 systemd[1]: Started Ceph iscsi.iscsi.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:52:26.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug Started the configuration object watcher 2024-03-24T11:52:26.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug Checking for config object changes every 1s 2024-03-24T11:52:26.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug Processing osd blocklist entries for this node 2024-03-24T11:52:27.009 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled prometheus update... 2024-03-24T11:52:27.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug Reading the configuration object to update local LIO configuration 2024-03-24T11:52:27.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug Configuration does not have an entry for this host(smithi193.front.sepia.ceph.com) - nothing to define to LIO 2024-03-24T11:52:27.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-24T11:52:27.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: pgmap v209: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 5.8 KiB/s wr, 153 op/s 2024-03-24T11:52:27.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: from='client.24466 v1:172.21.15.193:0/1954834267' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: Saving service prometheus spec with placement smithi193=a;count:1 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:27.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:27 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/3031130581' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:52:27.998 DEBUG:teuthology.orchestra.run.smithi193:prometheus.a> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-24T11:52:28.001 INFO:tasks.cephadm:Adding node-exporter.a on smithi064 2024-03-24T11:52:28.001 INFO:tasks.cephadm:Adding node-exporter.b on smithi193 2024-03-24T11:52:28.001 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi064=a;smithi193=b' 2024-03-24T11:52:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: pgmap v209: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 5.8 KiB/s wr, 153 op/s 2024-03-24T11:52:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:52:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: from='client.24466 v1:172.21.15.193:0/1954834267' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: Saving service prometheus spec with placement smithi193=a;count:1 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/3031130581' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: pgmap v209: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 5.8 KiB/s wr, 153 op/s 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:28.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: from='client.24466 v1:172.21.15.193:0/1954834267' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:28.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: Saving service prometheus spec with placement smithi193=a;count:1 2024-03-24T11:52:28.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:28.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:27 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/3031130581' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:52:28.791 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:28.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:28 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:52:28.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:28 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:52:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:28 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:52:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:28 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:52:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:28 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:52:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:28 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:52:29.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:29 smithi193 ceph-mon[33593]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.2 KiB/s wr, 166 op/s 2024-03-24T11:52:29.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:29 smithi193 ceph-mon[33593]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:52:29.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:29 smithi193 ceph-mon[33593]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:52:29.953 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:29 smithi064 ceph-mon[30223]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.2 KiB/s wr, 166 op/s 2024-03-24T11:52:29.953 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:29 smithi064 ceph-mon[30223]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:52:29.954 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:29 smithi064 ceph-mon[30223]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:52:29.954 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:29 smithi064 ceph-mon[42243]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.2 KiB/s wr, 166 op/s 2024-03-24T11:52:29.954 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:29 smithi064 ceph-mon[42243]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:52:29.954 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:29 smithi064 ceph-mon[42243]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:52:30.891 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled node-exporter update... 2024-03-24T11:52:32.177 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.7 KiB/s wr, 146 op/s 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: from='client.24478 v1:172.21.15.193:0/2613000900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi064=a;smithi193=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: Saving service node-exporter spec with placement smithi064=a;smithi193=b;count:2 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:32.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:31 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.7 KiB/s wr, 146 op/s 2024-03-24T11:52:32.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: from='client.24478 v1:172.21.15.193:0/2613000900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi064=a;smithi193=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:32.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: Saving service node-exporter spec with placement smithi064=a;smithi193=b;count:2 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.7 KiB/s wr, 146 op/s 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: from='client.24478 v1:172.21.15.193:0/2613000900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi064=a;smithi193=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: Saving service node-exporter spec with placement smithi064=a;smithi193=b;count:2 2024-03-24T11:52:32.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.212 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.212 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.212 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.212 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:32.212 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:32.212 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:31 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:32.213 DEBUG:teuthology.orchestra.run.smithi064:node-exporter.a> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-24T11:52:32.216 DEBUG:teuthology.orchestra.run.smithi193:node-exporter.b> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-24T11:52:32.218 INFO:tasks.cephadm:Adding alertmanager.a on smithi064 2024-03-24T11:52:32.218 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi064=a' 2024-03-24T11:52:32.499 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:33.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:32 smithi064 ceph-mon[30223]: Deploying daemon prometheus.a on smithi193 2024-03-24T11:52:33.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:32 smithi064 ceph-mon[42243]: Deploying daemon prometheus.a on smithi193 2024-03-24T11:52:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:32 smithi193 ceph-mon[33593]: Deploying daemon prometheus.a on smithi193 2024-03-24T11:52:34.257 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:33 smithi193 ceph-mon[33593]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.4 KiB/s wr, 60 op/s 2024-03-24T11:52:34.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:33 smithi064 ceph-mon[30223]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.4 KiB/s wr, 60 op/s 2024-03-24T11:52:34.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:33 smithi064 ceph-mon[42243]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.4 KiB/s wr, 60 op/s 2024-03-24T11:52:34.772 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled alertmanager update... 2024-03-24T11:52:35.999 DEBUG:teuthology.orchestra.run.smithi064:alertmanager.a> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-24T11:52:36.002 INFO:tasks.cephadm:Adding grafana.a on smithi193 2024-03-24T11:52:36.002 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi193=a' 2024-03-24T11:52:36.027 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:35 smithi193 ceph-mon[33593]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 28 op/s 2024-03-24T11:52:36.027 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:35 smithi193 ceph-mon[33593]: from='client.24484 v1:172.21.15.193:0/4036308869' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:36.027 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:35 smithi193 ceph-mon[33593]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-03-24T11:52:36.027 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:35 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:36.027 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:35 smithi064 ceph-mon[30223]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 28 op/s 2024-03-24T11:52:36.027 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:35 smithi064 ceph-mon[30223]: from='client.24484 v1:172.21.15.193:0/4036308869' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:36.028 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:35 smithi064 ceph-mon[30223]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-03-24T11:52:36.028 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:35 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:36.028 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:35 smithi064 ceph-mon[42243]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 28 op/s 2024-03-24T11:52:36.028 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:35 smithi064 ceph-mon[42243]: from='client.24484 v1:172.21.15.193:0/4036308869' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:36.028 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:35 smithi064 ceph-mon[42243]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-03-24T11:52:36.028 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:35 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:36.238 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:37.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:52:37.023 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:36 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-03-24T11:52:37.023 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:36 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-24T11:52:37.023 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:36 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-24T11:52:37.023 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:36 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-24T11:52:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:36 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-03-24T11:52:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:36 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-24T11:52:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:36 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-24T11:52:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:36 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-24T11:52:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-03-24T11:52:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-24T11:52:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-24T11:52:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:36 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-24T11:52:37.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:37 smithi193 ceph-mon[33593]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 215 B/s wr, 11 op/s 2024-03-24T11:52:37.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:37 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:37.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]': finished 2024-03-24T11:52:37.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-24T11:52:37.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-24T11:52:37.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:37 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-24T11:52:37.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:37 smithi193 ceph-mon[33593]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:52:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[30223]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 215 B/s wr, 11 op/s 2024-03-24T11:52:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]': finished 2024-03-24T11:52:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-24T11:52:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-24T11:52:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-24T11:52:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[30223]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:52:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[42243]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 215 B/s wr, 11 op/s 2024-03-24T11:52:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]': finished 2024-03-24T11:52:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-24T11:52:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-24T11:52:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-24T11:52:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:37 smithi064 ceph-mon[42243]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:52:38.962 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled grafana update... 2024-03-24T11:52:38.962 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:38 smithi193 ceph-mon[33593]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:52:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:38 smithi064 ceph-mon[30223]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:52:39.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:38 smithi064 ceph-mon[42243]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:52:40.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:39 smithi064 ceph-mon[30223]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:52:40.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:39 smithi064 ceph-mon[30223]: from='client.24490 v1:172.21.15.193:0/4204998709' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:40.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:39 smithi064 ceph-mon[30223]: Saving service grafana spec with placement smithi193=a;count:1 2024-03-24T11:52:40.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:39 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:39 smithi064 ceph-mon[42243]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:52:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:39 smithi064 ceph-mon[42243]: from='client.24490 v1:172.21.15.193:0/4204998709' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:40.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:39 smithi064 ceph-mon[42243]: Saving service grafana spec with placement smithi193=a;count:1 2024-03-24T11:52:40.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:39 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:39 smithi193 ceph-mon[33593]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:52:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:39 smithi193 ceph-mon[33593]: from='client.24490 v1:172.21.15.193:0/4204998709' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:39 smithi193 ceph-mon[33593]: Saving service grafana spec with placement smithi193=a;count:1 2024-03-24T11:52:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:39 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:40.602 DEBUG:teuthology.orchestra.run.smithi193:grafana.a> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@grafana.a.service 2024-03-24T11:52:40.644 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T11:52:40.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:52:40.901 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:42.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:41 smithi064 ceph-mon[30223]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 121 B/s, 1 objects/s recovering 2024-03-24T11:52:42.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:41 smithi064 ceph-mon[42243]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 121 B/s, 1 objects/s recovering 2024-03-24T11:52:42.269 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:41 smithi193 ceph-mon[33593]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 121 B/s, 1 objects/s recovering 2024-03-24T11:52:42.932 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-03-24T11:52:42.932 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAKFABmx9x3NxAAQB5Ps/laftDfeHOyHQQhTA== 2024-03-24T11:52:43.145 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3955365753' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:43.145 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3955365753' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:52:43.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3955365753' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:43.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3955365753' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:52:43.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3955365753' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:43.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3955365753' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:52:43.641 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-24T11:52:43.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T11:52:43.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T11:52:43.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:52:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:43 smithi193 ceph-mon[33593]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 149 B/s, 2 objects/s recovering 2024-03-24T11:52:44.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:43 smithi064 ceph-mon[30223]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 149 B/s, 2 objects/s recovering 2024-03-24T11:52:44.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:43 smithi064 ceph-mon[42243]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 149 B/s, 2 objects/s recovering 2024-03-24T11:52:44.781 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:52:46.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:45 smithi193 ceph-mon[33593]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 149 B/s, 2 objects/s recovering 2024-03-24T11:52:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:45 smithi064 ceph-mon[30223]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 149 B/s, 2 objects/s recovering 2024-03-24T11:52:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:45 smithi064 ceph-mon[42243]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 149 B/s, 2 objects/s recovering 2024-03-24T11:52:46.336 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 podman[64807]: 2024-03-24 11:52:46.246690341 +0000 UTC m=+0.147725678 container create e39a1ce87a77071da96a4227ef42f3e9ba882ffdb77049b4dd147b828347881f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:52:46.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:52:46.739 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 podman[64807]: 2024-03-24 11:52:46.684874444 +0000 UTC m=+0.585909797 container init e39a1ce87a77071da96a4227ef42f3e9ba882ffdb77049b4dd147b828347881f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:52:46.739 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 podman[64807]: 2024-03-24 11:52:46.690663718 +0000 UTC m=+0.591699055 container start e39a1ce87a77071da96a4227ef42f3e9ba882ffdb77049b4dd147b828347881f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:52:46.739 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.715Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:52:46.740 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.715Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:52:46.740 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.715Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi193 (none))" 2024-03-24T11:52:46.740 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.715Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:52:46.740 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.715Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:52:46.740 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.716Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:52:46.740 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.716Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:52:46.741 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.717Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:52:46.741 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.717Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:52:46.741 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.719Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:52:46.741 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.719Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.618µs 2024-03-24T11:52:46.741 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.719Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:52:46.741 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.720Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-24T11:52:46.742 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.720Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=33.565µs wal_replay_duration=1.068272ms wbl_replay_duration=121ns total_replay_duration=1.198534ms 2024-03-24T11:52:46.742 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.721Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:52:46.742 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.721Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:52:46.742 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.721Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:52:46.742 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.738Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.29487ms db_storage=730ns remote_storage=1.185µs web_handler=369ns query_engine=554ns scrape=391.471µs scrape_sd=71.863µs notify=1.047µs notify_sd=1.912µs rules=16.638142ms tracing=5.478µs 2024-03-24T11:52:46.742 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.738Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:52:46.743 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:52:46.738Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:52:46.993 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 bash[64807]: e39a1ce87a77071da96a4227ef42f3e9ba882ffdb77049b4dd147b828347881f 2024-03-24T11:52:46.993 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:52:46 smithi193 systemd[1]: Started Ceph prometheus.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:52:46.994 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:52:46.994 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:52:46.994 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-mon[33593]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:52:46.994 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:46.994 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:46.994 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:46.994 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:46 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:52:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:52:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:52:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[30223]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:52:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[42243]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' 2024-03-24T11:52:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:46 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:52:47.393 INFO:teuthology.orchestra.run.smithi193.stdout:[client.1] 2024-03-24T11:52:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAPFABmh+wWFxAArA8d62CxqOLDTR7+2aEVdA== 2024-03-24T11:52:47.990 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: ignoring --setuser ceph since I am not root 2024-03-24T11:52:47.990 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: ignoring --setgroup ceph since I am not root 2024-03-24T11:52:48.101 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T11:52:48.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T11:52:48.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T11:52:48.144 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T11:52:48.145 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T11:52:48.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T11:52:48.302 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ignoring --setuser ceph since I am not root 2024-03-24T11:52:48.302 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ignoring --setgroup ceph since I am not root 2024-03-24T11:52:48.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[30223]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 369 B/s rd, 0 op/s; 143 B/s, 2 objects/s recovering 2024-03-24T11:52:48.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:48.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.193:0/96763005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:48.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[30223]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[30223]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[30223]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[30223]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[42243]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 369 B/s rd, 0 op/s; 143 B/s, 2 objects/s recovering 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.193:0/96763005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[42243]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[42243]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[42243]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:52:48.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:47 smithi064 ceph-mon[42243]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:52:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-mon[33593]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 369 B/s rd, 0 op/s; 143 B/s, 2 objects/s recovering 2024-03-24T11:52:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.193:0/96763005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-mon[33593]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:52:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-mon[33593]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:52:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-mon[33593]: from='mgr.14152 v1:172.21.15.64:0/3318477317' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:52:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:47 smithi193 ceph-mon[33593]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:52:48.393 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:48.754 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:48 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:48.493+0000 7f5630b30200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:52:48.754 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:48 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:48.589+0000 7f5630b30200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:52:48.836 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:48.466+0000 7f9e97671200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:52:48.836 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:48.558+0000 7f9e97671200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:52:49.302 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:49.260+0000 7f5630b30200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:52:49.529 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:49.196+0000 7f9e97671200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:52:49.529 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:49.289+0000 7f9e97671200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:52:49.529 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:49.434+0000 7f9e97671200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:52:49.530 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:49.529+0000 7f9e97671200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:52:49.687 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:49.356+0000 7f5630b30200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:52:49.687 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:49.502+0000 7f5630b30200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:52:49.687 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:49.601+0000 7f5630b30200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:52:49.687 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:49.686+0000 7f5630b30200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:52:49.794 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:49.616+0000 7f9e97671200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:52:50.086 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:49.892+0000 7f9e97671200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:52:50.086 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:50.009+0000 7f9e97671200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:52:50.263 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:49 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:49.961+0000 7f5630b30200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:52:50.263 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:50.074+0000 7f5630b30200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:52:50.263 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:50.166+0000 7f5630b30200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:52:50.391 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:50.098+0000 7f9e97671200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:52:50.391 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:50.196+0000 7f9e97671200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:52:50.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:50.265+0000 7f5630b30200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:52:50.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:50.449+0000 7f5630b30200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:52:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:52:50.696 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:50.390+0000 7f9e97671200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:52:50.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:50 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1251698061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:52:50.958 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:50 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1251698061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:52:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:50 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1251698061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:52:51.405 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6800","nonce":3538503390}]},"active_addr":"172.21.15.64:6800/3538503390","active_change":"2024-03-24T11:47:36.035779+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.64:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1459540658}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1277626303}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3351953651}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3753895773}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3407451558}]}]} 2024-03-24T11:52:51.409 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T11:52:51.409 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:52:51.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:52:51.664 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:51.786 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:51.449+0000 7f5630b30200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:52:51.786 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:51.589+0000 7f5630b30200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:52:51.822 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:51.353+0000 7f9e97671200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:52:51.822 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:51.486+0000 7f9e97671200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:52:51.822 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:51.821+0000 7f9e97671200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:52:52.052 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:51.962+0000 7f5630b30200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:52:52.086 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:51.911+0000 7f9e97671200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:52:52.459 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:52.058+0000 7f5630b30200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:52:52.460 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:52.264+0000 7f5630b30200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:52:52.577 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:52.106+0000 7f9e97671200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:52:52.577 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:52.290+0000 7f9e97671200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:52:52.732 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:52.459+0000 7f5630b30200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:52:52.836 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:52.576+0000 7f9e97671200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:52:53.021 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:52 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:52.756+0000 7f5630b30200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:52:53.265 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:52.838+0000 7f9e97671200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:52:53.265 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:53.071+0000 7f9e97671200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:52:53.294 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:53.020+0000 7f5630b30200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:52:53.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:53 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:52:53.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:53 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:52:53.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:53 smithi064 ceph-mon[30223]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:52:53.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:53 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:52:53.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:53 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:52:53.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:53 smithi064 ceph-mon[42243]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:52:53.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:53.294+0000 7f5630b30200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:52:53.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:53 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:52:53.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:53 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:52:53.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:53 smithi193 ceph-mon[33593]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:52:53.806 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:52:53.806 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":80,"fsid":"06f23456-e9d4-11ee-95cd-87774f69a715","created":"2024-03-24T11:46:40.657065+0000","modified":"2024-03-24T11:52:52.263530+0000","last_up_change":"2024-03-24T11:52:01.590830+0000","last_in_change":"2024-03-24T11:51:43.667453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:49:56.369674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T11:52:13.871974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-24T11:52:14.117018+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:52:15.453744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:52:17.611004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:52:19.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc14a64f-9e6c-41f1-901e-c49fabffb53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2675992571}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2675992571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2675992571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2675992571}]},"public_addr":"172.21.15.64:6801/2675992571","cluster_addr":"172.21.15.64:6802/2675992571","heartbeat_back_addr":"172.21.15.64:6804/2675992571","heartbeat_front_addr":"172.21.15.64:6803/2675992571","state":["exists","up"]},{"osd":1,"uuid":"74587332-f0b2-4905-9888-7bdb604cc0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":841667271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":841667271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":841667271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":841667271}]},"public_addr":"172.21.15.64:6805/841667271","cluster_addr":"172.21.15.64:6806/841667271","heartbeat_back_addr":"172.21.15.64:6808/841667271","heartbeat_front_addr":"172.21.15.64:6807/841667271","state":["exists","up"]},{"osd":2,"uuid":"a0c0ed85-9f33-4198-a1f2-447eb17c364b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":435147390}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":435147390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":435147390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":435147390}]},"public_addr":"172.21.15.64:6809/435147390","cluster_addr":"172.21.15.64:6810/435147390","heartbeat_back_addr":"172.21.15.64:6812/435147390","heartbeat_front_addr":"172.21.15.64:6811/435147390","state":["exists","up"]},{"osd":3,"uuid":"1396f66c-3e0c-4a01-b0a7-6f763989a479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3355318038}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3355318038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3355318038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3355318038}]},"public_addr":"172.21.15.64:6813/3355318038","cluster_addr":"172.21.15.64:6814/3355318038","heartbeat_back_addr":"172.21.15.64:6816/3355318038","heartbeat_front_addr":"172.21.15.64:6815/3355318038","state":["exists","up"]},{"osd":4,"uuid":"50d65d8b-3e90-46a4-b19d-50c3015af40b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3851339176}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3851339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3851339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3851339176}]},"public_addr":"172.21.15.193:6800/3851339176","cluster_addr":"172.21.15.193:6801/3851339176","heartbeat_back_addr":"172.21.15.193:6803/3851339176","heartbeat_front_addr":"172.21.15.193:6802/3851339176","state":["exists","up"]},{"osd":5,"uuid":"50c5d50e-7e68-487c-9f68-1abb8c927ff6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":1625306337}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":1625306337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":1625306337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":1625306337}]},"public_addr":"172.21.15.193:6804/1625306337","cluster_addr":"172.21.15.193:6805/1625306337","heartbeat_back_addr":"172.21.15.193:6807/1625306337","heartbeat_front_addr":"172.21.15.193:6806/1625306337","state":["exists","up"]},{"osd":6,"uuid":"7e74fe39-5707-4c90-ad38-1083cd7a6996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":2408237278}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":2408237278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":2408237278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":2408237278}]},"public_addr":"172.21.15.193:6808/2408237278","cluster_addr":"172.21.15.193:6809/2408237278","heartbeat_back_addr":"172.21.15.193:6811/2408237278","heartbeat_front_addr":"172.21.15.193:6810/2408237278","state":["exists","up"]},{"osd":7,"uuid":"e568df64-cc9a-43f8-a432-8628d1da1015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":2649989609}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":2649989609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":2649989609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":2649989609}]},"public_addr":"172.21.15.193:6812/2649989609","cluster_addr":"172.21.15.193:6813/2649989609","heartbeat_back_addr":"172.21.15.193:6815/2649989609","heartbeat_front_addr":"172.21.15.193:6814/2649989609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:29.543497+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:53.392293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:20.269652+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:52.213150+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:52.085478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:27.616713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:45.684029+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:00.443086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":7,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/842059688":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/1610565583":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/2730170969":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/3257265142":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/629947599":"2024-03-25T11:47:08.661692+0000","172.21.15.64:6800/520335082":"2024-03-25T11:47:08.661692+0000","172.21.15.64:6800/4027721305":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/4028269665":"2024-03-25T11:47:36.035658+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:52:54.266 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:53.867+0000 7f9e97671200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:52:54.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1439572602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:54.533 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:54 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:54.148+0000 7f5630b30200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:52:54.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1439572602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:54.537 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1439572602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:54.538 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:54.438+0000 7f9e97671200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:52:54.538 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:54.536+0000 7f9e97671200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:52:54.567 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:52:54.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:52:54.802 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:54 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:54.757+0000 7f5630b30200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:52:54.819 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:54.836 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:54.751+0000 7f9e97671200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:52:54.836 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: [24/Mar/2024:11:52:54] ENGINE Bus STARTING 2024-03-24T11:52:55.215 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:54 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:54.869+0000 7f5630b30200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:52:55.215 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:55.102+0000 7f5630b30200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:52:55.269 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: CherryPy Checker: 2024-03-24T11:52:55.269 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: The Application mounted at '' has an empty config. 2024-03-24T11:52:55.270 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: 2024-03-24T11:52:55.270 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: [24/Mar/2024:11:52:54] ENGINE Serving on http://:::9283 2024-03-24T11:52:55.270 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 11:52:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x[35673]: [24/Mar/2024:11:52:54] ENGINE Bus STARTED 2024-03-24T11:52:55.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:52:55] ENGINE Bus STARTING 2024-03-24T11:52:55.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: CherryPy Checker: 2024-03-24T11:52:55.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: The Application mounted at '' has an empty config. 2024-03-24T11:52:55.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T11:52:55.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:52:55] ENGINE Serving on http://:::9283 2024-03-24T11:52:55.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:52:55] ENGINE Bus STARTED 2024-03-24T11:52:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: Standby manager daemon x restarted 2024-03-24T11:52:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: Standby manager daemon x started 2024-03-24T11:52:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:52:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: Active manager daemon y restarted 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: Activating manager daemon y 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: mgrmap e18: y(active, starting, since 0.011506s), standbys: x 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:52:55.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: Manager daemon y is now available 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:55.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: Standby manager daemon x restarted 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: Standby manager daemon x started 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: Active manager daemon y restarted 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: Activating manager daemon y 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:52:55.556 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: mgrmap e18: y(active, starting, since 0.011506s), standbys: x 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:55.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: Manager daemon y is now available 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:55.558 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:55.559 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:52:55.559 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:52:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: Standby manager daemon x restarted 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: Standby manager daemon x started 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.? v1:172.21.15.193:0/1040814026' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: Active manager daemon y restarted 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: Activating manager daemon y 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:52:55.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: mgrmap e18: y(active, starting, since 0.011506s), standbys: x 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:52:55.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: Manager daemon y is now available 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:52:55.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:52:56.703 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:52:56.703 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":83,"fsid":"06f23456-e9d4-11ee-95cd-87774f69a715","created":"2024-03-24T11:46:40.657065+0000","modified":"2024-03-24T11:52:56.602017+0000","last_up_change":"2024-03-24T11:52:01.590830+0000","last_in_change":"2024-03-24T11:51:43.667453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:49:56.369674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T11:52:13.871974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-24T11:52:14.117018+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:52:15.453744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:52:17.611004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:52:19.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc14a64f-9e6c-41f1-901e-c49fabffb53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2675992571}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2675992571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2675992571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2675992571}]},"public_addr":"172.21.15.64:6801/2675992571","cluster_addr":"172.21.15.64:6802/2675992571","heartbeat_back_addr":"172.21.15.64:6804/2675992571","heartbeat_front_addr":"172.21.15.64:6803/2675992571","state":["exists","up"]},{"osd":1,"uuid":"74587332-f0b2-4905-9888-7bdb604cc0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":841667271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":841667271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":841667271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":841667271}]},"public_addr":"172.21.15.64:6805/841667271","cluster_addr":"172.21.15.64:6806/841667271","heartbeat_back_addr":"172.21.15.64:6808/841667271","heartbeat_front_addr":"172.21.15.64:6807/841667271","state":["exists","up"]},{"osd":2,"uuid":"a0c0ed85-9f33-4198-a1f2-447eb17c364b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":435147390}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":435147390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":435147390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":435147390}]},"public_addr":"172.21.15.64:6809/435147390","cluster_addr":"172.21.15.64:6810/435147390","heartbeat_back_addr":"172.21.15.64:6812/435147390","heartbeat_front_addr":"172.21.15.64:6811/435147390","state":["exists","up"]},{"osd":3,"uuid":"1396f66c-3e0c-4a01-b0a7-6f763989a479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3355318038}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3355318038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3355318038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3355318038}]},"public_addr":"172.21.15.64:6813/3355318038","cluster_addr":"172.21.15.64:6814/3355318038","heartbeat_back_addr":"172.21.15.64:6816/3355318038","heartbeat_front_addr":"172.21.15.64:6815/3355318038","state":["exists","up"]},{"osd":4,"uuid":"50d65d8b-3e90-46a4-b19d-50c3015af40b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3851339176}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3851339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3851339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3851339176}]},"public_addr":"172.21.15.193:6800/3851339176","cluster_addr":"172.21.15.193:6801/3851339176","heartbeat_back_addr":"172.21.15.193:6803/3851339176","heartbeat_front_addr":"172.21.15.193:6802/3851339176","state":["exists","up"]},{"osd":5,"uuid":"50c5d50e-7e68-487c-9f68-1abb8c927ff6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":1625306337}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":1625306337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":1625306337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":1625306337}]},"public_addr":"172.21.15.193:6804/1625306337","cluster_addr":"172.21.15.193:6805/1625306337","heartbeat_back_addr":"172.21.15.193:6807/1625306337","heartbeat_front_addr":"172.21.15.193:6806/1625306337","state":["exists","up"]},{"osd":6,"uuid":"7e74fe39-5707-4c90-ad38-1083cd7a6996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":2408237278}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":2408237278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":2408237278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":2408237278}]},"public_addr":"172.21.15.193:6808/2408237278","cluster_addr":"172.21.15.193:6809/2408237278","heartbeat_back_addr":"172.21.15.193:6811/2408237278","heartbeat_front_addr":"172.21.15.193:6810/2408237278","state":["exists","up"]},{"osd":7,"uuid":"e568df64-cc9a-43f8-a432-8628d1da1015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":2649989609}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":2649989609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":2649989609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":2649989609}]},"public_addr":"172.21.15.193:6812/2649989609","cluster_addr":"172.21.15.193:6813/2649989609","heartbeat_back_addr":"172.21.15.193:6815/2649989609","heartbeat_front_addr":"172.21.15.193:6814/2649989609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:29.543497+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:51:53.392293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:20.269652+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:52.213150+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:56.048317+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:27.616713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:45.684029+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:52:00.443086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":7,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6800/3538503390":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/1277626303":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/3407451558":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/3753895773":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/842059688":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/1610565583":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/2730170969":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/3257265142":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/1459540658":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/629947599":"2024-03-25T11:47:08.661692+0000","172.21.15.64:6800/520335082":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/3351953651":"2024-03-25T11:52:55.117233+0000","172.21.15.64:6800/4027721305":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/4028269665":"2024-03-25T11:47:36.035658+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:52:57.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:52:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: mgrmap e19: y(active, since 1.01024s), standbys: x 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: [24/Mar/2024:11:52:56] ENGINE Bus STARTING 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3296881317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:57.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: [24/Mar/2024:11:52:56] ENGINE Serving on https://172.21.15.64:7150 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: [24/Mar/2024:11:52:56] ENGINE Serving on http://172.21.15.64:8765 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[30223]: [24/Mar/2024:11:52:56] ENGINE Bus STARTED 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: mgrmap e19: y(active, since 1.01024s), standbys: x 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: [24/Mar/2024:11:52:56] ENGINE Bus STARTING 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3296881317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: [24/Mar/2024:11:52:56] ENGINE Serving on https://172.21.15.64:7150 2024-03-24T11:52:57.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: [24/Mar/2024:11:52:56] ENGINE Serving on http://172.21.15.64:8765 2024-03-24T11:52:57.343 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-mon[42243]: [24/Mar/2024:11:52:56] ENGINE Bus STARTED 2024-03-24T11:52:57.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:52:57.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:52:57.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:52:57.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T11:52:57.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T11:52:57.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T11:52:57.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T11:52:57.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T11:52:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:52:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:52:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: mgrmap e19: y(active, since 1.01024s), standbys: x 2024-03-24T11:52:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:52:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:52:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: [24/Mar/2024:11:52:56] ENGINE Bus STARTING 2024-03-24T11:52:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3296881317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:52:57.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:57.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: [24/Mar/2024:11:52:56] ENGINE Serving on https://172.21.15.64:7150 2024-03-24T11:52:57.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: [24/Mar/2024:11:52:56] ENGINE Serving on http://172.21.15.64:8765 2024-03-24T11:52:57.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:57 smithi193 ceph-mon[33593]: [24/Mar/2024:11:52:56] ENGINE Bus STARTED 2024-03-24T11:52:57.802 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:52:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:52:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:52:57.881 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:57.966 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:57.970 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:58.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:58 smithi064 ceph-mon[30223]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:58.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:58 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:52:58.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:58 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:52:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:58 smithi064 ceph-mon[42243]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:58 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:52:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:58 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:52:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:58 smithi193 ceph-mon[33593]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:58 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:52:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:58 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:52:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:58.928 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:59 smithi064 ceph-mon[42243]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:52:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:59 smithi064 ceph-mon[42243]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-24T11:52:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:59 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:52:59 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:59 smithi064 ceph-mon[30223]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:52:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:59 smithi064 ceph-mon[30223]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-24T11:52:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:59 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:52:59 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:59 smithi193 ceph-mon[33593]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:52:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:59 smithi193 ceph-mon[33593]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-24T11:52:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:59 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:52:59 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:52:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:52:59.759 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi193 to 3760M 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[30223]: mgrmap e21: y(active, since 5s), standbys: x 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi193 to 3760M 2024-03-24T11:53:00.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:00 smithi064 ceph-mon[42243]: mgrmap e21: y(active, since 5s), standbys: x 2024-03-24T11:53:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:00.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi193 to 3760M 2024-03-24T11:53:00.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:00.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:00 smithi193 ceph-mon[33593]: mgrmap e21: y(active, since 5s), standbys: x 2024-03-24T11:53:02.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:02 smithi064 ceph-mon[30223]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T11:53:02.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:02 smithi064 ceph-mon[30223]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:53:02.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:02 smithi064 ceph-mon[42243]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T11:53:02.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:02 smithi064 ceph-mon[42243]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:53:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:02 smithi193 ceph-mon[33593]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T11:53:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:02 smithi193 ceph-mon[33593]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:53:04.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:04 smithi064 ceph-mon[42243]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-24T11:53:04.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:04 smithi064 ceph-mon[30223]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-24T11:53:05.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:04 smithi193 ceph-mon[33593]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-24T11:53:06.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:06 smithi064 ceph-mon[30223]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-24T11:53:06.878 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:06 smithi064 ceph-mon[30223]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:53:06.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:06 smithi064 ceph-mon[42243]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-24T11:53:06.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:06 smithi064 ceph-mon[42243]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:53:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:06 smithi193 ceph-mon[33593]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-24T11:53:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:06 smithi193 ceph-mon[33593]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:53:07.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:53:06 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:53:07.510 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:53:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:07.767 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:07 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:07.767 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:07 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:08.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:07 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:08.279 INFO:teuthology.orchestra.run.smithi064.stdout:261993005070 2024-03-24T11:53:08.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T11:53:08.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:08 smithi064 ceph-mon[30223]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-24T11:53:08.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:08 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:53:08.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:08 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:53:08.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:08 smithi064 ceph-mon[30223]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:53:08.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:08 smithi064 ceph-mon[42243]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-24T11:53:08.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:08 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:53:08.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:08 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:53:08.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:08 smithi064 ceph-mon[42243]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:53:08.917 INFO:teuthology.orchestra.run.smithi064.stdout:34359738418 2024-03-24T11:53:08.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:53:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:08 smithi193 ceph-mon[33593]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-24T11:53:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:08 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:53:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:08 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:53:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:08 smithi193 ceph-mon[33593]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:53:10.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:10 smithi064 ceph-mon[30223]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-03-24T11:53:10.918 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:10.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:10 smithi064 ceph-mon[42243]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-03-24T11:53:10.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:11.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:10 smithi193 ceph-mon[33593]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-03-24T11:53:11.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:11.504 INFO:teuthology.orchestra.run.smithi064.stdout:77309411369 2024-03-24T11:53:11.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:53:11.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:11.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:11.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:11 smithi064 ceph-mon[42243]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:53:11.732 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:11.732 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:11.732 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:11 smithi064 ceph-mon[30223]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:53:12.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:12.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:12.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:11 smithi193 ceph-mon[33593]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:53:12.265 INFO:teuthology.orchestra.run.smithi064.stdout:55834574893 2024-03-24T11:53:12.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:53:12.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:12 smithi064 ceph-mon[30223]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:12.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:12 smithi064 ceph-mon[42243]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:13.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:12 smithi193 ceph-mon[33593]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:13.114 INFO:teuthology.orchestra.run.smithi064.stdout:141733920799 2024-03-24T11:53:13.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T11:53:13.797 INFO:teuthology.orchestra.run.smithi064.stdout:176093659162 2024-03-24T11:53:13.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T11:53:14.081 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-03-24T11:53:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:15.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:15.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:53:15.103 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:15.103 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:15.103 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:15.138 INFO:teuthology.orchestra.run.smithi064.stdout:115964117028 2024-03-24T11:53:15.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T11:53:15.546 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:15.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi064 to 3760M 2024-03-24T11:53:15.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-24T11:53:15.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:53:15.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:53:15.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:53:15.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi064 to 3760M 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:53:15.903 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.904 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:15 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:15.928 INFO:teuthology.orchestra.run.smithi064.stdout:219043332117 2024-03-24T11:53:15.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T11:53:16.070 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:16.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi064 to 3760M 2024-03-24T11:53:16.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-24T11:53:16.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T11:53:16.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:53:16.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:53:16.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:53:16.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:53:16.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Updating smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:53:16.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: Updating smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:53:16.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:16.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:16.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:16.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:16.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:15 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:16.200 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:16.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:16 smithi064 ceph-mon[30223]: Deploying daemon alertmanager.a on smithi064 2024-03-24T11:53:16.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:16 smithi064 ceph-mon[30223]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:53:16.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:16 smithi064 ceph-mon[30223]: 2.7 deep-scrub starts 2024-03-24T11:53:16.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:16 smithi064 ceph-mon[30223]: 2.7 deep-scrub ok 2024-03-24T11:53:16.942 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:16 smithi064 ceph-mon[42243]: Deploying daemon alertmanager.a on smithi064 2024-03-24T11:53:16.942 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:16 smithi064 ceph-mon[42243]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:53:16.942 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:16 smithi064 ceph-mon[42243]: 2.7 deep-scrub starts 2024-03-24T11:53:16.942 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:16 smithi064 ceph-mon[42243]: 2.7 deep-scrub ok 2024-03-24T11:53:17.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:53:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:53:17.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:16 smithi193 ceph-mon[33593]: Deploying daemon alertmanager.a on smithi064 2024-03-24T11:53:17.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:16 smithi193 ceph-mon[33593]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:53:17.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:16 smithi193 ceph-mon[33593]: 2.7 deep-scrub starts 2024-03-24T11:53:17.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:16 smithi193 ceph-mon[33593]: 2.7 deep-scrub ok 2024-03-24T11:53:17.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:53:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:18.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:17 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:18.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:17 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:18.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:17 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:18.991 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:18 smithi064 ceph-mon[30223]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:53:18.991 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:18 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T11:53:18.991 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:18 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T11:53:18.991 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:18 smithi064 ceph-mon[42243]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:53:18.991 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:18 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T11:53:18.991 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:18 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T11:53:19.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:18 smithi193 ceph-mon[33593]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:53:19.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:18 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T11:53:19.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:18 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T11:53:19.787 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:19 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T11:53:19.787 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:19 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T11:53:19.787 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:19 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T11:53:19.788 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:19 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T11:53:19.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:19 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T11:53:19.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:19 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T11:53:20.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:20 smithi064 ceph-mon[30223]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:20.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:20 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T11:53:20.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:20 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T11:53:20.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:20 smithi064 ceph-mon[42243]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:20.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:20 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T11:53:20.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:20 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T11:53:21.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:20 smithi193 ceph-mon[33593]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:21.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:20 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T11:53:21.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:20 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T11:53:21.088 INFO:teuthology.orchestra.run.smithi064.stdout:55834574895 2024-03-24T11:53:21.260 INFO:teuthology.orchestra.run.smithi064.stdout:261993005072 2024-03-24T11:53:21.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:21 smithi064 ceph-mon[42243]: 4.1b deep-scrub starts 2024-03-24T11:53:21.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:21 smithi064 ceph-mon[42243]: 4.1b deep-scrub ok 2024-03-24T11:53:21.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/230212913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:53:21.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2290304055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:53:21.817 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:21 smithi064 ceph-mon[30223]: 4.1b deep-scrub starts 2024-03-24T11:53:21.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:21 smithi064 ceph-mon[30223]: 4.1b deep-scrub ok 2024-03-24T11:53:21.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/230212913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:53:21.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2290304055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:53:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:21 smithi193 ceph-mon[33593]: 4.1b deep-scrub starts 2024-03-24T11:53:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:21 smithi193 ceph-mon[33593]: 4.1b deep-scrub ok 2024-03-24T11:53:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/230212913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:53:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2290304055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:53:23.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:22 smithi064 ceph-mon[42243]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:23.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:22 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T11:53:23.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:22 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T11:53:23.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:22 smithi064 ceph-mon[30223]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:23.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:22 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T11:53:23.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:22 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T11:53:23.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:22 smithi193 ceph-mon[33593]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:23.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:22 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T11:53:23.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:22 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T11:53:23.998 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:23 smithi064 ceph-mon[30223]: 5.1e deep-scrub starts 2024-03-24T11:53:23.998 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:23 smithi064 ceph-mon[30223]: 5.1e deep-scrub ok 2024-03-24T11:53:23.998 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:23 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T11:53:23.998 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:23 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T11:53:23.999 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:23 smithi064 ceph-mon[42243]: 5.1e deep-scrub starts 2024-03-24T11:53:23.999 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:23 smithi064 ceph-mon[42243]: 5.1e deep-scrub ok 2024-03-24T11:53:23.999 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:23 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T11:53:23.999 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:23 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T11:53:24.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:23 smithi193 ceph-mon[33593]: 5.1e deep-scrub starts 2024-03-24T11:53:24.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:23 smithi193 ceph-mon[33593]: 5.1e deep-scrub ok 2024-03-24T11:53:24.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:23 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T11:53:24.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:23 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T11:53:25.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:24 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T11:53:25.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:24 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T11:53:25.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:24 smithi064 ceph-mon[30223]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-24T11:53:25.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:24 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1880198540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:53:25.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:24 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T11:53:25.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:24 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T11:53:25.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:24 smithi064 ceph-mon[42243]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-24T11:53:25.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:24 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1880198540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:53:25.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:24 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T11:53:25.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:24 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T11:53:25.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:24 smithi193 ceph-mon[33593]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-24T11:53:25.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:24 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1880198540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:53:25.305 INFO:teuthology.orchestra.run.smithi064.stdout:34359738422 2024-03-24T11:53:26.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T11:53:26.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T11:53:26.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T11:53:26.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T11:53:26.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T11:53:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T11:53:26.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T11:53:26.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T11:53:26.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:25 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:25 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T11:53:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:25 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T11:53:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:25 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T11:53:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:25 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T11:53:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:25 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:26.950 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T11:53:26.950 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T11:53:26.950 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[30223]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:26.950 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T11:53:26.950 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T11:53:26.950 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T11:53:26.950 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T11:53:26.950 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[42243]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:26.951 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T11:53:26.951 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:26 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T11:53:27.057 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738422 for osd.0 2024-03-24T11:53:27.057 DEBUG:teuthology.parallel:result is None 2024-03-24T11:53:27.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:53:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:53:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:26 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T11:53:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:26 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T11:53:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:26 smithi193 ceph-mon[33593]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:26 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T11:53:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:26 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T11:53:27.453 INFO:teuthology.orchestra.run.smithi064.stdout:176093659164 2024-03-24T11:53:27.717 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:53:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[30223]: 6.19 deep-scrub starts 2024-03-24T11:53:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[30223]: 6.19 deep-scrub ok 2024-03-24T11:53:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T11:53:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T11:53:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[30223]: 4.1 deep-scrub starts 2024-03-24T11:53:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[30223]: 4.1 deep-scrub ok 2024-03-24T11:53:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/987774319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:53:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[42243]: 6.19 deep-scrub starts 2024-03-24T11:53:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[42243]: 6.19 deep-scrub ok 2024-03-24T11:53:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T11:53:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T11:53:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[42243]: 4.1 deep-scrub starts 2024-03-24T11:53:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[42243]: 4.1 deep-scrub ok 2024-03-24T11:53:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:28.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:27 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/987774319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:53:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:27 smithi193 ceph-mon[33593]: 6.19 deep-scrub starts 2024-03-24T11:53:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:27 smithi193 ceph-mon[33593]: 6.19 deep-scrub ok 2024-03-24T11:53:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:27 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T11:53:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:27 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T11:53:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:27 smithi193 ceph-mon[33593]: 4.1 deep-scrub starts 2024-03-24T11:53:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:27 smithi193 ceph-mon[33593]: 4.1 deep-scrub ok 2024-03-24T11:53:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:27 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:27 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/987774319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:53:28.099 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659162 got 176093659164 for osd.5 2024-03-24T11:53:28.142 DEBUG:teuthology.parallel:result is None 2024-03-24T11:53:28.688 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T11:53:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T11:53:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[30223]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T11:53:29.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T11:53:29.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T11:53:29.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T11:53:29.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[42243]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:29.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T11:53:29.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:28 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T11:53:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:28 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T11:53:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:28 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T11:53:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:28 smithi193 ceph-mon[33593]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:28 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T11:53:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:28 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T11:53:29.121 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574895 for osd.1 2024-03-24T11:53:29.121 DEBUG:teuthology.parallel:result is None 2024-03-24T11:53:29.794 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:29 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:53:29.794 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:29 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:53:29.794 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:29 smithi064 ceph-mon[30223]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:53:29.795 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:29 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:53:29.795 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:29 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:53:29.795 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:29 smithi064 ceph-mon[42243]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:53:30.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:29 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:53:30.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:29 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:53:30.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:29 smithi193 ceph-mon[33593]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:53:31.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 5.18 deep-scrub starts 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[30223]: 5.18 deep-scrub ok 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T11:53:31.007 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 5.18 deep-scrub starts 2024-03-24T11:53:31.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:30 smithi064 ceph-mon[42243]: 5.18 deep-scrub ok 2024-03-24T11:53:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T11:53:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T11:53:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T11:53:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T11:53:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T11:53:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T11:53:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T11:53:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T11:53:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 5.18 deep-scrub starts 2024-03-24T11:53:31.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:30 smithi193 ceph-mon[33593]: 5.18 deep-scrub ok 2024-03-24T11:53:31.262 INFO:teuthology.orchestra.run.smithi064.stdout:141733920802 2024-03-24T11:53:32.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T11:53:32.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T11:53:32.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T11:53:32.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T11:53:32.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T11:53:32.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T11:53:32.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4006274182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[42243]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T11:53:32.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T11:53:32.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T11:53:32.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T11:53:32.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T11:53:32.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4006274182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:53:32.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:31 smithi064 ceph-mon[30223]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:53:32.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T11:53:32.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T11:53:32.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T11:53:32.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T11:53:32.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T11:53:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T11:53:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T11:53:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T11:53:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T11:53:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T11:53:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4006274182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:53:32.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:31 smithi193 ceph-mon[33593]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:53:32.125 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920799 got 141733920802 for osd.4 2024-03-24T11:53:32.126 DEBUG:teuthology.parallel:result is None 2024-03-24T11:53:32.853 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[30223]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:32.853 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T11:53:32.853 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T11:53:32.853 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T11:53:32.854 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T11:53:32.854 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[42243]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:32.854 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T11:53:32.854 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T11:53:32.854 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T11:53:32.854 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:32 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T11:53:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:32 smithi193 ceph-mon[33593]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:32 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T11:53:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:32 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T11:53:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:32 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T11:53:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:32 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T11:53:33.296 INFO:teuthology.orchestra.run.smithi064.stdout:115964117032 2024-03-24T11:53:33.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005070 got 261993005072 for osd.7 2024-03-24T11:53:33.913 DEBUG:teuthology.parallel:result is None 2024-03-24T11:53:34.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:33 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T11:53:34.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:33 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T11:53:34.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:33 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2386817975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:53:34.007 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:33 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T11:53:34.007 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:33 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T11:53:34.007 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:33 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2386817975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:53:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:33 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T11:53:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:33 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T11:53:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:33 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2386817975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:53:34.326 INFO:teuthology.orchestra.run.smithi064.stdout:77309411374 2024-03-24T11:53:34.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T11:53:34.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T11:53:34.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T11:53:34.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T11:53:34.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[30223]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:34.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T11:53:34.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T11:53:34.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/39792465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:53:34.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T11:53:34.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T11:53:34.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T11:53:34.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T11:53:34.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[42243]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:34.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T11:53:34.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T11:53:34.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:34 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/39792465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:53:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:34 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T11:53:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:34 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T11:53:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:34 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T11:53:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:34 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T11:53:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:34 smithi193 ceph-mon[33593]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:34 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T11:53:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:34 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T11:53:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:34 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/39792465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:53:35.259 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117028 got 115964117032 for osd.3 2024-03-24T11:53:35.259 DEBUG:teuthology.parallel:result is None 2024-03-24T11:53:35.936 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T11:53:35.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T11:53:35.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[30223]: 4.c scrub starts 2024-03-24T11:53:35.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[30223]: 4.c scrub ok 2024-03-24T11:53:35.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[30223]: 4.0 scrub starts 2024-03-24T11:53:35.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[30223]: 4.0 scrub ok 2024-03-24T11:53:35.937 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T11:53:35.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T11:53:35.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[42243]: 4.c scrub starts 2024-03-24T11:53:35.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[42243]: 4.c scrub ok 2024-03-24T11:53:35.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[42243]: 4.0 scrub starts 2024-03-24T11:53:35.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:35 smithi064 ceph-mon[42243]: 4.0 scrub ok 2024-03-24T11:53:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:35 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T11:53:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:35 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T11:53:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:35 smithi193 ceph-mon[33593]: 4.c scrub starts 2024-03-24T11:53:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:35 smithi193 ceph-mon[33593]: 4.c scrub ok 2024-03-24T11:53:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:35 smithi193 ceph-mon[33593]: 4.0 scrub starts 2024-03-24T11:53:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:35 smithi193 ceph-mon[33593]: 4.0 scrub ok 2024-03-24T11:53:36.302 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 podman[79485]: 2024-03-24 11:53:36.112820723 +0000 UTC m=+1.544848764 container create 99c8402f534fe712e26b28028754771ba6acc170b5c72a44e73578811d266bdc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:53:36.737 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 podman[79485]: 2024-03-24 11:53:36.499241955 +0000 UTC m=+1.931269988 container init 99c8402f534fe712e26b28028754771ba6acc170b5c72a44e73578811d266bdc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 podman[79485]: 2024-03-24 11:53:36.505069186 +0000 UTC m=+1.937097219 container start 99c8402f534fe712e26b28028754771ba6acc170b5c72a44e73578811d266bdc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:36.519Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:36.519Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:36.520Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.64 port=9094 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:36.521Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:36.547Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:36.547Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:36.550Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:36.550Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 bash[79485]: 99c8402f534fe712e26b28028754771ba6acc170b5c72a44e73578811d266bdc 2024-03-24T11:53:36.738 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:36 smithi064 systemd[1]: Started Ceph alertmanager.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:53:36.741 INFO:teuthology.orchestra.run.smithi064.stdout:219043332122 2024-03-24T11:53:37.001 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: 2.18 scrub starts 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: 2.18 scrub ok 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2093977201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: 2.18 scrub starts 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: 2.18 scrub ok 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2093977201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:36] ENGINE Bus STOPPING 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:36] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:36] ENGINE Bus STOPPED 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:36 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:36] ENGINE Bus STARTING 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:37] ENGINE Serving on http://:::9283 2024-03-24T11:53:37.004 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:37] ENGINE Bus STARTED 2024-03-24T11:53:37.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:53:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:53:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T11:53:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T11:53:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T11:53:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T11:53:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T11:53:37.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T11:53:37.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: 2.18 scrub starts 2024-03-24T11:53:37.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: 2.18 scrub ok 2024-03-24T11:53:37.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:37.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:36 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2093977201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:53:37.450 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411374 for osd.2 2024-03-24T11:53:37.450 DEBUG:teuthology.parallel:result is None 2024-03-24T11:53:37.802 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:53:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:37.908 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332117 got 219043332122 for osd.6 2024-03-24T11:53:37.908 DEBUG:teuthology.parallel:result is None 2024-03-24T11:53:37.908 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:53:37.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:53:38.157 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:38.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T11:53:38.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: 2.2 scrub starts 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: 2.2 scrub ok 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: Deploying daemon grafana.a on smithi193 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: 6.1d scrub starts 2024-03-24T11:53:38.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[30223]: 6.1d scrub ok 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: 2.2 scrub starts 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: 2.2 scrub ok 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T11:53:38.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T11:53:38.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: Deploying daemon grafana.a on smithi193 2024-03-24T11:53:38.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: 6.1d scrub starts 2024-03-24T11:53:38.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:37 smithi064 ceph-mon[42243]: 6.1d scrub ok 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: 2.2 scrub starts 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: 2.2 scrub ok 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:53:38.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:53:38.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:38.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T11:53:38.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T11:53:38.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: Deploying daemon grafana.a on smithi193 2024-03-24T11:53:38.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: 6.1d scrub starts 2024-03-24T11:53:38.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:37 smithi193 ceph-mon[33593]: 6.1d scrub ok 2024-03-24T11:53:38.544 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:38.522Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000577716s 2024-03-24T11:53:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T11:53:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T11:53:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 6.1f scrub ok 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 4.8 scrub starts 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 4.8 scrub ok 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 6.10 scrub starts 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 2.d deep-scrub starts 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[30223]: 2.d deep-scrub ok 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 6.1f scrub ok 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 4.8 scrub starts 2024-03-24T11:53:39.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 4.8 scrub ok 2024-03-24T11:53:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 6.10 scrub starts 2024-03-24T11:53:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 2.d deep-scrub starts 2024-03-24T11:53:39.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:38 smithi064 ceph-mon[42243]: 2.d deep-scrub ok 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 6.1f scrub ok 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 4.8 scrub starts 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 4.8 scrub ok 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 6.10 scrub starts 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 2.d deep-scrub starts 2024-03-24T11:53:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:38 smithi193 ceph-mon[33593]: 2.d deep-scrub ok 2024-03-24T11:53:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[30223]: 6.10 scrub ok 2024-03-24T11:53:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T11:53:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T11:53:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T11:53:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T11:53:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T11:53:40.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T11:53:40.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[42243]: 6.10 scrub ok 2024-03-24T11:53:40.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T11:53:40.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T11:53:40.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T11:53:40.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T11:53:40.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T11:53:40.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:39 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T11:53:40.257 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:53:40.259 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:53:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:39 smithi193 ceph-mon[33593]: 6.10 scrub ok 2024-03-24T11:53:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:39 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T11:53:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:39 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T11:53:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:39 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T11:53:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:39 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T11:53:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:39 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T11:53:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:39 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T11:53:40.953 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-24T11:53:39.143006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1002,"num_read_kb":848,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":612,"ondisk_log_size":612,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244932,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716284,"statfs":{"total":767960285184,"available":767709474816,"internally_reserved":0,"allocated":8187904,"data_stored":4594306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002109"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615218+0000","last_change":"2024-03-24T11:52:21.854369+0000","last_active":"2024-03-24T11:53:31.615218+0000","last_peered":"2024-03-24T11:53:31.615218+0000","last_clean":"2024-03-24T11:53:31.615218+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:53:31.615218+0000","last_undegraded":"2024-03-24T11:53:31.615218+0000","last_fullsized":"2024-03-24T11:53:31.615218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.448132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667226+0000","last_change":"2024-03-24T11:53:29.715820+0000","last_active":"2024-03-24T11:53:31.667226+0000","last_peered":"2024-03-24T11:53:31.667226+0000","last_clean":"2024-03-24T11:53:31.667226+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:53:31.667226+0000","last_undegraded":"2024-03-24T11:53:31.667226+0000","last_fullsized":"2024-03-24T11:53:31.667226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.715710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:29.794324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616965+0000","last_change":"2024-03-24T11:52:15.137687+0000","last_active":"2024-03-24T11:53:31.616965+0000","last_peered":"2024-03-24T11:53:31.616965+0000","last_clean":"2024-03-24T11:53:31.616965+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:53:31.616965+0000","last_undegraded":"2024-03-24T11:53:31.616965+0000","last_fullsized":"2024-03-24T11:53:31.616965+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:41.640099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616599+0000","last_change":"2024-03-24T11:52:17.354145+0000","last_active":"2024-03-24T11:53:31.616599+0000","last_peered":"2024-03-24T11:53:31.616599+0000","last_clean":"2024-03-24T11:53:31.616599+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:53:31.616599+0000","last_undegraded":"2024-03-24T11:53:31.616599+0000","last_fullsized":"2024-03-24T11:53:31.616599+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667381+0000","last_change":"2024-03-24T11:53:30.710255+0000","last_active":"2024-03-24T11:53:31.667381+0000","last_peered":"2024-03-24T11:53:31.667381+0000","last_clean":"2024-03-24T11:53:31.667381+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:53:31.667381+0000","last_undegraded":"2024-03-24T11:53:31.667381+0000","last_fullsized":"2024-03-24T11:53:31.667381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.710143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.710143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.327391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720183+0000","last_change":"2024-03-24T11:52:19.378597+0000","last_active":"2024-03-24T11:53:31.720183+0000","last_peered":"2024-03-24T11:53:31.720183+0000","last_clean":"2024-03-24T11:53:31.720183+0000","last_became_active":"2024-03-24T11:52:19.378282+0000","last_became_peered":"2024-03-24T11:52:19.378282+0000","last_unstale":"2024-03-24T11:53:31.720183+0000","last_undegraded":"2024-03-24T11:53:31.720183+0000","last_fullsized":"2024-03-24T11:53:31.720183+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.189406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616026+0000","last_change":"2024-03-24T11:52:15.141672+0000","last_active":"2024-03-24T11:53:31.616026+0000","last_peered":"2024-03-24T11:53:31.616026+0000","last_clean":"2024-03-24T11:53:31.616026+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:53:31.616026+0000","last_undegraded":"2024-03-24T11:53:31.616026+0000","last_fullsized":"2024-03-24T11:53:31.616026+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:32.172017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667465+0000","last_change":"2024-03-24T11:52:17.350943+0000","last_active":"2024-03-24T11:53:31.667465+0000","last_peered":"2024-03-24T11:53:31.667465+0000","last_clean":"2024-03-24T11:53:31.667465+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:53:31.667465+0000","last_undegraded":"2024-03-24T11:53:31.667465+0000","last_fullsized":"2024-03-24T11:53:31.667465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:45.116502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011564+0000","last_change":"2024-03-24T11:53:25.779733+0000","last_active":"2024-03-24T11:53:32.011564+0000","last_peered":"2024-03-24T11:53:32.011564+0000","last_clean":"2024-03-24T11:53:32.011564+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:53:32.011564+0000","last_undegraded":"2024-03-24T11:53:32.011564+0000","last_fullsized":"2024-03-24T11:53:32.011564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.779615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:49.135209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616248+0000","last_change":"2024-03-24T11:52:19.384352+0000","last_active":"2024-03-24T11:53:31.616248+0000","last_peered":"2024-03-24T11:53:31.616248+0000","last_clean":"2024-03-24T11:53:31.616248+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:53:31.616248+0000","last_undegraded":"2024-03-24T11:53:31.616248+0000","last_fullsized":"2024-03-24T11:53:31.616248+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:33.719241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.593625+0000","last_change":"2024-03-24T11:53:35.593649+0000","last_active":"2024-03-24T11:53:35.593625+0000","last_peered":"2024-03-24T11:53:35.593625+0000","last_clean":"2024-03-24T11:53:35.593625+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:53:35.593625+0000","last_undegraded":"2024-03-24T11:53:35.593625+0000","last_fullsized":"2024-03-24T11:53:35.593625+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.593554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.593554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.653079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667601+0000","last_change":"2024-03-24T11:53:20.591717+0000","last_active":"2024-03-24T11:53:31.667601+0000","last_peered":"2024-03-24T11:53:31.667601+0000","last_clean":"2024-03-24T11:53:31.667601+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:53:31.667601+0000","last_undegraded":"2024-03-24T11:53:31.667601+0000","last_fullsized":"2024-03-24T11:53:31.667601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_clean_scrub_stamp":"2024-03-24T11:53:20.591632+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.910645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616332+0000","last_change":"2024-03-24T11:52:21.391890+0000","last_active":"2024-03-24T11:53:31.616332+0000","last_peered":"2024-03-24T11:53:31.616332+0000","last_clean":"2024-03-24T11:53:31.616332+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:53:31.616332+0000","last_undegraded":"2024-03-24T11:53:31.616332+0000","last_fullsized":"2024-03-24T11:53:31.616332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.108281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011962+0000","last_change":"2024-03-24T11:52:19.381366+0000","last_active":"2024-03-24T11:53:32.011962+0000","last_peered":"2024-03-24T11:53:32.011962+0000","last_clean":"2024-03-24T11:53:32.011962+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:53:32.011962+0000","last_undegraded":"2024-03-24T11:53:32.011962+0000","last_fullsized":"2024-03-24T11:53:32.011962+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.656333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616388+0000","last_change":"2024-03-24T11:52:15.144397+0000","last_active":"2024-03-24T11:53:31.616388+0000","last_peered":"2024-03-24T11:53:31.616388+0000","last_clean":"2024-03-24T11:53:31.616388+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:53:31.616388+0000","last_undegraded":"2024-03-24T11:53:31.616388+0000","last_fullsized":"2024-03-24T11:53:31.616388+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.800184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667464+0000","last_change":"2024-03-24T11:53:17.621045+0000","last_active":"2024-03-24T11:53:31.667464+0000","last_peered":"2024-03-24T11:53:31.667464+0000","last_clean":"2024-03-24T11:53:31.667464+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:53:31.667464+0000","last_undegraded":"2024-03-24T11:53:31.667464+0000","last_fullsized":"2024-03-24T11:53:31.667464+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:17.620911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:17.620911+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:20.912812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616403+0000","last_change":"2024-03-24T11:52:21.851424+0000","last_active":"2024-03-24T11:53:31.616403+0000","last_peered":"2024-03-24T11:53:31.616403+0000","last_clean":"2024-03-24T11:53:31.616403+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:53:31.616403+0000","last_undegraded":"2024-03-24T11:53:31.616403+0000","last_fullsized":"2024-03-24T11:53:31.616403+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667753+0000","last_change":"2024-03-24T11:52:19.382663+0000","last_active":"2024-03-24T11:53:31.667753+0000","last_peered":"2024-03-24T11:53:31.667753+0000","last_clean":"2024-03-24T11:53:31.667753+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:53:31.667753+0000","last_undegraded":"2024-03-24T11:53:31.667753+0000","last_fullsized":"2024-03-24T11:53:31.667753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.250149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615402+0000","last_change":"2024-03-24T11:53:30.739186+0000","last_active":"2024-03-24T11:53:31.615402+0000","last_peered":"2024-03-24T11:53:31.615402+0000","last_clean":"2024-03-24T11:53:31.615402+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:53:31.615402+0000","last_undegraded":"2024-03-24T11:53:31.615402+0000","last_fullsized":"2024-03-24T11:53:31.615402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:30.739065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.739065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.055226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615707+0000","last_change":"2024-03-24T11:52:37.803400+0000","last_active":"2024-03-24T11:53:31.615707+0000","last_peered":"2024-03-24T11:53:31.615707+0000","last_clean":"2024-03-24T11:53:31.615707+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:53:31.615707+0000","last_undegraded":"2024-03-24T11:53:31.615707+0000","last_fullsized":"2024-03-24T11:53:31.615707+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668264+0000","last_change":"2024-03-24T11:53:27.651110+0000","last_active":"2024-03-24T11:53:31.668264+0000","last_peered":"2024-03-24T11:53:31.668264+0000","last_clean":"2024-03-24T11:53:31.668264+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:53:31.668264+0000","last_undegraded":"2024-03-24T11:53:31.668264+0000","last_fullsized":"2024-03-24T11:53:31.668264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:27.650990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:27.650990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:31.773205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720918+0000","last_change":"2024-03-24T11:52:19.384621+0000","last_active":"2024-03-24T11:53:31.720918+0000","last_peered":"2024-03-24T11:53:31.720918+0000","last_clean":"2024-03-24T11:53:31.720918+0000","last_became_active":"2024-03-24T11:52:19.384249+0000","last_became_peered":"2024-03-24T11:52:19.384249+0000","last_unstale":"2024-03-24T11:53:31.720918+0000","last_undegraded":"2024-03-24T11:53:31.720918+0000","last_fullsized":"2024-03-24T11:53:31.720918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:33.356930+0000","last_change":"2024-03-24T11:53:33.356957+0000","last_active":"2024-03-24T11:53:33.356930+0000","last_peered":"2024-03-24T11:53:33.356930+0000","last_clean":"2024-03-24T11:53:33.356930+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:53:33.356930+0000","last_undegraded":"2024-03-24T11:53:33.356930+0000","last_fullsized":"2024-03-24T11:53:33.356930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:33.356847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:33.356847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.122844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722687+0000","last_change":"2024-03-24T11:52:17.359240+0000","last_active":"2024-03-24T11:53:31.722687+0000","last_peered":"2024-03-24T11:53:31.722687+0000","last_clean":"2024-03-24T11:53:31.722687+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:53:31.722687+0000","last_undegraded":"2024-03-24T11:53:31.722687+0000","last_fullsized":"2024-03-24T11:53:31.722687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720552+0000","last_change":"2024-03-24T11:52:21.391116+0000","last_active":"2024-03-24T11:53:31.720552+0000","last_peered":"2024-03-24T11:53:31.720552+0000","last_clean":"2024-03-24T11:53:31.720552+0000","last_became_active":"2024-03-24T11:52:21.390764+0000","last_became_peered":"2024-03-24T11:52:21.390764+0000","last_unstale":"2024-03-24T11:53:31.720552+0000","last_undegraded":"2024-03-24T11:53:31.720552+0000","last_fullsized":"2024-03-24T11:53:31.720552+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616952+0000","last_change":"2024-03-24T11:53:21.941429+0000","last_active":"2024-03-24T11:53:31.616952+0000","last_peered":"2024-03-24T11:53:31.616952+0000","last_clean":"2024-03-24T11:53:31.616952+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:53:31.616952+0000","last_undegraded":"2024-03-24T11:53:31.616952+0000","last_fullsized":"2024-03-24T11:53:31.616952+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_clean_scrub_stamp":"2024-03-24T11:53:21.941323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:31.533370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615213+0000","last_change":"2024-03-24T11:52:15.136259+0000","last_active":"2024-03-24T11:53:31.615213+0000","last_peered":"2024-03-24T11:53:31.615213+0000","last_clean":"2024-03-24T11:53:31.615213+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:53:31.615213+0000","last_undegraded":"2024-03-24T11:53:31.615213+0000","last_fullsized":"2024-03-24T11:53:31.615213+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.912582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.670242+0000","last_change":"2024-03-24T11:53:29.274282+0000","last_active":"2024-03-24T11:53:31.670242+0000","last_peered":"2024-03-24T11:53:31.670242+0000","last_clean":"2024-03-24T11:53:31.670242+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:53:31.670242+0000","last_undegraded":"2024-03-24T11:53:31.670242+0000","last_fullsized":"2024-03-24T11:53:31.670242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:53:29.274127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.274127+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:43.553788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616895+0000","last_change":"2024-03-24T11:52:21.390347+0000","last_active":"2024-03-24T11:53:31.616895+0000","last_peered":"2024-03-24T11:53:31.616895+0000","last_clean":"2024-03-24T11:53:31.616895+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:53:31.616895+0000","last_undegraded":"2024-03-24T11:53:31.616895+0000","last_fullsized":"2024-03-24T11:53:31.616895+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.670312+0000","last_change":"2024-03-24T11:52:19.387607+0000","last_active":"2024-03-24T11:53:31.670312+0000","last_peered":"2024-03-24T11:53:31.670312+0000","last_clean":"2024-03-24T11:53:31.670312+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:53:31.670312+0000","last_undegraded":"2024-03-24T11:53:31.670312+0000","last_fullsized":"2024-03-24T11:53:31.670312+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.239623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.814114+0000","last_change":"2024-03-24T11:53:35.814141+0000","last_active":"2024-03-24T11:53:35.814114+0000","last_peered":"2024-03-24T11:53:35.814114+0000","last_clean":"2024-03-24T11:53:35.814114+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:53:35.814114+0000","last_undegraded":"2024-03-24T11:53:35.814114+0000","last_fullsized":"2024-03-24T11:53:35.814114+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.814032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.814032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.550457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616926+0000","last_change":"2024-03-24T11:52:17.426191+0000","last_active":"2024-03-24T11:53:31.616926+0000","last_peered":"2024-03-24T11:53:31.616926+0000","last_clean":"2024-03-24T11:53:31.616926+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:53:31.616926+0000","last_undegraded":"2024-03-24T11:53:31.616926+0000","last_fullsized":"2024-03-24T11:53:31.616926+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615660+0000","last_change":"2024-03-24T11:52:21.854322+0000","last_active":"2024-03-24T11:53:31.615660+0000","last_peered":"2024-03-24T11:53:31.615660+0000","last_clean":"2024-03-24T11:53:31.615660+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:53:31.615660+0000","last_undegraded":"2024-03-24T11:53:31.615660+0000","last_fullsized":"2024-03-24T11:53:31.615660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:46.474982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616522+0000","last_change":"2024-03-24T11:52:19.385213+0000","last_active":"2024-03-24T11:53:31.616522+0000","last_peered":"2024-03-24T11:53:31.616522+0000","last_clean":"2024-03-24T11:53:31.616522+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:53:31.616522+0000","last_undegraded":"2024-03-24T11:53:31.616522+0000","last_fullsized":"2024-03-24T11:53:31.616522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.014947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.343665+0000","last_change":"2024-03-24T11:53:35.343688+0000","last_active":"2024-03-24T11:53:35.343665+0000","last_peered":"2024-03-24T11:53:35.343665+0000","last_clean":"2024-03-24T11:53:35.343665+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:53:35.343665+0000","last_undegraded":"2024-03-24T11:53:35.343665+0000","last_fullsized":"2024-03-24T11:53:35.343665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.343594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.343594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.531863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615527+0000","last_change":"2024-03-24T11:52:17.423884+0000","last_active":"2024-03-24T11:53:31.615527+0000","last_peered":"2024-03-24T11:53:31.615527+0000","last_clean":"2024-03-24T11:53:31.615527+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:53:31.615527+0000","last_undegraded":"2024-03-24T11:53:31.615527+0000","last_fullsized":"2024-03-24T11:53:31.615527+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:42.196185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615940+0000","last_change":"2024-03-24T11:52:21.390345+0000","last_active":"2024-03-24T11:53:31.615940+0000","last_peered":"2024-03-24T11:53:31.615940+0000","last_clean":"2024-03-24T11:53:31.615940+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:53:31.615940+0000","last_undegraded":"2024-03-24T11:53:31.615940+0000","last_fullsized":"2024-03-24T11:53:31.615940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:37.603841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669533+0000","last_change":"2024-03-24T11:53:30.303601+0000","last_active":"2024-03-24T11:53:31.669533+0000","last_peered":"2024-03-24T11:53:31.669533+0000","last_clean":"2024-03-24T11:53:31.669533+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:53:31.669533+0000","last_undegraded":"2024-03-24T11:53:31.669533+0000","last_fullsized":"2024-03-24T11:53:31.669533+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.303505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.303505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.216822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012336+0000","last_change":"2024-03-24T11:52:15.142488+0000","last_active":"2024-03-24T11:53:32.012336+0000","last_peered":"2024-03-24T11:53:32.012336+0000","last_clean":"2024-03-24T11:53:32.012336+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:53:32.012336+0000","last_undegraded":"2024-03-24T11:53:32.012336+0000","last_fullsized":"2024-03-24T11:53:32.012336+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.932815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615195+0000","last_change":"2024-03-24T11:52:17.356722+0000","last_active":"2024-03-24T11:53:31.615195+0000","last_peered":"2024-03-24T11:53:31.615195+0000","last_clean":"2024-03-24T11:53:31.615195+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:53:31.615195+0000","last_undegraded":"2024-03-24T11:53:31.615195+0000","last_fullsized":"2024-03-24T11:53:31.615195+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615400+0000","last_change":"2024-03-24T11:52:21.381608+0000","last_active":"2024-03-24T11:53:31.615400+0000","last_peered":"2024-03-24T11:53:31.615400+0000","last_clean":"2024-03-24T11:53:31.615400+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:53:31.615400+0000","last_undegraded":"2024-03-24T11:53:31.615400+0000","last_fullsized":"2024-03-24T11:53:31.615400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:47.376283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720980+0000","last_change":"2024-03-24T11:52:19.377820+0000","last_active":"2024-03-24T11:53:31.720980+0000","last_peered":"2024-03-24T11:53:31.720980+0000","last_clean":"2024-03-24T11:53:31.720980+0000","last_became_active":"2024-03-24T11:52:19.377484+0000","last_became_peered":"2024-03-24T11:52:19.377484+0000","last_unstale":"2024-03-24T11:53:31.720980+0000","last_undegraded":"2024-03-24T11:53:31.720980+0000","last_fullsized":"2024-03-24T11:53:31.720980+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.981260+0000","last_change":"2024-03-24T11:53:32.981301+0000","last_active":"2024-03-24T11:53:32.981260+0000","last_peered":"2024-03-24T11:53:32.981260+0000","last_clean":"2024-03-24T11:53:32.981260+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:53:32.981260+0000","last_undegraded":"2024-03-24T11:53:32.981260+0000","last_fullsized":"2024-03-24T11:53:32.981260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:32.981156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:32.981156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.401524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667703+0000","last_change":"2024-03-24T11:52:17.359275+0000","last_active":"2024-03-24T11:53:31.667703+0000","last_peered":"2024-03-24T11:53:31.667703+0000","last_clean":"2024-03-24T11:53:31.667703+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:53:31.667703+0000","last_undegraded":"2024-03-24T11:53:31.667703+0000","last_fullsized":"2024-03-24T11:53:31.667703+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:41.172845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615880+0000","last_change":"2024-03-24T11:52:21.391761+0000","last_active":"2024-03-24T11:53:31.615880+0000","last_peered":"2024-03-24T11:53:31.615880+0000","last_clean":"2024-03-24T11:53:31.615880+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:53:31.615880+0000","last_undegraded":"2024-03-24T11:53:31.615880+0000","last_fullsized":"2024-03-24T11:53:31.615880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615457+0000","last_change":"2024-03-24T11:52:19.378422+0000","last_active":"2024-03-24T11:53:31.615457+0000","last_peered":"2024-03-24T11:53:31.615457+0000","last_clean":"2024-03-24T11:53:31.615457+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:53:31.615457+0000","last_undegraded":"2024-03-24T11:53:31.615457+0000","last_fullsized":"2024-03-24T11:53:31.615457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.553300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669907+0000","last_change":"2024-03-24T11:52:15.137397+0000","last_active":"2024-03-24T11:53:31.669907+0000","last_peered":"2024-03-24T11:53:31.669907+0000","last_clean":"2024-03-24T11:53:31.669907+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:53:31.669907+0000","last_undegraded":"2024-03-24T11:53:31.669907+0000","last_fullsized":"2024-03-24T11:53:31.669907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.101813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720875+0000","last_change":"2024-03-24T11:53:25.921423+0000","last_active":"2024-03-24T11:53:31.720875+0000","last_peered":"2024-03-24T11:53:31.720875+0000","last_clean":"2024-03-24T11:53:31.720875+0000","last_became_active":"2024-03-24T11:52:17.355956+0000","last_became_peered":"2024-03-24T11:52:17.355956+0000","last_unstale":"2024-03-24T11:53:31.720875+0000","last_undegraded":"2024-03-24T11:53:31.720875+0000","last_fullsized":"2024-03-24T11:53:31.720875+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:25.921326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.921326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:33.718763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668353+0000","last_change":"2024-03-24T11:52:21.385764+0000","last_active":"2024-03-24T11:53:31.668353+0000","last_peered":"2024-03-24T11:53:31.668353+0000","last_clean":"2024-03-24T11:53:31.668353+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:53:31.668353+0000","last_undegraded":"2024-03-24T11:53:31.668353+0000","last_fullsized":"2024-03-24T11:53:31.668353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:40.650453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616061+0000","last_change":"2024-03-24T11:53:28.755945+0000","last_active":"2024-03-24T11:53:31.616061+0000","last_peered":"2024-03-24T11:53:31.616061+0000","last_clean":"2024-03-24T11:53:31.616061+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:53:31.616061+0000","last_undegraded":"2024-03-24T11:53:31.616061+0000","last_fullsized":"2024-03-24T11:53:31.616061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:28.755826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:28.755826+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.854085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616539+0000","last_change":"2024-03-24T11:52:15.135724+0000","last_active":"2024-03-24T11:53:31.616539+0000","last_peered":"2024-03-24T11:53:31.616539+0000","last_clean":"2024-03-24T11:53:31.616539+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:53:31.616539+0000","last_undegraded":"2024-03-24T11:53:31.616539+0000","last_fullsized":"2024-03-24T11:53:31.616539+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.639731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012261+0000","last_change":"2024-03-24T11:52:17.425314+0000","last_active":"2024-03-24T11:53:32.012261+0000","last_peered":"2024-03-24T11:53:32.012261+0000","last_clean":"2024-03-24T11:53:32.012261+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:53:32.012261+0000","last_undegraded":"2024-03-24T11:53:32.012261+0000","last_fullsized":"2024-03-24T11:53:32.012261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:32.078400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616520+0000","last_change":"2024-03-24T11:52:21.382650+0000","last_active":"2024-03-24T11:53:31.616520+0000","last_peered":"2024-03-24T11:53:31.616520+0000","last_clean":"2024-03-24T11:53:31.616520+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:53:31.616520+0000","last_undegraded":"2024-03-24T11:53:31.616520+0000","last_fullsized":"2024-03-24T11:53:31.616520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:33.822928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.812255+0000","last_change":"2024-03-24T11:53:32.812255+0000","last_active":"2024-03-24T11:53:32.812255+0000","last_peered":"2024-03-24T11:53:32.812255+0000","last_clean":"2024-03-24T11:53:32.812255+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:53:32.812255+0000","last_undegraded":"2024-03-24T11:53:32.812255+0000","last_fullsized":"2024-03-24T11:53:32.812255+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:32.812047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:32.812047+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:50.393185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011722+0000","last_change":"2024-03-24T11:52:15.138567+0000","last_active":"2024-03-24T11:53:32.011722+0000","last_peered":"2024-03-24T11:53:32.011722+0000","last_clean":"2024-03-24T11:53:32.011722+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:53:32.011722+0000","last_undegraded":"2024-03-24T11:53:32.011722+0000","last_fullsized":"2024-03-24T11:53:32.011722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.728957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615854+0000","last_change":"2024-03-24T11:52:17.424012+0000","last_active":"2024-03-24T11:53:31.615854+0000","last_peered":"2024-03-24T11:53:31.615854+0000","last_clean":"2024-03-24T11:53:31.615854+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:53:31.615854+0000","last_undegraded":"2024-03-24T11:53:31.615854+0000","last_fullsized":"2024-03-24T11:53:31.615854+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616448+0000","last_change":"2024-03-24T11:52:21.852181+0000","last_active":"2024-03-24T11:53:31.616448+0000","last_peered":"2024-03-24T11:53:31.616448+0000","last_clean":"2024-03-24T11:53:31.616448+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:53:31.616448+0000","last_undegraded":"2024-03-24T11:53:31.616448+0000","last_fullsized":"2024-03-24T11:53:31.616448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012291+0000","last_change":"2024-03-24T11:52:19.379744+0000","last_active":"2024-03-24T11:53:32.012291+0000","last_peered":"2024-03-24T11:53:32.012291+0000","last_clean":"2024-03-24T11:53:32.012291+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:53:32.012291+0000","last_undegraded":"2024-03-24T11:53:32.012291+0000","last_fullsized":"2024-03-24T11:53:32.012291+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:38.441046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669576+0000","last_change":"2024-03-24T11:53:21.338998+0000","last_active":"2024-03-24T11:53:31.669576+0000","last_peered":"2024-03-24T11:53:31.669576+0000","last_clean":"2024-03-24T11:53:31.669576+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:53:31.669576+0000","last_undegraded":"2024-03-24T11:53:31.669576+0000","last_fullsized":"2024-03-24T11:53:31.669576+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:21.338907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:21.338907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:27.393056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616422+0000","last_change":"2024-03-24T11:53:22.834856+0000","last_active":"2024-03-24T11:53:31.616422+0000","last_peered":"2024-03-24T11:53:31.616422+0000","last_clean":"2024-03-24T11:53:31.616422+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:53:31.616422+0000","last_undegraded":"2024-03-24T11:53:31.616422+0000","last_fullsized":"2024-03-24T11:53:31.616422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:53:22.834770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:22.834770+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:35.505245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722320+0000","last_change":"2024-03-24T11:52:21.386021+0000","last_active":"2024-03-24T11:53:31.722320+0000","last_peered":"2024-03-24T11:53:31.722320+0000","last_clean":"2024-03-24T11:53:31.722320+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:53:31.722320+0000","last_undegraded":"2024-03-24T11:53:31.722320+0000","last_fullsized":"2024-03-24T11:53:31.722320+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:42.752706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616493+0000","last_change":"2024-03-24T11:52:19.376888+0000","last_active":"2024-03-24T11:53:31.616493+0000","last_peered":"2024-03-24T11:53:31.616493+0000","last_clean":"2024-03-24T11:53:31.616493+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:53:31.616493+0000","last_undegraded":"2024-03-24T11:53:31.616493+0000","last_fullsized":"2024-03-24T11:53:31.616493+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:43.718577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722040+0000","last_change":"2024-03-24T11:52:15.146614+0000","last_active":"2024-03-24T11:53:31.722040+0000","last_peered":"2024-03-24T11:53:31.722040+0000","last_clean":"2024-03-24T11:53:31.722040+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:53:31.722040+0000","last_undegraded":"2024-03-24T11:53:31.722040+0000","last_fullsized":"2024-03-24T11:53:31.722040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616740+0000","last_change":"2024-03-24T11:52:17.425677+0000","last_active":"2024-03-24T11:53:31.616740+0000","last_peered":"2024-03-24T11:53:31.616740+0000","last_clean":"2024-03-24T11:53:31.616740+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:53:31.616740+0000","last_undegraded":"2024-03-24T11:53:31.616740+0000","last_fullsized":"2024-03-24T11:53:31.616740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615568+0000","last_change":"2024-03-24T11:52:21.389019+0000","last_active":"2024-03-24T11:53:31.615568+0000","last_peered":"2024-03-24T11:53:31.615568+0000","last_clean":"2024-03-24T11:53:31.615568+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:53:31.615568+0000","last_undegraded":"2024-03-24T11:53:31.615568+0000","last_fullsized":"2024-03-24T11:53:31.615568+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.579510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.720510+0000","last_change":"2024-03-24T11:53:35.720564+0000","last_active":"2024-03-24T11:53:35.720510+0000","last_peered":"2024-03-24T11:53:35.720510+0000","last_clean":"2024-03-24T11:53:35.720510+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:53:35.720510+0000","last_undegraded":"2024-03-24T11:53:35.720510+0000","last_fullsized":"2024-03-24T11:53:35.720510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:53:35.720425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.720425+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.137342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667773+0000","last_change":"2024-03-24T11:53:22.656282+0000","last_active":"2024-03-24T11:53:31.667773+0000","last_peered":"2024-03-24T11:53:31.667773+0000","last_clean":"2024-03-24T11:53:31.667773+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:53:31.667773+0000","last_undegraded":"2024-03-24T11:53:31.667773+0000","last_fullsized":"2024-03-24T11:53:31.667773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:53:22.656153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:22.656153+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:23.354452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.721995+0000","last_change":"2024-03-24T11:52:19.378313+0000","last_active":"2024-03-24T11:53:31.721995+0000","last_peered":"2024-03-24T11:53:31.721995+0000","last_clean":"2024-03-24T11:53:31.721995+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:53:31.721995+0000","last_undegraded":"2024-03-24T11:53:31.721995+0000","last_fullsized":"2024-03-24T11:53:31.721995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:31.964988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615935+0000","last_change":"2024-03-24T11:52:21.386419+0000","last_active":"2024-03-24T11:53:31.615935+0000","last_peered":"2024-03-24T11:53:31.615935+0000","last_clean":"2024-03-24T11:53:31.615935+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:53:31.615935+0000","last_undegraded":"2024-03-24T11:53:31.615935+0000","last_fullsized":"2024-03-24T11:53:31.615935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.600678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720346+0000","last_change":"2024-03-24T11:52:15.139215+0000","last_active":"2024-03-24T11:53:31.720346+0000","last_peered":"2024-03-24T11:53:31.720346+0000","last_clean":"2024-03-24T11:53:31.720346+0000","last_became_active":"2024-03-24T11:52:15.138792+0000","last_became_peered":"2024-03-24T11:52:15.138792+0000","last_unstale":"2024-03-24T11:53:31.720346+0000","last_undegraded":"2024-03-24T11:53:31.720346+0000","last_fullsized":"2024-03-24T11:53:31.720346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:33.434907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720946+0000","last_change":"2024-03-24T11:53:26.940720+0000","last_active":"2024-03-24T11:53:31.720946+0000","last_peered":"2024-03-24T11:53:31.720946+0000","last_clean":"2024-03-24T11:53:31.720946+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:53:31.720946+0000","last_undegraded":"2024-03-24T11:53:31.720946+0000","last_fullsized":"2024-03-24T11:53:31.720946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:53:26.940542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:26.940542+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.977884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616764+0000","last_change":"2024-03-24T11:52:19.378322+0000","last_active":"2024-03-24T11:53:31.616764+0000","last_peered":"2024-03-24T11:53:31.616764+0000","last_clean":"2024-03-24T11:53:31.616764+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:53:31.616764+0000","last_undegraded":"2024-03-24T11:53:31.616764+0000","last_fullsized":"2024-03-24T11:53:31.616764+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616111+0000","last_change":"2024-03-24T11:52:21.386459+0000","last_active":"2024-03-24T11:53:31.616111+0000","last_peered":"2024-03-24T11:53:31.616111+0000","last_clean":"2024-03-24T11:53:31.616111+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:53:31.616111+0000","last_undegraded":"2024-03-24T11:53:31.616111+0000","last_fullsized":"2024-03-24T11:53:31.616111+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011803+0000","last_change":"2024-03-24T11:52:15.144865+0000","last_active":"2024-03-24T11:53:32.011803+0000","last_peered":"2024-03-24T11:53:32.011803+0000","last_clean":"2024-03-24T11:53:32.011803+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:53:32.011803+0000","last_undegraded":"2024-03-24T11:53:32.011803+0000","last_fullsized":"2024-03-24T11:53:32.011803+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011915+0000","last_change":"2024-03-24T11:52:37.802552+0000","last_active":"2024-03-24T11:53:32.011915+0000","last_peered":"2024-03-24T11:53:32.011915+0000","last_clean":"2024-03-24T11:53:32.011915+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:53:32.011915+0000","last_undegraded":"2024-03-24T11:53:32.011915+0000","last_fullsized":"2024-03-24T11:53:32.011915+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:43.551457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616757+0000","last_change":"2024-03-24T11:52:19.386393+0000","last_active":"2024-03-24T11:53:31.616757+0000","last_peered":"2024-03-24T11:53:31.616757+0000","last_clean":"2024-03-24T11:53:31.616757+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:53:31.616757+0000","last_undegraded":"2024-03-24T11:53:31.616757+0000","last_fullsized":"2024-03-24T11:53:31.616757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:34.632307+0000","last_change":"2024-03-24T11:53:34.632333+0000","last_active":"2024-03-24T11:53:34.632307+0000","last_peered":"2024-03-24T11:53:34.632307+0000","last_clean":"2024-03-24T11:53:34.632307+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:53:34.632307+0000","last_undegraded":"2024-03-24T11:53:34.632307+0000","last_fullsized":"2024-03-24T11:53:34.632307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:34.632228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:34.632228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.019688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":142,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:34.814785+0000","last_change":"2024-03-24T11:52:19.373651+0000","last_active":"2024-03-24T11:53:34.814785+0000","last_peered":"2024-03-24T11:53:34.814785+0000","last_clean":"2024-03-24T11:53:34.814785+0000","last_became_active":"2024-03-24T11:52:15.144954+0000","last_became_peered":"2024-03-24T11:52:15.144954+0000","last_unstale":"2024-03-24T11:53:34.814785+0000","last_undegraded":"2024-03-24T11:53:34.814785+0000","last_fullsized":"2024-03-24T11:53:34.814785+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722063+0000","last_change":"2024-03-24T11:53:29.429575+0000","last_active":"2024-03-24T11:53:31.722063+0000","last_peered":"2024-03-24T11:53:31.722063+0000","last_clean":"2024-03-24T11:53:31.722063+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:53:31.722063+0000","last_undegraded":"2024-03-24T11:53:31.722063+0000","last_fullsized":"2024-03-24T11:53:31.722063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.429483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.429483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.616922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720731+0000","last_change":"2024-03-24T11:52:17.358816+0000","last_active":"2024-03-24T11:53:31.720731+0000","last_peered":"2024-03-24T11:53:31.720731+0000","last_clean":"2024-03-24T11:53:31.720731+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:53:31.720731+0000","last_undegraded":"2024-03-24T11:53:31.720731+0000","last_fullsized":"2024-03-24T11:53:31.720731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:37.920801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722360+0000","last_change":"2024-03-24T11:52:19.383334+0000","last_active":"2024-03-24T11:53:31.722360+0000","last_peered":"2024-03-24T11:53:31.722360+0000","last_clean":"2024-03-24T11:53:31.722360+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:53:31.722360+0000","last_undegraded":"2024-03-24T11:53:31.722360+0000","last_fullsized":"2024-03-24T11:53:31.722360+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012565+0000","last_change":"2024-03-24T11:52:21.383692+0000","last_active":"2024-03-24T11:53:32.012565+0000","last_peered":"2024-03-24T11:53:32.012565+0000","last_clean":"2024-03-24T11:53:32.012565+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:53:32.012565+0000","last_undegraded":"2024-03-24T11:53:32.012565+0000","last_fullsized":"2024-03-24T11:53:32.012565+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615171+0000","last_change":"2024-03-24T11:52:19.368496+0000","last_active":"2024-03-24T11:53:31.615171+0000","last_peered":"2024-03-24T11:53:31.615171+0000","last_clean":"2024-03-24T11:53:31.615171+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:53:31.615171+0000","last_undegraded":"2024-03-24T11:53:31.615171+0000","last_fullsized":"2024-03-24T11:53:31.615171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:34.091439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011858+0000","last_change":"2024-03-24T11:52:15.141494+0000","last_active":"2024-03-24T11:53:32.011858+0000","last_peered":"2024-03-24T11:53:32.011858+0000","last_clean":"2024-03-24T11:53:32.011858+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:53:32.011858+0000","last_undegraded":"2024-03-24T11:53:32.011858+0000","last_fullsized":"2024-03-24T11:53:32.011858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.375141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":311,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615970+0000","last_change":"2024-03-24T11:52:19.498367+0000","last_active":"2024-03-24T11:53:31.615970+0000","last_peered":"2024-03-24T11:53:31.615970+0000","last_clean":"2024-03-24T11:53:31.615970+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:53:31.615970+0000","last_undegraded":"2024-03-24T11:53:31.615970+0000","last_fullsized":"2024-03-24T11:53:31.615970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'119","last_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_deep_scrub":"71'119","last_deep_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_clean_scrub_stamp":"2024-03-24T11:52:19.498276+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:44.340116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.670184+0000","last_change":"2024-03-24T11:52:17.425751+0000","last_active":"2024-03-24T11:53:31.670184+0000","last_peered":"2024-03-24T11:53:31.670184+0000","last_clean":"2024-03-24T11:53:31.670184+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:53:31.670184+0000","last_undegraded":"2024-03-24T11:53:31.670184+0000","last_fullsized":"2024-03-24T11:53:31.670184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616541+0000","last_change":"2024-03-24T11:53:29.691119+0000","last_active":"2024-03-24T11:53:31.616541+0000","last_peered":"2024-03-24T11:53:31.616541+0000","last_clean":"2024-03-24T11:53:31.616541+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:53:31.616541+0000","last_undegraded":"2024-03-24T11:53:31.616541+0000","last_fullsized":"2024-03-24T11:53:31.616541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.691006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.691006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:54.100657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720471+0000","last_change":"2024-03-24T11:52:21.390110+0000","last_active":"2024-03-24T11:53:31.720471+0000","last_peered":"2024-03-24T11:53:31.720471+0000","last_clean":"2024-03-24T11:53:31.720471+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:53:31.720471+0000","last_undegraded":"2024-03-24T11:53:31.720471+0000","last_fullsized":"2024-03-24T11:53:31.720471+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:31.975034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616906+0000","last_change":"2024-03-24T11:52:19.366588+0000","last_active":"2024-03-24T11:53:31.616906+0000","last_peered":"2024-03-24T11:53:31.616906+0000","last_clean":"2024-03-24T11:53:31.616906+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:53:31.616906+0000","last_undegraded":"2024-03-24T11:53:31.616906+0000","last_fullsized":"2024-03-24T11:53:31.616906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:33.102338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616081+0000","last_change":"2024-03-24T11:53:24.679709+0000","last_active":"2024-03-24T11:53:31.616081+0000","last_peered":"2024-03-24T11:53:31.616081+0000","last_clean":"2024-03-24T11:53:31.616081+0000","last_became_active":"2024-03-24T11:52:15.146310+0000","last_became_peered":"2024-03-24T11:52:15.146310+0000","last_unstale":"2024-03-24T11:53:31.616081+0000","last_undegraded":"2024-03-24T11:53:31.616081+0000","last_fullsized":"2024-03-24T11:53:31.616081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:24.679579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.679579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:29.622468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616630+0000","last_change":"2024-03-24T11:53:24.917322+0000","last_active":"2024-03-24T11:53:31.616630+0000","last_peered":"2024-03-24T11:53:31.616630+0000","last_clean":"2024-03-24T11:53:31.616630+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:53:31.616630+0000","last_undegraded":"2024-03-24T11:53:31.616630+0000","last_fullsized":"2024-03-24T11:53:31.616630+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:53:24.917220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.917220+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.119999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012502+0000","last_change":"2024-03-24T11:52:19.387410+0000","last_active":"2024-03-24T11:53:32.012502+0000","last_peered":"2024-03-24T11:53:32.012502+0000","last_clean":"2024-03-24T11:53:32.012502+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:53:32.012502+0000","last_undegraded":"2024-03-24T11:53:32.012502+0000","last_fullsized":"2024-03-24T11:53:32.012502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:42.573057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720132+0000","last_change":"2024-03-24T11:52:21.851445+0000","last_active":"2024-03-24T11:53:31.720132+0000","last_peered":"2024-03-24T11:53:31.720132+0000","last_clean":"2024-03-24T11:53:31.720132+0000","last_became_active":"2024-03-24T11:52:21.851069+0000","last_became_peered":"2024-03-24T11:52:21.851069+0000","last_unstale":"2024-03-24T11:53:31.720132+0000","last_undegraded":"2024-03-24T11:53:31.720132+0000","last_fullsized":"2024-03-24T11:53:31.720132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616680+0000","last_change":"2024-03-24T11:53:18.576655+0000","last_active":"2024-03-24T11:53:31.616680+0000","last_peered":"2024-03-24T11:53:31.616680+0000","last_clean":"2024-03-24T11:53:31.616680+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:53:31.616680+0000","last_undegraded":"2024-03-24T11:53:31.616680+0000","last_fullsized":"2024-03-24T11:53:31.616680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:18.576538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:18.576538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.141440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616453+0000","last_change":"2024-03-24T11:52:17.423874+0000","last_active":"2024-03-24T11:53:31.616453+0000","last_peered":"2024-03-24T11:53:31.616453+0000","last_clean":"2024-03-24T11:53:31.616453+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:53:31.616453+0000","last_undegraded":"2024-03-24T11:53:31.616453+0000","last_fullsized":"2024-03-24T11:53:31.616453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669986+0000","last_change":"2024-03-24T11:52:19.375675+0000","last_active":"2024-03-24T11:53:31.669986+0000","last_peered":"2024-03-24T11:53:31.669986+0000","last_clean":"2024-03-24T11:53:31.669986+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:53:31.669986+0000","last_undegraded":"2024-03-24T11:53:31.669986+0000","last_fullsized":"2024-03-24T11:53:31.669986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:43.200192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667837+0000","last_change":"2024-03-24T11:52:21.385850+0000","last_active":"2024-03-24T11:53:31.667837+0000","last_peered":"2024-03-24T11:53:31.667837+0000","last_clean":"2024-03-24T11:53:31.667837+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:53:31.667837+0000","last_undegraded":"2024-03-24T11:53:31.667837+0000","last_fullsized":"2024-03-24T11:53:31.667837+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:37.283134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720729+0000","last_change":"2024-03-24T11:52:15.143327+0000","last_active":"2024-03-24T11:53:31.720729+0000","last_peered":"2024-03-24T11:53:31.720729+0000","last_clean":"2024-03-24T11:53:31.720729+0000","last_became_active":"2024-03-24T11:52:15.142974+0000","last_became_peered":"2024-03-24T11:52:15.142974+0000","last_unstale":"2024-03-24T11:53:31.720729+0000","last_undegraded":"2024-03-24T11:53:31.720729+0000","last_fullsized":"2024-03-24T11:53:31.720729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.726697+0000","last_change":"2024-03-24T11:53:35.726728+0000","last_active":"2024-03-24T11:53:35.726697+0000","last_peered":"2024-03-24T11:53:35.726697+0000","last_clean":"2024-03-24T11:53:35.726697+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:53:35.726697+0000","last_undegraded":"2024-03-24T11:53:35.726697+0000","last_fullsized":"2024-03-24T11:53:35.726697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:53:35.726630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.726630+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:37.790720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667930+0000","last_change":"2024-03-24T11:52:19.389693+0000","last_active":"2024-03-24T11:53:31.667930+0000","last_peered":"2024-03-24T11:53:31.667930+0000","last_clean":"2024-03-24T11:53:31.667930+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:53:31.667930+0000","last_undegraded":"2024-03-24T11:53:31.667930+0000","last_fullsized":"2024-03-24T11:53:31.667930+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:47.063026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616276+0000","last_change":"2024-03-24T11:52:21.854938+0000","last_active":"2024-03-24T11:53:31.616276+0000","last_peered":"2024-03-24T11:53:31.616276+0000","last_clean":"2024-03-24T11:53:31.616276+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:53:31.616276+0000","last_undegraded":"2024-03-24T11:53:31.616276+0000","last_fullsized":"2024-03-24T11:53:31.616276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:40.554101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.670343+0000","last_change":"2024-03-24T11:53:15.292341+0000","last_active":"2024-03-24T11:53:31.670343+0000","last_peered":"2024-03-24T11:53:31.670343+0000","last_clean":"2024-03-24T11:53:31.670343+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:53:31.670343+0000","last_undegraded":"2024-03-24T11:53:31.670343+0000","last_fullsized":"2024-03-24T11:53:31.670343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:53:15.292233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.790414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667595+0000","last_change":"2024-03-24T11:53:26.665675+0000","last_active":"2024-03-24T11:53:31.667595+0000","last_peered":"2024-03-24T11:53:31.667595+0000","last_clean":"2024-03-24T11:53:31.667595+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:53:31.667595+0000","last_undegraded":"2024-03-24T11:53:31.667595+0000","last_fullsized":"2024-03-24T11:53:31.667595+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:53:26.665554+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:48.241747+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615117+0000","last_change":"2024-03-24T11:52:19.382757+0000","last_active":"2024-03-24T11:53:31.615117+0000","last_peered":"2024-03-24T11:53:31.615117+0000","last_clean":"2024-03-24T11:53:31.615117+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:53:31.615117+0000","last_undegraded":"2024-03-24T11:53:31.615117+0000","last_fullsized":"2024-03-24T11:53:31.615117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:41.610324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615765+0000","last_change":"2024-03-24T11:52:21.851542+0000","last_active":"2024-03-24T11:53:31.615765+0000","last_peered":"2024-03-24T11:53:31.615765+0000","last_clean":"2024-03-24T11:53:31.615765+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:53:31.615765+0000","last_undegraded":"2024-03-24T11:53:31.615765+0000","last_fullsized":"2024-03-24T11:53:31.615765+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.109536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012500+0000","last_change":"2024-03-24T11:52:19.388558+0000","last_active":"2024-03-24T11:53:32.012500+0000","last_peered":"2024-03-24T11:53:32.012500+0000","last_clean":"2024-03-24T11:53:32.012500+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:53:32.012500+0000","last_undegraded":"2024-03-24T11:53:32.012500+0000","last_fullsized":"2024-03-24T11:53:32.012500+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:38.924350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616185+0000","last_change":"2024-03-24T11:53:30.710184+0000","last_active":"2024-03-24T11:53:31.616185+0000","last_peered":"2024-03-24T11:53:31.616185+0000","last_clean":"2024-03-24T11:53:31.616185+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:53:31.616185+0000","last_undegraded":"2024-03-24T11:53:31.616185+0000","last_fullsized":"2024-03-24T11:53:31.616185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.710079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.710079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.148875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668261+0000","last_change":"2024-03-24T11:52:17.351215+0000","last_active":"2024-03-24T11:53:31.668261+0000","last_peered":"2024-03-24T11:53:31.668261+0000","last_clean":"2024-03-24T11:53:31.668261+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:53:31.668261+0000","last_undegraded":"2024-03-24T11:53:31.668261+0000","last_fullsized":"2024-03-24T11:53:31.668261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:43.604371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011981+0000","last_change":"2024-03-24T11:52:21.388941+0000","last_active":"2024-03-24T11:53:32.011981+0000","last_peered":"2024-03-24T11:53:32.011981+0000","last_clean":"2024-03-24T11:53:32.011981+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:53:32.011981+0000","last_undegraded":"2024-03-24T11:53:32.011981+0000","last_fullsized":"2024-03-24T11:53:32.011981+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:34.557613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668139+0000","last_change":"2024-03-24T11:52:19.383564+0000","last_active":"2024-03-24T11:53:31.668139+0000","last_peered":"2024-03-24T11:53:31.668139+0000","last_clean":"2024-03-24T11:53:31.668139+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:53:31.668139+0000","last_undegraded":"2024-03-24T11:53:31.668139+0000","last_fullsized":"2024-03-24T11:53:31.668139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:37.793519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720264+0000","last_change":"2024-03-24T11:52:15.135547+0000","last_active":"2024-03-24T11:53:31.720264+0000","last_peered":"2024-03-24T11:53:31.720264+0000","last_clean":"2024-03-24T11:53:31.720264+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:53:31.720264+0000","last_undegraded":"2024-03-24T11:53:31.720264+0000","last_fullsized":"2024-03-24T11:53:31.720264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.957766+0000","last_change":"2024-03-24T11:53:31.957804+0000","last_active":"2024-03-24T11:53:31.957766+0000","last_peered":"2024-03-24T11:53:31.957766+0000","last_clean":"2024-03-24T11:53:31.957766+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:53:31.957766+0000","last_undegraded":"2024-03-24T11:53:31.957766+0000","last_fullsized":"2024-03-24T11:53:31.957766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:31.957706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.957706+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.121723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.724334+0000","last_change":"2024-03-24T11:53:31.724380+0000","last_active":"2024-03-24T11:53:31.724334+0000","last_peered":"2024-03-24T11:53:31.724334+0000","last_clean":"2024-03-24T11:53:31.724334+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:53:31.724334+0000","last_undegraded":"2024-03-24T11:53:31.724334+0000","last_fullsized":"2024-03-24T11:53:31.724334+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:53:31.724235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.724235+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.862511+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722400+0000","last_change":"2024-03-24T11:53:19.389995+0000","last_active":"2024-03-24T11:53:31.722400+0000","last_peered":"2024-03-24T11:53:31.722400+0000","last_clean":"2024-03-24T11:53:31.722400+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:53:31.722400+0000","last_undegraded":"2024-03-24T11:53:31.722400+0000","last_fullsized":"2024-03-24T11:53:31.722400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:19.389895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:19.389895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:38.977195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720881+0000","last_change":"2024-03-24T11:52:15.134671+0000","last_active":"2024-03-24T11:53:31.720881+0000","last_peered":"2024-03-24T11:53:31.720881+0000","last_clean":"2024-03-24T11:53:31.720881+0000","last_became_active":"2024-03-24T11:52:15.134326+0000","last_became_peered":"2024-03-24T11:52:15.134326+0000","last_unstale":"2024-03-24T11:53:31.720881+0000","last_undegraded":"2024-03-24T11:53:31.720881+0000","last_fullsized":"2024-03-24T11:53:31.720881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:34.885035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:34.756150+0000","last_change":"2024-03-24T11:53:34.756186+0000","last_active":"2024-03-24T11:53:34.756150+0000","last_peered":"2024-03-24T11:53:34.756150+0000","last_clean":"2024-03-24T11:53:34.756150+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:53:34.756150+0000","last_undegraded":"2024-03-24T11:53:34.756150+0000","last_fullsized":"2024-03-24T11:53:34.756150+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:53:34.756054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:34.756054+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.815024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722392+0000","last_change":"2024-03-24T11:52:21.388549+0000","last_active":"2024-03-24T11:53:31.722392+0000","last_peered":"2024-03-24T11:53:31.722392+0000","last_clean":"2024-03-24T11:53:31.722392+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:53:31.722392+0000","last_undegraded":"2024-03-24T11:53:31.722392+0000","last_fullsized":"2024-03-24T11:53:31.722392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:38.978622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720558+0000","last_change":"2024-03-24T11:52:19.387447+0000","last_active":"2024-03-24T11:53:31.720558+0000","last_peered":"2024-03-24T11:53:31.720558+0000","last_clean":"2024-03-24T11:53:31.720558+0000","last_became_active":"2024-03-24T11:52:19.387115+0000","last_became_peered":"2024-03-24T11:52:19.387115+0000","last_unstale":"2024-03-24T11:53:31.720558+0000","last_undegraded":"2024-03-24T11:53:31.720558+0000","last_fullsized":"2024-03-24T11:53:31.720558+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.579121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722205+0000","last_change":"2024-03-24T11:53:24.386409+0000","last_active":"2024-03-24T11:53:31.722205+0000","last_peered":"2024-03-24T11:53:31.722205+0000","last_clean":"2024-03-24T11:53:31.722205+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:53:31.722205+0000","last_undegraded":"2024-03-24T11:53:31.722205+0000","last_fullsized":"2024-03-24T11:53:31.722205+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:24.386208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.386208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:43.956194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668010+0000","last_change":"2024-03-24T11:52:17.354890+0000","last_active":"2024-03-24T11:53:31.668010+0000","last_peered":"2024-03-24T11:53:31.668010+0000","last_clean":"2024-03-24T11:53:31.668010+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:53:31.668010+0000","last_undegraded":"2024-03-24T11:53:31.668010+0000","last_fullsized":"2024-03-24T11:53:31.668010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.488951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615816+0000","last_change":"2024-03-24T11:52:21.391660+0000","last_active":"2024-03-24T11:53:31.615816+0000","last_peered":"2024-03-24T11:53:31.615816+0000","last_clean":"2024-03-24T11:53:31.615816+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:53:31.615816+0000","last_undegraded":"2024-03-24T11:53:31.615816+0000","last_fullsized":"2024-03-24T11:53:31.615816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:37.583418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722329+0000","last_change":"2024-03-24T11:52:19.379452+0000","last_active":"2024-03-24T11:53:31.722329+0000","last_peered":"2024-03-24T11:53:31.722329+0000","last_clean":"2024-03-24T11:53:31.722329+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:53:31.722329+0000","last_undegraded":"2024-03-24T11:53:31.722329+0000","last_fullsized":"2024-03-24T11:53:31.722329+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722570+0000","last_change":"2024-03-24T11:53:30.449547+0000","last_active":"2024-03-24T11:53:31.722570+0000","last_peered":"2024-03-24T11:53:31.722570+0000","last_clean":"2024-03-24T11:53:31.722570+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:53:31.722570+0000","last_undegraded":"2024-03-24T11:53:31.722570+0000","last_fullsized":"2024-03-24T11:53:31.722570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.449442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.449442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:36.454836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669781+0000","last_change":"2024-03-24T11:52:17.424107+0000","last_active":"2024-03-24T11:53:31.669781+0000","last_peered":"2024-03-24T11:53:31.669781+0000","last_clean":"2024-03-24T11:53:31.669781+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:53:31.669781+0000","last_undegraded":"2024-03-24T11:53:31.669781+0000","last_fullsized":"2024-03-24T11:53:31.669781+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.849775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616605+0000","last_change":"2024-03-24T11:52:21.387355+0000","last_active":"2024-03-24T11:53:31.616605+0000","last_peered":"2024-03-24T11:53:31.616605+0000","last_clean":"2024-03-24T11:53:31.616605+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:53:31.616605+0000","last_undegraded":"2024-03-24T11:53:31.616605+0000","last_fullsized":"2024-03-24T11:53:31.616605+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.688825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722137+0000","last_change":"2024-03-24T11:52:19.384147+0000","last_active":"2024-03-24T11:53:31.722137+0000","last_peered":"2024-03-24T11:53:31.722137+0000","last_clean":"2024-03-24T11:53:31.722137+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:53:31.722137+0000","last_undegraded":"2024-03-24T11:53:31.722137+0000","last_fullsized":"2024-03-24T11:53:31.722137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:32.180888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720912+0000","last_change":"2024-03-24T11:52:15.140460+0000","last_active":"2024-03-24T11:53:31.720912+0000","last_peered":"2024-03-24T11:53:31.720912+0000","last_clean":"2024-03-24T11:53:31.720912+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:53:31.720912+0000","last_undegraded":"2024-03-24T11:53:31.720912+0000","last_fullsized":"2024-03-24T11:53:31.720912+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616605+0000","last_change":"2024-03-24T11:52:17.359266+0000","last_active":"2024-03-24T11:53:31.616605+0000","last_peered":"2024-03-24T11:53:31.616605+0000","last_clean":"2024-03-24T11:53:31.616605+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:53:31.616605+0000","last_undegraded":"2024-03-24T11:53:31.616605+0000","last_fullsized":"2024-03-24T11:53:31.616605+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011656+0000","last_change":"2024-03-24T11:52:21.852577+0000","last_active":"2024-03-24T11:53:32.011656+0000","last_peered":"2024-03-24T11:53:32.011656+0000","last_clean":"2024-03-24T11:53:32.011656+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:53:32.011656+0000","last_undegraded":"2024-03-24T11:53:32.011656+0000","last_fullsized":"2024-03-24T11:53:32.011656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722452+0000","last_change":"2024-03-24T11:53:25.429178+0000","last_active":"2024-03-24T11:53:31.722452+0000","last_peered":"2024-03-24T11:53:31.722452+0000","last_clean":"2024-03-24T11:53:31.722452+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:53:31.722452+0000","last_undegraded":"2024-03-24T11:53:31.722452+0000","last_fullsized":"2024-03-24T11:53:31.722452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:25.429098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.429098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:46.655256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011583+0000","last_change":"2024-03-24T11:52:17.423954+0000","last_active":"2024-03-24T11:53:32.011583+0000","last_peered":"2024-03-24T11:53:32.011583+0000","last_clean":"2024-03-24T11:53:32.011583+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:53:32.011583+0000","last_undegraded":"2024-03-24T11:53:32.011583+0000","last_fullsized":"2024-03-24T11:53:32.011583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.671828+0000","last_change":"2024-03-24T11:53:31.671828+0000","last_active":"2024-03-24T11:53:31.671828+0000","last_peered":"2024-03-24T11:53:31.671828+0000","last_clean":"2024-03-24T11:53:31.671828+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:53:31.671828+0000","last_undegraded":"2024-03-24T11:53:31.671828+0000","last_fullsized":"2024-03-24T11:53:31.671828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:31.671692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.671692+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:49.989770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005076,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1388544,"data_stored":943628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332123,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32528,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712624,"statfs":{"total":95995035648,"available":95961726976,"internally_reserved":0,"allocated":1384448,"data_stored":943446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":802816,"data_stored":352657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32532,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712620,"statfs":{"total":95995035648,"available":95961722880,"internally_reserved":0,"allocated":1388544,"data_stored":942495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":798720,"data_stored":353077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":353507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920804,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":806912,"data_stored":352692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659167,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":798720,"data_stored":352804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:53:40.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:53:41.215 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T11:53:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T11:53:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:40 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:53:41.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:41.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T11:53:41.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T11:53:41.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T11:53:41.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T11:53:41.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T11:53:41.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T11:53:41.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T11:53:41.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T11:53:41.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:41.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:41.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:53:41.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:40 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:53:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[30223]: 4.3 scrub starts 2024-03-24T11:53:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[30223]: 4.3 scrub ok 2024-03-24T11:53:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[30223]: from='client.14754 v1:172.21.15.64:0/3224293854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T11:53:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T11:53:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[30223]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:53:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[30223]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:53:42.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[42243]: 4.3 scrub starts 2024-03-24T11:53:42.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[42243]: 4.3 scrub ok 2024-03-24T11:53:42.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[42243]: from='client.14754 v1:172.21.15.64:0/3224293854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:42.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T11:53:42.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T11:53:42.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[42243]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:53:42.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:41 smithi064 ceph-mon[42243]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:53:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:41 smithi193 ceph-mon[33593]: 4.3 scrub starts 2024-03-24T11:53:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:41 smithi193 ceph-mon[33593]: 4.3 scrub ok 2024-03-24T11:53:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:41 smithi193 ceph-mon[33593]: from='client.14754 v1:172.21.15.64:0/3224293854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:41 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T11:53:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:41 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T11:53:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:41 smithi193 ceph-mon[33593]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:53:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:41 smithi193 ceph-mon[33593]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:53:43.215 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:53:43.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:43.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T11:53:43.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T11:53:43.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: 6.7 deep-scrub starts 2024-03-24T11:53:43.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: 6.7 deep-scrub ok 2024-03-24T11:53:43.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:53:43.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:53:43.216 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T11:53:43.216 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T11:53:43.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:43.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T11:53:43.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T11:53:43.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: 6.7 deep-scrub starts 2024-03-24T11:53:43.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: 6.7 deep-scrub ok 2024-03-24T11:53:43.217 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:53:43.217 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:53:43.217 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T11:53:43.217 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:42 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T11:53:43.222 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:53:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T11:53:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T11:53:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: 6.7 deep-scrub starts 2024-03-24T11:53:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: 6.7 deep-scrub ok 2024-03-24T11:53:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:53:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:53:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T11:53:43.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:42 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T11:53:43.856 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-03-24T11:53:43.144245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1007,"num_read_kb":853,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":612,"ondisk_log_size":612,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244924,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716292,"statfs":{"total":767960285184,"available":767709483008,"internally_reserved":0,"allocated":8187904,"data_stored":4594306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001888"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615218+0000","last_change":"2024-03-24T11:52:21.854369+0000","last_active":"2024-03-24T11:53:31.615218+0000","last_peered":"2024-03-24T11:53:31.615218+0000","last_clean":"2024-03-24T11:53:31.615218+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:53:31.615218+0000","last_undegraded":"2024-03-24T11:53:31.615218+0000","last_fullsized":"2024-03-24T11:53:31.615218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.448132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667226+0000","last_change":"2024-03-24T11:53:29.715820+0000","last_active":"2024-03-24T11:53:31.667226+0000","last_peered":"2024-03-24T11:53:31.667226+0000","last_clean":"2024-03-24T11:53:31.667226+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:53:31.667226+0000","last_undegraded":"2024-03-24T11:53:31.667226+0000","last_fullsized":"2024-03-24T11:53:31.667226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.715710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:29.794324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616965+0000","last_change":"2024-03-24T11:52:15.137687+0000","last_active":"2024-03-24T11:53:31.616965+0000","last_peered":"2024-03-24T11:53:31.616965+0000","last_clean":"2024-03-24T11:53:31.616965+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:53:31.616965+0000","last_undegraded":"2024-03-24T11:53:31.616965+0000","last_fullsized":"2024-03-24T11:53:31.616965+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:41.640099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616599+0000","last_change":"2024-03-24T11:52:17.354145+0000","last_active":"2024-03-24T11:53:31.616599+0000","last_peered":"2024-03-24T11:53:31.616599+0000","last_clean":"2024-03-24T11:53:31.616599+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:53:31.616599+0000","last_undegraded":"2024-03-24T11:53:31.616599+0000","last_fullsized":"2024-03-24T11:53:31.616599+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667381+0000","last_change":"2024-03-24T11:53:30.710255+0000","last_active":"2024-03-24T11:53:31.667381+0000","last_peered":"2024-03-24T11:53:31.667381+0000","last_clean":"2024-03-24T11:53:31.667381+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:53:31.667381+0000","last_undegraded":"2024-03-24T11:53:31.667381+0000","last_fullsized":"2024-03-24T11:53:31.667381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.710143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.710143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.327391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720183+0000","last_change":"2024-03-24T11:52:19.378597+0000","last_active":"2024-03-24T11:53:31.720183+0000","last_peered":"2024-03-24T11:53:31.720183+0000","last_clean":"2024-03-24T11:53:31.720183+0000","last_became_active":"2024-03-24T11:52:19.378282+0000","last_became_peered":"2024-03-24T11:52:19.378282+0000","last_unstale":"2024-03-24T11:53:31.720183+0000","last_undegraded":"2024-03-24T11:53:31.720183+0000","last_fullsized":"2024-03-24T11:53:31.720183+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.189406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616026+0000","last_change":"2024-03-24T11:52:15.141672+0000","last_active":"2024-03-24T11:53:31.616026+0000","last_peered":"2024-03-24T11:53:31.616026+0000","last_clean":"2024-03-24T11:53:31.616026+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:53:31.616026+0000","last_undegraded":"2024-03-24T11:53:31.616026+0000","last_fullsized":"2024-03-24T11:53:31.616026+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:32.172017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667465+0000","last_change":"2024-03-24T11:52:17.350943+0000","last_active":"2024-03-24T11:53:31.667465+0000","last_peered":"2024-03-24T11:53:31.667465+0000","last_clean":"2024-03-24T11:53:31.667465+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:53:31.667465+0000","last_undegraded":"2024-03-24T11:53:31.667465+0000","last_fullsized":"2024-03-24T11:53:31.667465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:45.116502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011564+0000","last_change":"2024-03-24T11:53:25.779733+0000","last_active":"2024-03-24T11:53:32.011564+0000","last_peered":"2024-03-24T11:53:32.011564+0000","last_clean":"2024-03-24T11:53:32.011564+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:53:32.011564+0000","last_undegraded":"2024-03-24T11:53:32.011564+0000","last_fullsized":"2024-03-24T11:53:32.011564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.779615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:49.135209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616248+0000","last_change":"2024-03-24T11:52:19.384352+0000","last_active":"2024-03-24T11:53:31.616248+0000","last_peered":"2024-03-24T11:53:31.616248+0000","last_clean":"2024-03-24T11:53:31.616248+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:53:31.616248+0000","last_undegraded":"2024-03-24T11:53:31.616248+0000","last_fullsized":"2024-03-24T11:53:31.616248+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:33.719241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.593625+0000","last_change":"2024-03-24T11:53:35.593649+0000","last_active":"2024-03-24T11:53:35.593625+0000","last_peered":"2024-03-24T11:53:35.593625+0000","last_clean":"2024-03-24T11:53:35.593625+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:53:35.593625+0000","last_undegraded":"2024-03-24T11:53:35.593625+0000","last_fullsized":"2024-03-24T11:53:35.593625+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.593554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.593554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.653079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667601+0000","last_change":"2024-03-24T11:53:20.591717+0000","last_active":"2024-03-24T11:53:31.667601+0000","last_peered":"2024-03-24T11:53:31.667601+0000","last_clean":"2024-03-24T11:53:31.667601+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:53:31.667601+0000","last_undegraded":"2024-03-24T11:53:31.667601+0000","last_fullsized":"2024-03-24T11:53:31.667601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_clean_scrub_stamp":"2024-03-24T11:53:20.591632+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.910645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616332+0000","last_change":"2024-03-24T11:52:21.391890+0000","last_active":"2024-03-24T11:53:31.616332+0000","last_peered":"2024-03-24T11:53:31.616332+0000","last_clean":"2024-03-24T11:53:31.616332+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:53:31.616332+0000","last_undegraded":"2024-03-24T11:53:31.616332+0000","last_fullsized":"2024-03-24T11:53:31.616332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.108281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011962+0000","last_change":"2024-03-24T11:52:19.381366+0000","last_active":"2024-03-24T11:53:32.011962+0000","last_peered":"2024-03-24T11:53:32.011962+0000","last_clean":"2024-03-24T11:53:32.011962+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:53:32.011962+0000","last_undegraded":"2024-03-24T11:53:32.011962+0000","last_fullsized":"2024-03-24T11:53:32.011962+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.656333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616388+0000","last_change":"2024-03-24T11:52:15.144397+0000","last_active":"2024-03-24T11:53:31.616388+0000","last_peered":"2024-03-24T11:53:31.616388+0000","last_clean":"2024-03-24T11:53:31.616388+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:53:31.616388+0000","last_undegraded":"2024-03-24T11:53:31.616388+0000","last_fullsized":"2024-03-24T11:53:31.616388+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.800184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667464+0000","last_change":"2024-03-24T11:53:17.621045+0000","last_active":"2024-03-24T11:53:31.667464+0000","last_peered":"2024-03-24T11:53:31.667464+0000","last_clean":"2024-03-24T11:53:31.667464+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:53:31.667464+0000","last_undegraded":"2024-03-24T11:53:31.667464+0000","last_fullsized":"2024-03-24T11:53:31.667464+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:17.620911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:17.620911+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:20.912812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:37.743881+0000","last_change":"2024-03-24T11:53:37.743905+0000","last_active":"2024-03-24T11:53:37.743881+0000","last_peered":"2024-03-24T11:53:37.743881+0000","last_clean":"2024-03-24T11:53:37.743881+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:53:37.743881+0000","last_undegraded":"2024-03-24T11:53:37.743881+0000","last_fullsized":"2024-03-24T11:53:37.743881+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.743809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.743809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.308954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667753+0000","last_change":"2024-03-24T11:52:19.382663+0000","last_active":"2024-03-24T11:53:31.667753+0000","last_peered":"2024-03-24T11:53:31.667753+0000","last_clean":"2024-03-24T11:53:31.667753+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:53:31.667753+0000","last_undegraded":"2024-03-24T11:53:31.667753+0000","last_fullsized":"2024-03-24T11:53:31.667753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.250149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615402+0000","last_change":"2024-03-24T11:53:30.739186+0000","last_active":"2024-03-24T11:53:31.615402+0000","last_peered":"2024-03-24T11:53:31.615402+0000","last_clean":"2024-03-24T11:53:31.615402+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:53:31.615402+0000","last_undegraded":"2024-03-24T11:53:31.615402+0000","last_fullsized":"2024-03-24T11:53:31.615402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:30.739065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.739065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.055226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615707+0000","last_change":"2024-03-24T11:52:37.803400+0000","last_active":"2024-03-24T11:53:31.615707+0000","last_peered":"2024-03-24T11:53:31.615707+0000","last_clean":"2024-03-24T11:53:31.615707+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:53:31.615707+0000","last_undegraded":"2024-03-24T11:53:31.615707+0000","last_fullsized":"2024-03-24T11:53:31.615707+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668264+0000","last_change":"2024-03-24T11:53:27.651110+0000","last_active":"2024-03-24T11:53:31.668264+0000","last_peered":"2024-03-24T11:53:31.668264+0000","last_clean":"2024-03-24T11:53:31.668264+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:53:31.668264+0000","last_undegraded":"2024-03-24T11:53:31.668264+0000","last_fullsized":"2024-03-24T11:53:31.668264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:27.650990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:27.650990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:31.773205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720918+0000","last_change":"2024-03-24T11:52:19.384621+0000","last_active":"2024-03-24T11:53:31.720918+0000","last_peered":"2024-03-24T11:53:31.720918+0000","last_clean":"2024-03-24T11:53:31.720918+0000","last_became_active":"2024-03-24T11:52:19.384249+0000","last_became_peered":"2024-03-24T11:52:19.384249+0000","last_unstale":"2024-03-24T11:53:31.720918+0000","last_undegraded":"2024-03-24T11:53:31.720918+0000","last_fullsized":"2024-03-24T11:53:31.720918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:33.356930+0000","last_change":"2024-03-24T11:53:33.356957+0000","last_active":"2024-03-24T11:53:33.356930+0000","last_peered":"2024-03-24T11:53:33.356930+0000","last_clean":"2024-03-24T11:53:33.356930+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:53:33.356930+0000","last_undegraded":"2024-03-24T11:53:33.356930+0000","last_fullsized":"2024-03-24T11:53:33.356930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:33.356847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:33.356847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.122844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:36.510482+0000","last_change":"2024-03-24T11:53:36.510517+0000","last_active":"2024-03-24T11:53:36.510482+0000","last_peered":"2024-03-24T11:53:36.510482+0000","last_clean":"2024-03-24T11:53:36.510482+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:53:36.510482+0000","last_undegraded":"2024-03-24T11:53:36.510482+0000","last_fullsized":"2024-03-24T11:53:36.510482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:36.510415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.510415+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.640197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:36.945647+0000","last_change":"2024-03-24T11:53:36.945667+0000","last_active":"2024-03-24T11:53:36.945647+0000","last_peered":"2024-03-24T11:53:36.945647+0000","last_clean":"2024-03-24T11:53:36.945647+0000","last_became_active":"2024-03-24T11:52:21.390764+0000","last_became_peered":"2024-03-24T11:52:21.390764+0000","last_unstale":"2024-03-24T11:53:36.945647+0000","last_undegraded":"2024-03-24T11:53:36.945647+0000","last_fullsized":"2024-03-24T11:53:36.945647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:36.945578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.945578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:05.683323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616952+0000","last_change":"2024-03-24T11:53:21.941429+0000","last_active":"2024-03-24T11:53:31.616952+0000","last_peered":"2024-03-24T11:53:31.616952+0000","last_clean":"2024-03-24T11:53:31.616952+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:53:31.616952+0000","last_undegraded":"2024-03-24T11:53:31.616952+0000","last_fullsized":"2024-03-24T11:53:31.616952+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_clean_scrub_stamp":"2024-03-24T11:53:21.941323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:31.533370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615213+0000","last_change":"2024-03-24T11:52:15.136259+0000","last_active":"2024-03-24T11:53:31.615213+0000","last_peered":"2024-03-24T11:53:31.615213+0000","last_clean":"2024-03-24T11:53:31.615213+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:53:31.615213+0000","last_undegraded":"2024-03-24T11:53:31.615213+0000","last_fullsized":"2024-03-24T11:53:31.615213+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.912582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.670242+0000","last_change":"2024-03-24T11:53:29.274282+0000","last_active":"2024-03-24T11:53:31.670242+0000","last_peered":"2024-03-24T11:53:31.670242+0000","last_clean":"2024-03-24T11:53:31.670242+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:53:31.670242+0000","last_undegraded":"2024-03-24T11:53:31.670242+0000","last_fullsized":"2024-03-24T11:53:31.670242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:53:29.274127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.274127+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:43.553788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:39.602014+0000","last_change":"2024-03-24T11:53:39.602043+0000","last_active":"2024-03-24T11:53:39.602014+0000","last_peered":"2024-03-24T11:53:39.602014+0000","last_clean":"2024-03-24T11:53:39.602014+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:53:39.602014+0000","last_undegraded":"2024-03-24T11:53:39.602014+0000","last_fullsized":"2024-03-24T11:53:39.602014+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:53:39.601937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.601937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.250197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198667+0000","last_change":"2024-03-24T11:52:19.387607+0000","last_active":"2024-03-24T11:53:35.198667+0000","last_peered":"2024-03-24T11:53:35.198667+0000","last_clean":"2024-03-24T11:53:35.198667+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:53:35.198667+0000","last_undegraded":"2024-03-24T11:53:35.198667+0000","last_fullsized":"2024-03-24T11:53:35.198667+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.814114+0000","last_change":"2024-03-24T11:53:35.814141+0000","last_active":"2024-03-24T11:53:35.814114+0000","last_peered":"2024-03-24T11:53:35.814114+0000","last_clean":"2024-03-24T11:53:35.814114+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:53:35.814114+0000","last_undegraded":"2024-03-24T11:53:35.814114+0000","last_fullsized":"2024-03-24T11:53:35.814114+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.814032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.814032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.550457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616926+0000","last_change":"2024-03-24T11:52:17.426191+0000","last_active":"2024-03-24T11:53:31.616926+0000","last_peered":"2024-03-24T11:53:31.616926+0000","last_clean":"2024-03-24T11:53:31.616926+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:53:31.616926+0000","last_undegraded":"2024-03-24T11:53:31.616926+0000","last_fullsized":"2024-03-24T11:53:31.616926+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615660+0000","last_change":"2024-03-24T11:52:21.854322+0000","last_active":"2024-03-24T11:53:31.615660+0000","last_peered":"2024-03-24T11:53:31.615660+0000","last_clean":"2024-03-24T11:53:31.615660+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:53:31.615660+0000","last_undegraded":"2024-03-24T11:53:31.615660+0000","last_fullsized":"2024-03-24T11:53:31.615660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:46.474982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616522+0000","last_change":"2024-03-24T11:52:19.385213+0000","last_active":"2024-03-24T11:53:31.616522+0000","last_peered":"2024-03-24T11:53:31.616522+0000","last_clean":"2024-03-24T11:53:31.616522+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:53:31.616522+0000","last_undegraded":"2024-03-24T11:53:31.616522+0000","last_fullsized":"2024-03-24T11:53:31.616522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.014947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.343665+0000","last_change":"2024-03-24T11:53:35.343688+0000","last_active":"2024-03-24T11:53:35.343665+0000","last_peered":"2024-03-24T11:53:35.343665+0000","last_clean":"2024-03-24T11:53:35.343665+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:53:35.343665+0000","last_undegraded":"2024-03-24T11:53:35.343665+0000","last_fullsized":"2024-03-24T11:53:35.343665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.343594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.343594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.531863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615527+0000","last_change":"2024-03-24T11:52:17.423884+0000","last_active":"2024-03-24T11:53:31.615527+0000","last_peered":"2024-03-24T11:53:31.615527+0000","last_clean":"2024-03-24T11:53:31.615527+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:53:31.615527+0000","last_undegraded":"2024-03-24T11:53:31.615527+0000","last_fullsized":"2024-03-24T11:53:31.615527+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:42.196185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615940+0000","last_change":"2024-03-24T11:52:21.390345+0000","last_active":"2024-03-24T11:53:31.615940+0000","last_peered":"2024-03-24T11:53:31.615940+0000","last_clean":"2024-03-24T11:53:31.615940+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:53:31.615940+0000","last_undegraded":"2024-03-24T11:53:31.615940+0000","last_fullsized":"2024-03-24T11:53:31.615940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:37.603841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669533+0000","last_change":"2024-03-24T11:53:30.303601+0000","last_active":"2024-03-24T11:53:31.669533+0000","last_peered":"2024-03-24T11:53:31.669533+0000","last_clean":"2024-03-24T11:53:31.669533+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:53:31.669533+0000","last_undegraded":"2024-03-24T11:53:31.669533+0000","last_fullsized":"2024-03-24T11:53:31.669533+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.303505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.303505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.216822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012336+0000","last_change":"2024-03-24T11:52:15.142488+0000","last_active":"2024-03-24T11:53:32.012336+0000","last_peered":"2024-03-24T11:53:32.012336+0000","last_clean":"2024-03-24T11:53:32.012336+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:53:32.012336+0000","last_undegraded":"2024-03-24T11:53:32.012336+0000","last_fullsized":"2024-03-24T11:53:32.012336+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.932815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:39.743768+0000","last_change":"2024-03-24T11:53:39.743799+0000","last_active":"2024-03-24T11:53:39.743768+0000","last_peered":"2024-03-24T11:53:39.743768+0000","last_clean":"2024-03-24T11:53:39.743768+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:53:39.743768+0000","last_undegraded":"2024-03-24T11:53:39.743768+0000","last_fullsized":"2024-03-24T11:53:39.743768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T11:53:39.743685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.743685+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.765998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615400+0000","last_change":"2024-03-24T11:52:21.381608+0000","last_active":"2024-03-24T11:53:31.615400+0000","last_peered":"2024-03-24T11:53:31.615400+0000","last_clean":"2024-03-24T11:53:31.615400+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:53:31.615400+0000","last_undegraded":"2024-03-24T11:53:31.615400+0000","last_fullsized":"2024-03-24T11:53:31.615400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:47.376283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720980+0000","last_change":"2024-03-24T11:52:19.377820+0000","last_active":"2024-03-24T11:53:31.720980+0000","last_peered":"2024-03-24T11:53:31.720980+0000","last_clean":"2024-03-24T11:53:31.720980+0000","last_became_active":"2024-03-24T11:52:19.377484+0000","last_became_peered":"2024-03-24T11:52:19.377484+0000","last_unstale":"2024-03-24T11:53:31.720980+0000","last_undegraded":"2024-03-24T11:53:31.720980+0000","last_fullsized":"2024-03-24T11:53:31.720980+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.981260+0000","last_change":"2024-03-24T11:53:32.981301+0000","last_active":"2024-03-24T11:53:32.981260+0000","last_peered":"2024-03-24T11:53:32.981260+0000","last_clean":"2024-03-24T11:53:32.981260+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:53:32.981260+0000","last_undegraded":"2024-03-24T11:53:32.981260+0000","last_fullsized":"2024-03-24T11:53:32.981260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:32.981156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:32.981156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.401524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667703+0000","last_change":"2024-03-24T11:52:17.359275+0000","last_active":"2024-03-24T11:53:31.667703+0000","last_peered":"2024-03-24T11:53:31.667703+0000","last_clean":"2024-03-24T11:53:31.667703+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:53:31.667703+0000","last_undegraded":"2024-03-24T11:53:31.667703+0000","last_fullsized":"2024-03-24T11:53:31.667703+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:41.172845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:37.904754+0000","last_change":"2024-03-24T11:53:37.904794+0000","last_active":"2024-03-24T11:53:37.904754+0000","last_peered":"2024-03-24T11:53:37.904754+0000","last_clean":"2024-03-24T11:53:37.904754+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:53:37.904754+0000","last_undegraded":"2024-03-24T11:53:37.904754+0000","last_fullsized":"2024-03-24T11:53:37.904754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.904622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.904622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.565966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615457+0000","last_change":"2024-03-24T11:52:19.378422+0000","last_active":"2024-03-24T11:53:31.615457+0000","last_peered":"2024-03-24T11:53:31.615457+0000","last_clean":"2024-03-24T11:53:31.615457+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:53:31.615457+0000","last_undegraded":"2024-03-24T11:53:31.615457+0000","last_fullsized":"2024-03-24T11:53:31.615457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.553300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669907+0000","last_change":"2024-03-24T11:52:15.137397+0000","last_active":"2024-03-24T11:53:31.669907+0000","last_peered":"2024-03-24T11:53:31.669907+0000","last_clean":"2024-03-24T11:53:31.669907+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:53:31.669907+0000","last_undegraded":"2024-03-24T11:53:31.669907+0000","last_fullsized":"2024-03-24T11:53:31.669907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.101813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720875+0000","last_change":"2024-03-24T11:53:25.921423+0000","last_active":"2024-03-24T11:53:31.720875+0000","last_peered":"2024-03-24T11:53:31.720875+0000","last_clean":"2024-03-24T11:53:31.720875+0000","last_became_active":"2024-03-24T11:52:17.355956+0000","last_became_peered":"2024-03-24T11:52:17.355956+0000","last_unstale":"2024-03-24T11:53:31.720875+0000","last_undegraded":"2024-03-24T11:53:31.720875+0000","last_fullsized":"2024-03-24T11:53:31.720875+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:25.921326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.921326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:33.718763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668353+0000","last_change":"2024-03-24T11:52:21.385764+0000","last_active":"2024-03-24T11:53:31.668353+0000","last_peered":"2024-03-24T11:53:31.668353+0000","last_clean":"2024-03-24T11:53:31.668353+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:53:31.668353+0000","last_undegraded":"2024-03-24T11:53:31.668353+0000","last_fullsized":"2024-03-24T11:53:31.668353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:40.650453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198428+0000","last_change":"2024-03-24T11:53:28.755945+0000","last_active":"2024-03-24T11:53:35.198428+0000","last_peered":"2024-03-24T11:53:35.198428+0000","last_clean":"2024-03-24T11:53:35.198428+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:53:35.198428+0000","last_undegraded":"2024-03-24T11:53:35.198428+0000","last_fullsized":"2024-03-24T11:53:35.198428+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:28.755826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:28.755826+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.854085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616539+0000","last_change":"2024-03-24T11:52:15.135724+0000","last_active":"2024-03-24T11:53:31.616539+0000","last_peered":"2024-03-24T11:53:31.616539+0000","last_clean":"2024-03-24T11:53:31.616539+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:53:31.616539+0000","last_undegraded":"2024-03-24T11:53:31.616539+0000","last_fullsized":"2024-03-24T11:53:31.616539+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.639731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012261+0000","last_change":"2024-03-24T11:52:17.425314+0000","last_active":"2024-03-24T11:53:32.012261+0000","last_peered":"2024-03-24T11:53:32.012261+0000","last_clean":"2024-03-24T11:53:32.012261+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:53:32.012261+0000","last_undegraded":"2024-03-24T11:53:32.012261+0000","last_fullsized":"2024-03-24T11:53:32.012261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:32.078400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616520+0000","last_change":"2024-03-24T11:52:21.382650+0000","last_active":"2024-03-24T11:53:31.616520+0000","last_peered":"2024-03-24T11:53:31.616520+0000","last_clean":"2024-03-24T11:53:31.616520+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:53:31.616520+0000","last_undegraded":"2024-03-24T11:53:31.616520+0000","last_fullsized":"2024-03-24T11:53:31.616520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:33.822928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198708+0000","last_change":"2024-03-24T11:53:32.812255+0000","last_active":"2024-03-24T11:53:35.198708+0000","last_peered":"2024-03-24T11:53:35.198708+0000","last_clean":"2024-03-24T11:53:35.198708+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:53:35.198708+0000","last_undegraded":"2024-03-24T11:53:35.198708+0000","last_fullsized":"2024-03-24T11:53:35.198708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:32.812047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:32.812047+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:50.393185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011722+0000","last_change":"2024-03-24T11:52:15.138567+0000","last_active":"2024-03-24T11:53:32.011722+0000","last_peered":"2024-03-24T11:53:32.011722+0000","last_clean":"2024-03-24T11:53:32.011722+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:53:32.011722+0000","last_undegraded":"2024-03-24T11:53:32.011722+0000","last_fullsized":"2024-03-24T11:53:32.011722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.728957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615854+0000","last_change":"2024-03-24T11:52:17.424012+0000","last_active":"2024-03-24T11:53:31.615854+0000","last_peered":"2024-03-24T11:53:31.615854+0000","last_clean":"2024-03-24T11:53:31.615854+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:53:31.615854+0000","last_undegraded":"2024-03-24T11:53:31.615854+0000","last_fullsized":"2024-03-24T11:53:31.615854+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616448+0000","last_change":"2024-03-24T11:52:21.852181+0000","last_active":"2024-03-24T11:53:31.616448+0000","last_peered":"2024-03-24T11:53:31.616448+0000","last_clean":"2024-03-24T11:53:31.616448+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:53:31.616448+0000","last_undegraded":"2024-03-24T11:53:31.616448+0000","last_fullsized":"2024-03-24T11:53:31.616448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012291+0000","last_change":"2024-03-24T11:52:19.379744+0000","last_active":"2024-03-24T11:53:32.012291+0000","last_peered":"2024-03-24T11:53:32.012291+0000","last_clean":"2024-03-24T11:53:32.012291+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:53:32.012291+0000","last_undegraded":"2024-03-24T11:53:32.012291+0000","last_fullsized":"2024-03-24T11:53:32.012291+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:38.441046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669576+0000","last_change":"2024-03-24T11:53:21.338998+0000","last_active":"2024-03-24T11:53:31.669576+0000","last_peered":"2024-03-24T11:53:31.669576+0000","last_clean":"2024-03-24T11:53:31.669576+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:53:31.669576+0000","last_undegraded":"2024-03-24T11:53:31.669576+0000","last_fullsized":"2024-03-24T11:53:31.669576+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:21.338907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:21.338907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:27.393056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616422+0000","last_change":"2024-03-24T11:53:22.834856+0000","last_active":"2024-03-24T11:53:31.616422+0000","last_peered":"2024-03-24T11:53:31.616422+0000","last_clean":"2024-03-24T11:53:31.616422+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:53:31.616422+0000","last_undegraded":"2024-03-24T11:53:31.616422+0000","last_fullsized":"2024-03-24T11:53:31.616422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:53:22.834770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:22.834770+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:35.505245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722320+0000","last_change":"2024-03-24T11:52:21.386021+0000","last_active":"2024-03-24T11:53:31.722320+0000","last_peered":"2024-03-24T11:53:31.722320+0000","last_clean":"2024-03-24T11:53:31.722320+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:53:31.722320+0000","last_undegraded":"2024-03-24T11:53:31.722320+0000","last_fullsized":"2024-03-24T11:53:31.722320+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:42.752706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616493+0000","last_change":"2024-03-24T11:52:19.376888+0000","last_active":"2024-03-24T11:53:31.616493+0000","last_peered":"2024-03-24T11:53:31.616493+0000","last_clean":"2024-03-24T11:53:31.616493+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:53:31.616493+0000","last_undegraded":"2024-03-24T11:53:31.616493+0000","last_fullsized":"2024-03-24T11:53:31.616493+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:43.718577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:38.430957+0000","last_change":"2024-03-24T11:53:38.430978+0000","last_active":"2024-03-24T11:53:38.430957+0000","last_peered":"2024-03-24T11:53:38.430957+0000","last_clean":"2024-03-24T11:53:38.430957+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:53:38.430957+0000","last_undegraded":"2024-03-24T11:53:38.430957+0000","last_fullsized":"2024-03-24T11:53:38.430957+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:38.430896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:38.430896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:46.558321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:38.899904+0000","last_change":"2024-03-24T11:53:38.899939+0000","last_active":"2024-03-24T11:53:38.899904+0000","last_peered":"2024-03-24T11:53:38.899904+0000","last_clean":"2024-03-24T11:53:38.899904+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:53:38.899904+0000","last_undegraded":"2024-03-24T11:53:38.899904+0000","last_fullsized":"2024-03-24T11:53:38.899904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:38.899823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:38.899823+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.812037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615568+0000","last_change":"2024-03-24T11:52:21.389019+0000","last_active":"2024-03-24T11:53:31.615568+0000","last_peered":"2024-03-24T11:53:31.615568+0000","last_clean":"2024-03-24T11:53:31.615568+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:53:31.615568+0000","last_undegraded":"2024-03-24T11:53:31.615568+0000","last_fullsized":"2024-03-24T11:53:31.615568+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.579510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.720510+0000","last_change":"2024-03-24T11:53:35.720564+0000","last_active":"2024-03-24T11:53:35.720510+0000","last_peered":"2024-03-24T11:53:35.720510+0000","last_clean":"2024-03-24T11:53:35.720510+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:53:35.720510+0000","last_undegraded":"2024-03-24T11:53:35.720510+0000","last_fullsized":"2024-03-24T11:53:35.720510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:53:35.720425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.720425+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.137342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667773+0000","last_change":"2024-03-24T11:53:22.656282+0000","last_active":"2024-03-24T11:53:31.667773+0000","last_peered":"2024-03-24T11:53:31.667773+0000","last_clean":"2024-03-24T11:53:31.667773+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:53:31.667773+0000","last_undegraded":"2024-03-24T11:53:31.667773+0000","last_fullsized":"2024-03-24T11:53:31.667773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:53:22.656153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:22.656153+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:23.354452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.721995+0000","last_change":"2024-03-24T11:52:19.378313+0000","last_active":"2024-03-24T11:53:31.721995+0000","last_peered":"2024-03-24T11:53:31.721995+0000","last_clean":"2024-03-24T11:53:31.721995+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:53:31.721995+0000","last_undegraded":"2024-03-24T11:53:31.721995+0000","last_fullsized":"2024-03-24T11:53:31.721995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:31.964988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615935+0000","last_change":"2024-03-24T11:52:21.386419+0000","last_active":"2024-03-24T11:53:31.615935+0000","last_peered":"2024-03-24T11:53:31.615935+0000","last_clean":"2024-03-24T11:53:31.615935+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:53:31.615935+0000","last_undegraded":"2024-03-24T11:53:31.615935+0000","last_fullsized":"2024-03-24T11:53:31.615935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.600678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720346+0000","last_change":"2024-03-24T11:52:15.139215+0000","last_active":"2024-03-24T11:53:31.720346+0000","last_peered":"2024-03-24T11:53:31.720346+0000","last_clean":"2024-03-24T11:53:31.720346+0000","last_became_active":"2024-03-24T11:52:15.138792+0000","last_became_peered":"2024-03-24T11:52:15.138792+0000","last_unstale":"2024-03-24T11:53:31.720346+0000","last_undegraded":"2024-03-24T11:53:31.720346+0000","last_fullsized":"2024-03-24T11:53:31.720346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:33.434907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720946+0000","last_change":"2024-03-24T11:53:26.940720+0000","last_active":"2024-03-24T11:53:31.720946+0000","last_peered":"2024-03-24T11:53:31.720946+0000","last_clean":"2024-03-24T11:53:31.720946+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:53:31.720946+0000","last_undegraded":"2024-03-24T11:53:31.720946+0000","last_fullsized":"2024-03-24T11:53:31.720946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:53:26.940542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:26.940542+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.977884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198345+0000","last_change":"2024-03-24T11:52:19.378322+0000","last_active":"2024-03-24T11:53:35.198345+0000","last_peered":"2024-03-24T11:53:35.198345+0000","last_clean":"2024-03-24T11:53:35.198345+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:53:35.198345+0000","last_undegraded":"2024-03-24T11:53:35.198345+0000","last_fullsized":"2024-03-24T11:53:35.198345+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616111+0000","last_change":"2024-03-24T11:52:21.386459+0000","last_active":"2024-03-24T11:53:31.616111+0000","last_peered":"2024-03-24T11:53:31.616111+0000","last_clean":"2024-03-24T11:53:31.616111+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:53:31.616111+0000","last_undegraded":"2024-03-24T11:53:31.616111+0000","last_fullsized":"2024-03-24T11:53:31.616111+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:36.855503+0000","last_change":"2024-03-24T11:53:36.855528+0000","last_active":"2024-03-24T11:53:36.855503+0000","last_peered":"2024-03-24T11:53:36.855503+0000","last_clean":"2024-03-24T11:53:36.855503+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:53:36.855503+0000","last_undegraded":"2024-03-24T11:53:36.855503+0000","last_fullsized":"2024-03-24T11:53:36.855503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:36.855425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.855425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:37.779842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198686+0000","last_change":"2024-03-24T11:52:37.802552+0000","last_active":"2024-03-24T11:53:35.198686+0000","last_peered":"2024-03-24T11:53:35.198686+0000","last_clean":"2024-03-24T11:53:35.198686+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:53:35.198686+0000","last_undegraded":"2024-03-24T11:53:35.198686+0000","last_fullsized":"2024-03-24T11:53:35.198686+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:43.551457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616757+0000","last_change":"2024-03-24T11:52:19.386393+0000","last_active":"2024-03-24T11:53:31.616757+0000","last_peered":"2024-03-24T11:53:31.616757+0000","last_clean":"2024-03-24T11:53:31.616757+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:53:31.616757+0000","last_undegraded":"2024-03-24T11:53:31.616757+0000","last_fullsized":"2024-03-24T11:53:31.616757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:34.632307+0000","last_change":"2024-03-24T11:53:34.632333+0000","last_active":"2024-03-24T11:53:34.632307+0000","last_peered":"2024-03-24T11:53:34.632307+0000","last_clean":"2024-03-24T11:53:34.632307+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:53:34.632307+0000","last_undegraded":"2024-03-24T11:53:34.632307+0000","last_fullsized":"2024-03-24T11:53:34.632307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:34.632228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:34.632228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.019688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":155,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:39.821210+0000","last_change":"2024-03-24T11:53:39.041508+0000","last_active":"2024-03-24T11:53:39.821210+0000","last_peered":"2024-03-24T11:53:39.821210+0000","last_clean":"2024-03-24T11:53:39.821210+0000","last_became_active":"2024-03-24T11:52:15.144954+0000","last_became_peered":"2024-03-24T11:52:15.144954+0000","last_unstale":"2024-03-24T11:53:39.821210+0000","last_undegraded":"2024-03-24T11:53:39.821210+0000","last_fullsized":"2024-03-24T11:53:39.821210+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:53:39.041412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.041412+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:06.745360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722063+0000","last_change":"2024-03-24T11:53:29.429575+0000","last_active":"2024-03-24T11:53:31.722063+0000","last_peered":"2024-03-24T11:53:31.722063+0000","last_clean":"2024-03-24T11:53:31.722063+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:53:31.722063+0000","last_undegraded":"2024-03-24T11:53:31.722063+0000","last_fullsized":"2024-03-24T11:53:31.722063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.429483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.429483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.616922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720731+0000","last_change":"2024-03-24T11:52:17.358816+0000","last_active":"2024-03-24T11:53:31.720731+0000","last_peered":"2024-03-24T11:53:31.720731+0000","last_clean":"2024-03-24T11:53:31.720731+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:53:31.720731+0000","last_undegraded":"2024-03-24T11:53:31.720731+0000","last_fullsized":"2024-03-24T11:53:31.720731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:37.920801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722360+0000","last_change":"2024-03-24T11:52:19.383334+0000","last_active":"2024-03-24T11:53:31.722360+0000","last_peered":"2024-03-24T11:53:31.722360+0000","last_clean":"2024-03-24T11:53:31.722360+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:53:31.722360+0000","last_undegraded":"2024-03-24T11:53:31.722360+0000","last_fullsized":"2024-03-24T11:53:31.722360+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012565+0000","last_change":"2024-03-24T11:52:21.383692+0000","last_active":"2024-03-24T11:53:32.012565+0000","last_peered":"2024-03-24T11:53:32.012565+0000","last_clean":"2024-03-24T11:53:32.012565+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:53:32.012565+0000","last_undegraded":"2024-03-24T11:53:32.012565+0000","last_fullsized":"2024-03-24T11:53:32.012565+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615171+0000","last_change":"2024-03-24T11:52:19.368496+0000","last_active":"2024-03-24T11:53:31.615171+0000","last_peered":"2024-03-24T11:53:31.615171+0000","last_clean":"2024-03-24T11:53:31.615171+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:53:31.615171+0000","last_undegraded":"2024-03-24T11:53:31.615171+0000","last_fullsized":"2024-03-24T11:53:31.615171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:34.091439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011858+0000","last_change":"2024-03-24T11:52:15.141494+0000","last_active":"2024-03-24T11:53:32.011858+0000","last_peered":"2024-03-24T11:53:32.011858+0000","last_clean":"2024-03-24T11:53:32.011858+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:53:32.011858+0000","last_undegraded":"2024-03-24T11:53:32.011858+0000","last_fullsized":"2024-03-24T11:53:32.011858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:35.375141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":311,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615970+0000","last_change":"2024-03-24T11:52:19.498367+0000","last_active":"2024-03-24T11:53:31.615970+0000","last_peered":"2024-03-24T11:53:31.615970+0000","last_clean":"2024-03-24T11:53:31.615970+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:53:31.615970+0000","last_undegraded":"2024-03-24T11:53:31.615970+0000","last_fullsized":"2024-03-24T11:53:31.615970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'119","last_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_deep_scrub":"71'119","last_deep_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_clean_scrub_stamp":"2024-03-24T11:52:19.498276+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:44.340116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198678+0000","last_change":"2024-03-24T11:52:17.425751+0000","last_active":"2024-03-24T11:53:35.198678+0000","last_peered":"2024-03-24T11:53:35.198678+0000","last_clean":"2024-03-24T11:53:35.198678+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:53:35.198678+0000","last_undegraded":"2024-03-24T11:53:35.198678+0000","last_fullsized":"2024-03-24T11:53:35.198678+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616541+0000","last_change":"2024-03-24T11:53:29.691119+0000","last_active":"2024-03-24T11:53:31.616541+0000","last_peered":"2024-03-24T11:53:31.616541+0000","last_clean":"2024-03-24T11:53:31.616541+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:53:31.616541+0000","last_undegraded":"2024-03-24T11:53:31.616541+0000","last_fullsized":"2024-03-24T11:53:31.616541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.691006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.691006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:54.100657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720471+0000","last_change":"2024-03-24T11:52:21.390110+0000","last_active":"2024-03-24T11:53:31.720471+0000","last_peered":"2024-03-24T11:53:31.720471+0000","last_clean":"2024-03-24T11:53:31.720471+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:53:31.720471+0000","last_undegraded":"2024-03-24T11:53:31.720471+0000","last_fullsized":"2024-03-24T11:53:31.720471+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:31.975034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616906+0000","last_change":"2024-03-24T11:52:19.366588+0000","last_active":"2024-03-24T11:53:31.616906+0000","last_peered":"2024-03-24T11:53:31.616906+0000","last_clean":"2024-03-24T11:53:31.616906+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:53:31.616906+0000","last_undegraded":"2024-03-24T11:53:31.616906+0000","last_fullsized":"2024-03-24T11:53:31.616906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:33.102338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616081+0000","last_change":"2024-03-24T11:53:24.679709+0000","last_active":"2024-03-24T11:53:31.616081+0000","last_peered":"2024-03-24T11:53:31.616081+0000","last_clean":"2024-03-24T11:53:31.616081+0000","last_became_active":"2024-03-24T11:52:15.146310+0000","last_became_peered":"2024-03-24T11:52:15.146310+0000","last_unstale":"2024-03-24T11:53:31.616081+0000","last_undegraded":"2024-03-24T11:53:31.616081+0000","last_fullsized":"2024-03-24T11:53:31.616081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:24.679579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.679579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:29.622468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616630+0000","last_change":"2024-03-24T11:53:24.917322+0000","last_active":"2024-03-24T11:53:31.616630+0000","last_peered":"2024-03-24T11:53:31.616630+0000","last_clean":"2024-03-24T11:53:31.616630+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:53:31.616630+0000","last_undegraded":"2024-03-24T11:53:31.616630+0000","last_fullsized":"2024-03-24T11:53:31.616630+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:53:24.917220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.917220+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.119999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012502+0000","last_change":"2024-03-24T11:52:19.387410+0000","last_active":"2024-03-24T11:53:32.012502+0000","last_peered":"2024-03-24T11:53:32.012502+0000","last_clean":"2024-03-24T11:53:32.012502+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:53:32.012502+0000","last_undegraded":"2024-03-24T11:53:32.012502+0000","last_fullsized":"2024-03-24T11:53:32.012502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:42.573057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:41.036651+0000","last_change":"2024-03-24T11:53:41.036691+0000","last_active":"2024-03-24T11:53:41.036651+0000","last_peered":"2024-03-24T11:53:41.036651+0000","last_clean":"2024-03-24T11:53:41.036651+0000","last_became_active":"2024-03-24T11:52:21.851069+0000","last_became_peered":"2024-03-24T11:52:21.851069+0000","last_unstale":"2024-03-24T11:53:41.036651+0000","last_undegraded":"2024-03-24T11:53:41.036651+0000","last_fullsized":"2024-03-24T11:53:41.036651+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:41.036572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:41.036572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:42.293567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616680+0000","last_change":"2024-03-24T11:53:18.576655+0000","last_active":"2024-03-24T11:53:31.616680+0000","last_peered":"2024-03-24T11:53:31.616680+0000","last_clean":"2024-03-24T11:53:31.616680+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:53:31.616680+0000","last_undegraded":"2024-03-24T11:53:31.616680+0000","last_fullsized":"2024-03-24T11:53:31.616680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:18.576538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:18.576538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.141440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:39.923797+0000","last_change":"2024-03-24T11:53:39.923840+0000","last_active":"2024-03-24T11:53:39.923797+0000","last_peered":"2024-03-24T11:53:39.923797+0000","last_clean":"2024-03-24T11:53:39.923797+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:53:39.923797+0000","last_undegraded":"2024-03-24T11:53:39.923797+0000","last_fullsized":"2024-03-24T11:53:39.923797+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:53:39.923699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.923699+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.413674+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669986+0000","last_change":"2024-03-24T11:52:19.375675+0000","last_active":"2024-03-24T11:53:31.669986+0000","last_peered":"2024-03-24T11:53:31.669986+0000","last_clean":"2024-03-24T11:53:31.669986+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:53:31.669986+0000","last_undegraded":"2024-03-24T11:53:31.669986+0000","last_fullsized":"2024-03-24T11:53:31.669986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:43.200192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:39.653174+0000","last_change":"2024-03-24T11:53:39.653196+0000","last_active":"2024-03-24T11:53:39.653174+0000","last_peered":"2024-03-24T11:53:39.653174+0000","last_clean":"2024-03-24T11:53:39.653174+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:53:39.653174+0000","last_undegraded":"2024-03-24T11:53:39.653174+0000","last_fullsized":"2024-03-24T11:53:39.653174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:39.653110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.653110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.778426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:40.019240+0000","last_change":"2024-03-24T11:53:40.019269+0000","last_active":"2024-03-24T11:53:40.019240+0000","last_peered":"2024-03-24T11:53:40.019240+0000","last_clean":"2024-03-24T11:53:40.019240+0000","last_became_active":"2024-03-24T11:52:15.142974+0000","last_became_peered":"2024-03-24T11:52:15.142974+0000","last_unstale":"2024-03-24T11:53:40.019240+0000","last_undegraded":"2024-03-24T11:53:40.019240+0000","last_fullsized":"2024-03-24T11:53:40.019240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:40.019165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:40.019165+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.538489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.726697+0000","last_change":"2024-03-24T11:53:35.726728+0000","last_active":"2024-03-24T11:53:35.726697+0000","last_peered":"2024-03-24T11:53:35.726697+0000","last_clean":"2024-03-24T11:53:35.726697+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:53:35.726697+0000","last_undegraded":"2024-03-24T11:53:35.726697+0000","last_fullsized":"2024-03-24T11:53:35.726697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:53:35.726630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.726630+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:37.790720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667930+0000","last_change":"2024-03-24T11:52:19.389693+0000","last_active":"2024-03-24T11:53:31.667930+0000","last_peered":"2024-03-24T11:53:31.667930+0000","last_clean":"2024-03-24T11:53:31.667930+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:53:31.667930+0000","last_undegraded":"2024-03-24T11:53:31.667930+0000","last_fullsized":"2024-03-24T11:53:31.667930+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:47.063026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616276+0000","last_change":"2024-03-24T11:52:21.854938+0000","last_active":"2024-03-24T11:53:31.616276+0000","last_peered":"2024-03-24T11:53:31.616276+0000","last_clean":"2024-03-24T11:53:31.616276+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:53:31.616276+0000","last_undegraded":"2024-03-24T11:53:31.616276+0000","last_fullsized":"2024-03-24T11:53:31.616276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:40.554101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.670343+0000","last_change":"2024-03-24T11:53:15.292341+0000","last_active":"2024-03-24T11:53:31.670343+0000","last_peered":"2024-03-24T11:53:31.670343+0000","last_clean":"2024-03-24T11:53:31.670343+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:53:31.670343+0000","last_undegraded":"2024-03-24T11:53:31.670343+0000","last_fullsized":"2024-03-24T11:53:31.670343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:53:15.292233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.790414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.667595+0000","last_change":"2024-03-24T11:53:26.665675+0000","last_active":"2024-03-24T11:53:31.667595+0000","last_peered":"2024-03-24T11:53:31.667595+0000","last_clean":"2024-03-24T11:53:31.667595+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:53:31.667595+0000","last_undegraded":"2024-03-24T11:53:31.667595+0000","last_fullsized":"2024-03-24T11:53:31.667595+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:53:26.665554+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:48.241747+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198323+0000","last_change":"2024-03-24T11:52:19.382757+0000","last_active":"2024-03-24T11:53:35.198323+0000","last_peered":"2024-03-24T11:53:35.198323+0000","last_clean":"2024-03-24T11:53:35.198323+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:53:35.198323+0000","last_undegraded":"2024-03-24T11:53:35.198323+0000","last_fullsized":"2024-03-24T11:53:35.198323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:41.610324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615765+0000","last_change":"2024-03-24T11:52:21.851542+0000","last_active":"2024-03-24T11:53:31.615765+0000","last_peered":"2024-03-24T11:53:31.615765+0000","last_clean":"2024-03-24T11:53:31.615765+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:53:31.615765+0000","last_undegraded":"2024-03-24T11:53:31.615765+0000","last_fullsized":"2024-03-24T11:53:31.615765+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.109536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.012500+0000","last_change":"2024-03-24T11:52:19.388558+0000","last_active":"2024-03-24T11:53:32.012500+0000","last_peered":"2024-03-24T11:53:32.012500+0000","last_clean":"2024-03-24T11:53:32.012500+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:53:32.012500+0000","last_undegraded":"2024-03-24T11:53:32.012500+0000","last_fullsized":"2024-03-24T11:53:32.012500+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:38.924350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616185+0000","last_change":"2024-03-24T11:53:30.710184+0000","last_active":"2024-03-24T11:53:31.616185+0000","last_peered":"2024-03-24T11:53:31.616185+0000","last_clean":"2024-03-24T11:53:31.616185+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:53:31.616185+0000","last_undegraded":"2024-03-24T11:53:31.616185+0000","last_fullsized":"2024-03-24T11:53:31.616185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.710079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.710079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.148875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668261+0000","last_change":"2024-03-24T11:52:17.351215+0000","last_active":"2024-03-24T11:53:31.668261+0000","last_peered":"2024-03-24T11:53:31.668261+0000","last_clean":"2024-03-24T11:53:31.668261+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:53:31.668261+0000","last_undegraded":"2024-03-24T11:53:31.668261+0000","last_fullsized":"2024-03-24T11:53:31.668261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:43.604371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011981+0000","last_change":"2024-03-24T11:52:21.388941+0000","last_active":"2024-03-24T11:53:32.011981+0000","last_peered":"2024-03-24T11:53:32.011981+0000","last_clean":"2024-03-24T11:53:32.011981+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:53:32.011981+0000","last_undegraded":"2024-03-24T11:53:32.011981+0000","last_fullsized":"2024-03-24T11:53:32.011981+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:34.557613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198802+0000","last_change":"2024-03-24T11:52:19.383564+0000","last_active":"2024-03-24T11:53:35.198802+0000","last_peered":"2024-03-24T11:53:35.198802+0000","last_clean":"2024-03-24T11:53:35.198802+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:53:35.198802+0000","last_undegraded":"2024-03-24T11:53:35.198802+0000","last_fullsized":"2024-03-24T11:53:35.198802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720264+0000","last_change":"2024-03-24T11:52:15.135547+0000","last_active":"2024-03-24T11:53:31.720264+0000","last_peered":"2024-03-24T11:53:31.720264+0000","last_clean":"2024-03-24T11:53:31.720264+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:53:31.720264+0000","last_undegraded":"2024-03-24T11:53:31.720264+0000","last_fullsized":"2024-03-24T11:53:31.720264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.957766+0000","last_change":"2024-03-24T11:53:31.957804+0000","last_active":"2024-03-24T11:53:31.957766+0000","last_peered":"2024-03-24T11:53:31.957766+0000","last_clean":"2024-03-24T11:53:31.957766+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:53:31.957766+0000","last_undegraded":"2024-03-24T11:53:31.957766+0000","last_fullsized":"2024-03-24T11:53:31.957766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:31.957706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.957706+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.121723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.724334+0000","last_change":"2024-03-24T11:53:31.724380+0000","last_active":"2024-03-24T11:53:31.724334+0000","last_peered":"2024-03-24T11:53:31.724334+0000","last_clean":"2024-03-24T11:53:31.724334+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:53:31.724334+0000","last_undegraded":"2024-03-24T11:53:31.724334+0000","last_fullsized":"2024-03-24T11:53:31.724334+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:53:31.724235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.724235+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.862511+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198355+0000","last_change":"2024-03-24T11:53:19.389995+0000","last_active":"2024-03-24T11:53:35.198355+0000","last_peered":"2024-03-24T11:53:35.198355+0000","last_clean":"2024-03-24T11:53:35.198355+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:53:35.198355+0000","last_undegraded":"2024-03-24T11:53:35.198355+0000","last_fullsized":"2024-03-24T11:53:35.198355+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:19.389895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:19.389895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:38.977195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720881+0000","last_change":"2024-03-24T11:52:15.134671+0000","last_active":"2024-03-24T11:53:31.720881+0000","last_peered":"2024-03-24T11:53:31.720881+0000","last_clean":"2024-03-24T11:53:31.720881+0000","last_became_active":"2024-03-24T11:52:15.134326+0000","last_became_peered":"2024-03-24T11:52:15.134326+0000","last_unstale":"2024-03-24T11:53:31.720881+0000","last_undegraded":"2024-03-24T11:53:31.720881+0000","last_fullsized":"2024-03-24T11:53:31.720881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:34.885035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:34.756150+0000","last_change":"2024-03-24T11:53:34.756186+0000","last_active":"2024-03-24T11:53:34.756150+0000","last_peered":"2024-03-24T11:53:34.756150+0000","last_clean":"2024-03-24T11:53:34.756150+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:53:34.756150+0000","last_undegraded":"2024-03-24T11:53:34.756150+0000","last_fullsized":"2024-03-24T11:53:34.756150+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:53:34.756054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:34.756054+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.815024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722392+0000","last_change":"2024-03-24T11:52:21.388549+0000","last_active":"2024-03-24T11:53:31.722392+0000","last_peered":"2024-03-24T11:53:31.722392+0000","last_clean":"2024-03-24T11:53:31.722392+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:53:31.722392+0000","last_undegraded":"2024-03-24T11:53:31.722392+0000","last_fullsized":"2024-03-24T11:53:31.722392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:38.978622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.720558+0000","last_change":"2024-03-24T11:52:19.387447+0000","last_active":"2024-03-24T11:53:31.720558+0000","last_peered":"2024-03-24T11:53:31.720558+0000","last_clean":"2024-03-24T11:53:31.720558+0000","last_became_active":"2024-03-24T11:52:19.387115+0000","last_became_peered":"2024-03-24T11:52:19.387115+0000","last_unstale":"2024-03-24T11:53:31.720558+0000","last_undegraded":"2024-03-24T11:53:31.720558+0000","last_fullsized":"2024-03-24T11:53:31.720558+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.579121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722205+0000","last_change":"2024-03-24T11:53:24.386409+0000","last_active":"2024-03-24T11:53:31.722205+0000","last_peered":"2024-03-24T11:53:31.722205+0000","last_clean":"2024-03-24T11:53:31.722205+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:53:31.722205+0000","last_undegraded":"2024-03-24T11:53:31.722205+0000","last_fullsized":"2024-03-24T11:53:31.722205+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:24.386208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.386208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:43.956194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.668010+0000","last_change":"2024-03-24T11:52:17.354890+0000","last_active":"2024-03-24T11:53:31.668010+0000","last_peered":"2024-03-24T11:53:31.668010+0000","last_clean":"2024-03-24T11:53:31.668010+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:53:31.668010+0000","last_undegraded":"2024-03-24T11:53:31.668010+0000","last_fullsized":"2024-03-24T11:53:31.668010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:39.488951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.615816+0000","last_change":"2024-03-24T11:52:21.391660+0000","last_active":"2024-03-24T11:53:31.615816+0000","last_peered":"2024-03-24T11:53:31.615816+0000","last_clean":"2024-03-24T11:53:31.615816+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:53:31.615816+0000","last_undegraded":"2024-03-24T11:53:31.615816+0000","last_fullsized":"2024-03-24T11:53:31.615816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:37.583418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:37.463182+0000","last_change":"2024-03-24T11:53:37.463221+0000","last_active":"2024-03-24T11:53:37.463182+0000","last_peered":"2024-03-24T11:53:37.463182+0000","last_clean":"2024-03-24T11:53:37.463182+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:53:37.463182+0000","last_undegraded":"2024-03-24T11:53:37.463182+0000","last_fullsized":"2024-03-24T11:53:37.463182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.463110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.463110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.363432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722570+0000","last_change":"2024-03-24T11:53:30.449547+0000","last_active":"2024-03-24T11:53:31.722570+0000","last_peered":"2024-03-24T11:53:31.722570+0000","last_clean":"2024-03-24T11:53:31.722570+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:53:31.722570+0000","last_undegraded":"2024-03-24T11:53:31.722570+0000","last_fullsized":"2024-03-24T11:53:31.722570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.449442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.449442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:36.454836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.669781+0000","last_change":"2024-03-24T11:52:17.424107+0000","last_active":"2024-03-24T11:53:31.669781+0000","last_peered":"2024-03-24T11:53:31.669781+0000","last_clean":"2024-03-24T11:53:31.669781+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:53:31.669781+0000","last_undegraded":"2024-03-24T11:53:31.669781+0000","last_fullsized":"2024-03-24T11:53:31.669781+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:36.849775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.616605+0000","last_change":"2024-03-24T11:52:21.387355+0000","last_active":"2024-03-24T11:53:31.616605+0000","last_peered":"2024-03-24T11:53:31.616605+0000","last_clean":"2024-03-24T11:53:31.616605+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:53:31.616605+0000","last_undegraded":"2024-03-24T11:53:31.616605+0000","last_fullsized":"2024-03-24T11:53:31.616605+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.688825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722137+0000","last_change":"2024-03-24T11:52:19.384147+0000","last_active":"2024-03-24T11:53:31.722137+0000","last_peered":"2024-03-24T11:53:31.722137+0000","last_clean":"2024-03-24T11:53:31.722137+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:53:31.722137+0000","last_undegraded":"2024-03-24T11:53:31.722137+0000","last_fullsized":"2024-03-24T11:53:31.722137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:32.180888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:37.995330+0000","last_change":"2024-03-24T11:53:37.995353+0000","last_active":"2024-03-24T11:53:37.995330+0000","last_peered":"2024-03-24T11:53:37.995330+0000","last_clean":"2024-03-24T11:53:37.995330+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:53:37.995330+0000","last_undegraded":"2024-03-24T11:53:37.995330+0000","last_fullsized":"2024-03-24T11:53:37.995330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.995261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:56.380355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:36.884150+0000","last_change":"2024-03-24T11:53:36.884179+0000","last_active":"2024-03-24T11:53:36.884150+0000","last_peered":"2024-03-24T11:53:36.884150+0000","last_clean":"2024-03-24T11:53:36.884150+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:53:36.884150+0000","last_undegraded":"2024-03-24T11:53:36.884150+0000","last_fullsized":"2024-03-24T11:53:36.884150+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:36.884086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.884086+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.784016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:32.011656+0000","last_change":"2024-03-24T11:52:21.852577+0000","last_active":"2024-03-24T11:53:32.011656+0000","last_peered":"2024-03-24T11:53:32.011656+0000","last_clean":"2024-03-24T11:53:32.011656+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:53:32.011656+0000","last_undegraded":"2024-03-24T11:53:32.011656+0000","last_fullsized":"2024-03-24T11:53:32.011656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:31.722452+0000","last_change":"2024-03-24T11:53:25.429178+0000","last_active":"2024-03-24T11:53:31.722452+0000","last_peered":"2024-03-24T11:53:31.722452+0000","last_clean":"2024-03-24T11:53:31.722452+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:53:31.722452+0000","last_undegraded":"2024-03-24T11:53:31.722452+0000","last_fullsized":"2024-03-24T11:53:31.722452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:25.429098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.429098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:46.655256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:37.846009+0000","last_change":"2024-03-24T11:53:37.846040+0000","last_active":"2024-03-24T11:53:37.846009+0000","last_peered":"2024-03-24T11:53:37.846009+0000","last_clean":"2024-03-24T11:53:37.846009+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:53:37.846009+0000","last_undegraded":"2024-03-24T11:53:37.846009+0000","last_fullsized":"2024-03-24T11:53:37.846009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:37.845952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.845952+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.539200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T11:53:35.198925+0000","last_change":"2024-03-24T11:53:31.671828+0000","last_active":"2024-03-24T11:53:35.198925+0000","last_peered":"2024-03-24T11:53:35.198925+0000","last_clean":"2024-03-24T11:53:35.198925+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:53:35.198925+0000","last_undegraded":"2024-03-24T11:53:35.198925+0000","last_fullsized":"2024-03-24T11:53:35.198925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:31.671692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.671692+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:49.989770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005077,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1388544,"data_stored":943628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332124,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32528,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712624,"statfs":{"total":95995035648,"available":95961726976,"internally_reserved":0,"allocated":1384448,"data_stored":943446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":802816,"data_stored":352657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32532,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712620,"statfs":{"total":95995035648,"available":95961722880,"internally_reserved":0,"allocated":1388544,"data_stored":942495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":353077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":353507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920805,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":806912,"data_stored":352692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27796,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717356,"statfs":{"total":95995035648,"available":95966572544,"internally_reserved":0,"allocated":798720,"data_stored":352804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:53:43.862 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:53:43.862 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T11:53:43.862 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T11:53:43.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T11:53:44.111 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:44.222 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: 4.19 scrub starts 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[30223]: 4.19 scrub ok 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T11:53:44.223 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T11:53:44.224 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:53:44.224 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T11:53:44.224 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T11:53:44.224 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: 4.19 scrub starts 2024-03-24T11:53:44.224 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:43 smithi064 ceph-mon[42243]: 4.19 scrub ok 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: 4.19 scrub starts 2024-03-24T11:53:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:43 smithi193 ceph-mon[33593]: 4.19 scrub ok 2024-03-24T11:53:45.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[30223]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:45.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[30223]: from='client.14760 v1:172.21.15.64:0/3280748574' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:45.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T11:53:45.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T11:53:45.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[30223]: 4.d scrub starts 2024-03-24T11:53:45.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[30223]: 4.d scrub ok 2024-03-24T11:53:45.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[30223]: 4.1d deep-scrub starts 2024-03-24T11:53:45.210 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[30223]: 4.1d deep-scrub ok 2024-03-24T11:53:45.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[42243]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:45.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[42243]: from='client.14760 v1:172.21.15.64:0/3280748574' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:45.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T11:53:45.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T11:53:45.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[42243]: 4.d scrub starts 2024-03-24T11:53:45.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[42243]: 4.d scrub ok 2024-03-24T11:53:45.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[42243]: 4.1d deep-scrub starts 2024-03-24T11:53:45.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:44 smithi064 ceph-mon[42243]: 4.1d deep-scrub ok 2024-03-24T11:53:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:44 smithi193 ceph-mon[33593]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:44 smithi193 ceph-mon[33593]: from='client.14760 v1:172.21.15.64:0/3280748574' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:44 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T11:53:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:44 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T11:53:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:44 smithi193 ceph-mon[33593]: 4.d scrub starts 2024-03-24T11:53:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:44 smithi193 ceph-mon[33593]: 4.d scrub ok 2024-03-24T11:53:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:44 smithi193 ceph-mon[33593]: 4.1d deep-scrub starts 2024-03-24T11:53:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:44 smithi193 ceph-mon[33593]: 4.1d deep-scrub ok 2024-03-24T11:53:46.020 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:53:46.021 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T11:53:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[42243]: 6.17 scrub starts 2024-03-24T11:53:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[42243]: 6.17 scrub ok 2024-03-24T11:53:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[42243]: 5.12 deep-scrub starts 2024-03-24T11:53:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[42243]: 5.12 deep-scrub ok 2024-03-24T11:53:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T11:53:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T11:53:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[30223]: 6.17 scrub starts 2024-03-24T11:53:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[30223]: 6.17 scrub ok 2024-03-24T11:53:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[30223]: 5.12 deep-scrub starts 2024-03-24T11:53:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[30223]: 5.12 deep-scrub ok 2024-03-24T11:53:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T11:53:46.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:45 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T11:53:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:45 smithi193 ceph-mon[33593]: 6.17 scrub starts 2024-03-24T11:53:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:45 smithi193 ceph-mon[33593]: 6.17 scrub ok 2024-03-24T11:53:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:45 smithi193 ceph-mon[33593]: 5.12 deep-scrub starts 2024-03-24T11:53:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:45 smithi193 ceph-mon[33593]: 5.12 deep-scrub ok 2024-03-24T11:53:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:45 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T11:53:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:45 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T11:53:46.718 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:53:46.523Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001816633s 2024-03-24T11:53:46.749 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T11:53:46.749 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T11:53:46.750 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-24T11:53:46.763 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-24T11:53:46.763 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': 'f73f1dfd96849300c4322bd7259a3ab37f94c200'}} 2024-03-24T11:53:46.763 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-24T11:53:46.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-24T11:53:47.014 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 6.a deep-scrub starts 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 6.a deep-scrub ok 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T11:53:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 4.1e deep-scrub starts 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: 4.1e deep-scrub ok 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2562287297' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[30223]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 6.a deep-scrub starts 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 6.a deep-scrub ok 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 4.1e deep-scrub starts 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: 4.1e deep-scrub ok 2024-03-24T11:53:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2562287297' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:53:47.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:46 smithi064 ceph-mon[42243]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:53:47.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:53:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:53:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T11:53:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T11:53:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T11:53:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T11:53:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 6.a deep-scrub starts 2024-03-24T11:53:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 6.a deep-scrub ok 2024-03-24T11:53:47.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T11:53:47.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T11:53:47.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 4.1e deep-scrub starts 2024-03-24T11:53:47.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: 4.1e deep-scrub ok 2024-03-24T11:53:47.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2562287297' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:53:47.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:46 smithi193 ceph-mon[33593]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:53:47.395 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:53:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:47 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:47 smithi193 ceph-mon[33593]: 4.14 deep-scrub starts 2024-03-24T11:53:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:47 smithi193 ceph-mon[33593]: 4.14 deep-scrub ok 2024-03-24T11:53:48.100 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:47 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:48.100 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:47 smithi064 ceph-mon[30223]: 4.14 deep-scrub starts 2024-03-24T11:53:48.100 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:47 smithi064 ceph-mon[30223]: 4.14 deep-scrub ok 2024-03-24T11:53:48.100 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:47 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:48.100 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:47 smithi064 ceph-mon[42243]: 4.14 deep-scrub starts 2024-03-24T11:53:48.100 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:47 smithi064 ceph-mon[42243]: 4.14 deep-scrub ok 2024-03-24T11:53:48.916 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:48] ENGINE Bus STOPPING 2024-03-24T11:53:48.916 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:53:48.917 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:48] ENGINE Bus STOPPED 2024-03-24T11:53:48.917 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:48] ENGINE Bus STARTING 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T11:53:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:48 smithi193 ceph-mon[33593]: from='client.34655 ' entity='client.admin' 2024-03-24T11:53:49.190 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:48] ENGINE Serving on http://:::9283 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:53:48] ENGINE Bus STARTED 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T11:53:49.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[30223]: from='client.34655 ' entity='client.admin' 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T11:53:49.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:48 smithi064 ceph-mon[42243]: from='client.34655 ' entity='client.admin' 2024-03-24T11:53:49.587 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-24T11:53:49.587 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:53:49.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:53:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:50.775 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:50 smithi193 systemd[1]: Starting Ceph grafana.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:53:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[30223]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-24T11:53:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T11:53:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T11:53:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[30223]: 6.d scrub starts 2024-03-24T11:53:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[30223]: 6.d scrub ok 2024-03-24T11:53:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[42243]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-24T11:53:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T11:53:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T11:53:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[42243]: 6.d scrub starts 2024-03-24T11:53:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:50 smithi064 ceph-mon[42243]: 6.d scrub ok 2024-03-24T11:53:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:50 smithi193 ceph-mon[33593]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-24T11:53:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:50 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T11:53:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:50 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T11:53:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:50 smithi193 ceph-mon[33593]: 6.d scrub starts 2024-03-24T11:53:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:50 smithi193 ceph-mon[33593]: 6.d scrub ok 2024-03-24T11:53:51.086 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:50 smithi193 podman[66917]: 2024-03-24 11:53:50.86945067 +0000 UTC m=+0.020200603 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-24T11:53:51.086 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:50 smithi193 podman[66917]: 2024-03-24 11:53:50.988025592 +0000 UTC m=+0.138775528 container create ccaac67d7ff6bb7226d9a3c5515ff4d8a7f695f1e1968872399fd5c648fccd2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, name=ubi8, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.buildah.version=1.32.0, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , architecture=x86_64, release=239.1651231664) 2024-03-24T11:53:51.532 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 podman[66917]: 2024-03-24 11:53:51.282032489 +0000 UTC m=+0.432782436 container init ccaac67d7ff6bb7226d9a3c5515ff4d8a7f695f1e1968872399fd5c648fccd2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-type=git, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, architecture=x86_64, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, distribution-scope=public, version=8.5, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-24T11:53:51.532 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 podman[66917]: 2024-03-24 11:53:51.287262862 +0000 UTC m=+0.438012798 container start ccaac67d7ff6bb7226d9a3c5515ff4d8a7f695f1e1968872399fd5c648fccd2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a, version=8.5, description=Ceph Grafana Container, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, name=ubi8, com.redhat.component=ubi8-container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.buildah.version=1.32.0, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8) 2024-03-24T11:53:51.532 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 bash[66917]: ccaac67d7ff6bb7226d9a3c5515ff4d8a7f695f1e1968872399fd5c648fccd2f 2024-03-24T11:53:51.532 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 systemd[1]: Started Ceph grafana.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:53:51.532 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.442728963Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-24T11:53:51.532 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.442978703Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-24T11:53:51.532 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443003305Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-24T11:53:51.532 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443013397Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-24T11:53:51.533 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443024056Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T11:53:51.533 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443032398Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T11:53:51.533 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443040536Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T11:53:51.533 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443048586Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T11:53:51.533 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443057323Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-24T11:53:51.533 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443065551Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-24T11:53:51.533 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443088345Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-24T11:53:51.533 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443095522Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443102817Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=settings t=2024-03-24T11:53:51.443110603Z level=info msg="App mode production" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=sqlstore t=2024-03-24T11:53:51.443181131Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=sqlstore t=2024-03-24T11:53:51.443202334Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.444087053Z level=info msg="Starting DB migrations" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.44417683Z level=info msg="Executing migration" id="create migration_log table" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.444752318Z level=info msg="Executing migration" id="create user table" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.44523817Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.445711626Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.446184206Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.44677588Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.447304673Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-24T11:53:51.534 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.44824995Z level=info msg="Executing migration" id="create user table v2" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.448754082Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.449274306Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.449825269Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.450196411Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.450550158Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.450979762Z level=info msg="Executing migration" id="Update user table charset" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.451242856Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.451637867Z level=info msg="Executing migration" id="Add missing user data" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.452009193Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.452432546Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.452821924Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.453245757Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-24T11:53:51.535 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.454352768Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-24T11:53:51.536 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.454856861Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-24T11:53:51.536 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.455304186Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-24T11:53:51.536 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.455685658Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-24T11:53:51.536 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.456108152Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-24T11:53:51.536 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.456496098Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.456756687Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.45714172Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.457482231Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.457830372Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.458200412Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.458784658Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.459198528Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.459571964Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.459961657Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.46200788Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-24T11:53:51.537 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.462404291Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.462701389Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.46304571Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.463376537Z level=info msg="Executing migration" id="create star table" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.463708446Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.464108077Z level=info msg="Executing migration" id="create org table v1" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.464487491Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.464880075Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.465248362Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.465632704Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.466038419Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.466420205Z level=info msg="Executing migration" id="Update org table charset" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.466670761Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-24T11:53:51.538 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.466959024Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.467257241Z level=info msg="Executing migration" id="create dashboard table" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.467617929Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.46801606Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.468400185Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.468804757Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.471343232Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.471766336Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.474567423Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.47731872Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.480339086Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.480776667Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.481123033Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.481491059Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.481763178Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-24T11:53:51.539 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.482296674Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.484149759Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.48691494Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.489573455Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.492279091Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.495020001Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.495437834Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.495703247Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.498132313Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.498675879Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.501257747Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.501775974Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.504309061Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.50686322Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.509413032Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-24T11:53:51.540 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.509826884Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.512575432Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.515376987Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.517994198Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.518882833Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.521554794Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.5223629Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.523184089Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.523517914Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.524274546Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.524873092Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.525299851Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.526070118Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.526405923Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.526833149Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-24T11:53:51.541 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.527679715Z level=info msg="Executing migration" id="create data_source table" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.52848332Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.528910517Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.529378886Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.530169455Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.53057491Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.531593011Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.532427934Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.532958875Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.533822218Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.534672585Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.535514908Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-24T11:53:51.542 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.536279409Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-24T11:53:51.784 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.537103766Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-24T11:53:51.784 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.537946724Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.538798724Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.539126081Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.539952809Z level=info msg="Executing migration" id="Add uid column" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.540490486Z level=info msg="Executing migration" id="Update uid value" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.540818742Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.541261765Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.541698741Z level=info msg="Executing migration" id="create api_key table" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.542116844Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.54296654Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.543819751Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.544690578Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-24T11:53:51.785 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.54512165Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-24T11:53:51.786 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.545906279Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-24T11:53:51.786 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.546711707Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-24T11:53:51.786 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.548085578Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-24T11:53:51.786 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.548905492Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-24T11:53:51.786 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.54973005Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.550540629Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.551370911Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.552203239Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.553001307Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.55387521Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.554687215Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.555479492Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.556336678Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-24T11:53:51.787 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.557101967Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.557961387Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.558822638Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.559661019Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.560540179Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.561395138Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.562238378Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.563106345Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.563879379Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.564731122Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.565544763Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.568301485Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.571020745Z level=info msg="Executing migration" id="create quota table v1" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.573614637Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.576382534Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-24T11:53:51.788 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.579103199Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.581889057Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.584611238Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.587315529Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.589896473Z level=info msg="Executing migration" id="create session table" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.592843185Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.595624563Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.598325789Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.60115202Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.60390411Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.606556395Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.60928106Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.611945022Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.614675099Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.617342367Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.619843146Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-24T11:53:51.789 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.62244105Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.622840024Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.625397635Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.626274935Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.628955222Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.631521492Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.634249655Z level=info msg="Executing migration" id="create alert table v1" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.639824132Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.640425867Z level=info msg="Executing migration" id="add index alert state" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.640998005Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.643589787Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.64627522Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.648915752Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-24T11:53:51.790 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.651574163Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.655290329Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.657530479Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.658371801Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.659118942Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.659490031Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.660294978Z level=info msg="Executing migration" id="Add column is_default" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.660915418Z level=info msg="Executing migration" id="Add column frequency" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.661727158Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.66254715Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.663371701Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.66416814Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.664939168Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.665809435Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.666648427Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-24T11:53:51.791 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.667488082Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.668313476Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.669151983Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.671820423Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.674495453Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.677062045Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.67970075Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.682590546Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.685283546Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.687961369Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.690636724Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.693297098Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.696191825Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.698913371Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-24T11:53:51.792 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.701528919Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.704389097Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.707113087Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.70969581Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.712475591Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.715191871Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.7178129Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.720516305Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.720956911Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.721837625Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.722680591Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.723522731Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.726305778Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.729177944Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-24T11:53:51.793 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.731952862Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-24T11:53:51.794 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.734601006Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-24T11:53:51.794 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.73736844Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-24T11:53:51.794 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.740172986Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-24T11:53:51.794 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.742889979Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-24T11:53:51.794 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.745707571Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-24T11:53:51.794 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.748453625Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-24T11:53:51.794 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.751299211Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.754003017Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.756781923Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.75965557Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.762517967Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.765359227Z level=info msg="Executing migration" id="Move region to single row" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.768046158Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.770777527Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.773598337Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.776384536Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.779017567Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.781743252Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.784497575Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.787103371Z level=info msg="Executing migration" id="create test_data table" 2024-03-24T11:53:51.795 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.787532359Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-24T11:53:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:53:51.818 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 5.2 deep-scrub starts 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 5.2 deep-scrub ok 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 2.4 deep-scrub starts 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 2.4 deep-scrub ok 2024-03-24T11:53:52.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 2.1e scrub starts 2024-03-24T11:53:52.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-mon[33593]: 2.1e scrub ok 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.789871522Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.792276901Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.79511024Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.797848094Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.800490119Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.803186327Z level=info msg="Executing migration" id="create team table" 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.805867304Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.808615501Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-24T11:53:52.037 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.811409897Z level=info msg="Executing migration" id="create team member table" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.814196823Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.816893471Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.819539338Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.820440326Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.82124203Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.822082743Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.822876024Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.823770017Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.824591506Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.825216471Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-24T11:53:52.038 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.825679052Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.82650185Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.827268593Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.828062271Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.828520444Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.829370882Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.82983308Z level=info msg="Executing migration" id="create tag table" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.830212769Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.831122121Z level=info msg="Executing migration" id="create login attempt table" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.831572654Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.832373506Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.833235773Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.835126504Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.835645987Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-24T11:53:52.039 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.836482332Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.837315456Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.838311582Z level=info msg="Executing migration" id="create user auth table" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.838801275Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.83961994Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.840406479Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.841195167Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.841923146Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.842646815Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.843576443Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.844402459Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.845248392Z level=info msg="Executing migration" id="create server_lock table" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.845680274Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.846531209Z level=info msg="Executing migration" id="create user auth token table" 2024-03-24T11:53:52.040 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.847339289Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.850272378Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.850740821Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.853463932Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.856066037Z level=info msg="Executing migration" id="create cache_data table" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.858866617Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.85932861Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.861962902Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.864738594Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-24T11:53:52.041 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.865077009Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-24T11:53:52.042 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.865391545Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-24T11:53:52.042 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.867463745Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-24T11:53:52.042 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.870178315Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.870724445Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.871053334Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.871500827Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.871953787Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.872402321Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.872856595Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.873661253Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.874140787Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.874467185Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.874877829Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.875328147Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.875747981Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.876077567Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.876507126Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-24T11:53:52.043 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.87695105Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.877400494Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.877861676Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.878684379Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.879123537Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.879571046Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.882972297Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.886346592Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.886838194Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.88727848Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.888122071Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.888554642Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.88900947Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.889452054Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-24T11:53:52.044 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.889891098Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.890204808Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.891080114Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.891893344Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.89272707Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.893170811Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.893614802Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.894472934Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.895356029Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.895836624Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.89666148Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.897489735Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.897838874Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-24T11:53:52.045 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.89831677Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.90104592Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.901542056Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.901836064Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.90270729Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.903579198Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.906247036Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.909058973Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.911833022Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.914354765Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.917014197Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.919692969Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.922261386Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-24T11:53:52.046 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.925062055Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.927516989Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.930257543Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.93264452Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.935245423Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.937868499Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.940556953Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.943212687Z level=info msg="Executing migration" id="create alert_image table" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.943634276Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.946284866Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.94899234Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.951502097Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.951937138Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:53:52.047 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.952013Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.954601196Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.955493119Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.956194215Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.956318054Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.956427522Z level=info msg="alerts found to migrate" alerts=0 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.956814195Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.95772416Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.9583589Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.959244664Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.959813333Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.960726287Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-24T11:53:52.048 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.961427569Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.96221926Z level=info msg="Executing migration" id="create data_keys table" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.962672484Z level=info msg="Executing migration" id="create secrets table" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.963100708Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.967302206Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.968205029Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.96916322Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.973507984Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.97774231Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.978640793Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.979305183Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.980114462Z level=info msg="Executing migration" id="create permission table" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.980527322Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.980966519Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-24T11:53:52.049 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.981871575Z level=info msg="Executing migration" id="create role table" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.982563428Z level=info msg="Executing migration" id="add column display_name" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.983507187Z level=info msg="Executing migration" id="add column group_name" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.984446901Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.985215288Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.986171957Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.986955915Z level=info msg="Executing migration" id="create team role table" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.987980937Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.988687731Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.989618197Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.990308893Z level=info msg="Executing migration" id="create user role table" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.991267226Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.991917758Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.994593851Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-24T11:53:52.050 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:51.997268762Z level=info msg="Executing migration" id="create builtin role table" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.000014502Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.002522293Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.005441401Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.008115835Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.010892787Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.013413886Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.016169754Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.019050355Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.021785865Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.024672558Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-24T11:53:52.051 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.027248543Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-24T11:53:52.052 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.030111155Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-24T11:53:52.052 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.033034232Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-24T11:53:52.052 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.035623034Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-24T11:53:52.052 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.038170544Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 5.2 deep-scrub starts 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 5.2 deep-scrub ok 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 2.4 deep-scrub starts 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 2.4 deep-scrub ok 2024-03-24T11:53:52.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 2.1e scrub starts 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[30223]: 2.1e scrub ok 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 5.2 deep-scrub starts 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 5.2 deep-scrub ok 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T11:53:52.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 2.4 deep-scrub starts 2024-03-24T11:53:52.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 2.4 deep-scrub ok 2024-03-24T11:53:52.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:52.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 2.1e scrub starts 2024-03-24T11:53:52.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:51 smithi064 ceph-mon[42243]: 2.1e scrub ok 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.040836788Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.043518454Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.046157665Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.04884663Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.051471467Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.054040445Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.056697908Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.059216865Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-24T11:53:52.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.061687564Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.064371866Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.066941088Z level=info msg="Executing migration" id="add correlation config column" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.069635324Z level=info msg="Executing migration" id="create entity_events table" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.072082827Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.074664864Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.074762674Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.077599081Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.077710339Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:53:52.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.080112882Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-24T11:53:52.339 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.082909827Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-24T11:53:52.339 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.085478457Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:53:52.339 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.086213656Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:53:52.339 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.087022731Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:53:52.339 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.087981553Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:53:52.339 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.088629014Z level=info msg="Executing migration" id="Drop public config table" 2024-03-24T11:53:52.339 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.089543374Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.090222725Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.091150317Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.091813936Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.092737401Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.096032209Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.097288364Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.098348792Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.099156716Z level=info msg="Executing migration" id="add share column" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.100145311Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.101134959Z level=info msg="Executing migration" id="create file table" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.101799017Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.102763232Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.103454885Z level=info msg="Executing migration" id="create file_meta table" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.104386835Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-24T11:53:52.340 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.105028792Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.105826074Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.106724794Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.107544706Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.108265881Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.10932585Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.112141602Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.114991386Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.117837953Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.120391059Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.123207905Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.12595439Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.128894842Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.131490517Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-24T11:53:52.341 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.141143154Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.144011318Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.146652968Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.150122829Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.152778147Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.155562166Z level=info msg="Executing migration" id="create folder table" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.15818649Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.161004488Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.163571537Z level=info msg="Executing migration" id="Update folder title length" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.166021346Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=migrator t=2024-03-24T11:53:52.168777515Z level=info msg="migrations completed" performed=485 skipped=0 duration=724.608412ms 2024-03-24T11:53:52.342 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=sqlstore t=2024-03-24T11:53:52.169192838Z level=info msg="Created default organization" 2024-03-24T11:53:52.343 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=plugin.loader t=2024-03-24T11:53:52.223979134Z level=info msg="Plugin registered" pluginID=input 2024-03-24T11:53:52.343 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=plugin.loader t=2024-03-24T11:53:52.252333971Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-24T11:53:52.343 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=plugin.loader t=2024-03-24T11:53:52.252368338Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-24T11:53:52.343 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=secrets t=2024-03-24T11:53:52.25240182Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-24T11:53:52.343 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=query_data t=2024-03-24T11:53:52.25448928Z level=info msg="Query Service initialization" 2024-03-24T11:53:52.343 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=live.push_http t=2024-03-24T11:53:52.256784154Z level=info msg="Live Push Gateway initialization" 2024-03-24T11:53:52.610 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-03-24T11:53:51.145807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1017,"num_read_kb":863,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":627,"ondisk_log_size":627,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245320,"kb_used_data":8380,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715896,"statfs":{"total":767960285184,"available":767709077504,"internally_reserved":0,"allocated":8581120,"data_stored":4802722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.253690"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617008+0000","last_change":"2024-03-24T11:52:21.854369+0000","last_active":"2024-03-24T11:53:46.617008+0000","last_peered":"2024-03-24T11:53:46.617008+0000","last_clean":"2024-03-24T11:53:46.617008+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:53:46.617008+0000","last_undegraded":"2024-03-24T11:53:46.617008+0000","last_fullsized":"2024-03-24T11:53:46.617008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166063+0000","last_change":"2024-03-24T11:53:29.715820+0000","last_active":"2024-03-24T11:53:47.166063+0000","last_peered":"2024-03-24T11:53:47.166063+0000","last_clean":"2024-03-24T11:53:47.166063+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:53:47.166063+0000","last_undegraded":"2024-03-24T11:53:47.166063+0000","last_fullsized":"2024-03-24T11:53:47.166063+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.715710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:29.794324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.304010+0000","last_change":"2024-03-24T11:52:15.137687+0000","last_active":"2024-03-24T11:53:47.304010+0000","last_peered":"2024-03-24T11:53:47.304010+0000","last_clean":"2024-03-24T11:53:47.304010+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:53:47.304010+0000","last_undegraded":"2024-03-24T11:53:47.304010+0000","last_fullsized":"2024-03-24T11:53:47.304010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617878+0000","last_change":"2024-03-24T11:53:43.627728+0000","last_active":"2024-03-24T11:53:46.617878+0000","last_peered":"2024-03-24T11:53:46.617878+0000","last_clean":"2024-03-24T11:53:46.617878+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:53:46.617878+0000","last_undegraded":"2024-03-24T11:53:46.617878+0000","last_fullsized":"2024-03-24T11:53:46.617878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:43.627630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:43.627630+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:10.204254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166243+0000","last_change":"2024-03-24T11:53:30.710255+0000","last_active":"2024-03-24T11:53:47.166243+0000","last_peered":"2024-03-24T11:53:47.166243+0000","last_clean":"2024-03-24T11:53:47.166243+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:53:47.166243+0000","last_undegraded":"2024-03-24T11:53:47.166243+0000","last_fullsized":"2024-03-24T11:53:47.166243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.710143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.710143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.327391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617873+0000","last_change":"2024-03-24T11:52:19.378597+0000","last_active":"2024-03-24T11:53:46.617873+0000","last_peered":"2024-03-24T11:53:46.617873+0000","last_clean":"2024-03-24T11:53:46.617873+0000","last_became_active":"2024-03-24T11:52:19.378282+0000","last_became_peered":"2024-03-24T11:52:19.378282+0000","last_unstale":"2024-03-24T11:53:46.617873+0000","last_undegraded":"2024-03-24T11:53:46.617873+0000","last_fullsized":"2024-03-24T11:53:46.617873+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619200+0000","last_change":"2024-03-24T11:52:15.141672+0000","last_active":"2024-03-24T11:53:46.619200+0000","last_peered":"2024-03-24T11:53:46.619200+0000","last_clean":"2024-03-24T11:53:46.619200+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:53:46.619200+0000","last_undegraded":"2024-03-24T11:53:46.619200+0000","last_fullsized":"2024-03-24T11:53:46.619200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166240+0000","last_change":"2024-03-24T11:52:17.350943+0000","last_active":"2024-03-24T11:53:47.166240+0000","last_peered":"2024-03-24T11:53:47.166240+0000","last_clean":"2024-03-24T11:53:47.166240+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:53:47.166240+0000","last_undegraded":"2024-03-24T11:53:47.166240+0000","last_fullsized":"2024-03-24T11:53:47.166240+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748344+0000","last_change":"2024-03-24T11:53:25.779733+0000","last_active":"2024-03-24T11:53:46.748344+0000","last_peered":"2024-03-24T11:53:46.748344+0000","last_clean":"2024-03-24T11:53:46.748344+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:53:46.748344+0000","last_undegraded":"2024-03-24T11:53:46.748344+0000","last_fullsized":"2024-03-24T11:53:46.748344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.779615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:49.135209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331945+0000","last_change":"2024-03-24T11:53:45.463943+0000","last_active":"2024-03-24T11:53:47.331945+0000","last_peered":"2024-03-24T11:53:47.331945+0000","last_clean":"2024-03-24T11:53:47.331945+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:53:47.331945+0000","last_undegraded":"2024-03-24T11:53:47.331945+0000","last_fullsized":"2024-03-24T11:53:47.331945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.463814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.463814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:10.450722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332054+0000","last_change":"2024-03-24T11:53:35.593649+0000","last_active":"2024-03-24T11:53:47.332054+0000","last_peered":"2024-03-24T11:53:47.332054+0000","last_clean":"2024-03-24T11:53:47.332054+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:53:47.332054+0000","last_undegraded":"2024-03-24T11:53:47.332054+0000","last_fullsized":"2024-03-24T11:53:47.332054+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.593554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.593554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.653079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166350+0000","last_change":"2024-03-24T11:53:20.591717+0000","last_active":"2024-03-24T11:53:47.166350+0000","last_peered":"2024-03-24T11:53:47.166350+0000","last_clean":"2024-03-24T11:53:47.166350+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:53:47.166350+0000","last_undegraded":"2024-03-24T11:53:47.166350+0000","last_fullsized":"2024-03-24T11:53:47.166350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_clean_scrub_stamp":"2024-03-24T11:53:20.591632+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.910645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303458+0000","last_change":"2024-03-24T11:52:21.391890+0000","last_active":"2024-03-24T11:53:47.303458+0000","last_peered":"2024-03-24T11:53:47.303458+0000","last_clean":"2024-03-24T11:53:47.303458+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:53:47.303458+0000","last_undegraded":"2024-03-24T11:53:47.303458+0000","last_fullsized":"2024-03-24T11:53:47.303458+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.108281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748596+0000","last_change":"2024-03-24T11:52:19.381366+0000","last_active":"2024-03-24T11:53:46.748596+0000","last_peered":"2024-03-24T11:53:46.748596+0000","last_clean":"2024-03-24T11:53:46.748596+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:53:46.748596+0000","last_undegraded":"2024-03-24T11:53:46.748596+0000","last_fullsized":"2024-03-24T11:53:46.748596+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.508572+0000","last_change":"2024-03-24T11:53:47.508597+0000","last_active":"2024-03-24T11:53:47.508572+0000","last_peered":"2024-03-24T11:53:47.508572+0000","last_clean":"2024-03-24T11:53:47.508572+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:53:47.508572+0000","last_undegraded":"2024-03-24T11:53:47.508572+0000","last_fullsized":"2024-03-24T11:53:47.508572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:47.508490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.508490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:58.461301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166307+0000","last_change":"2024-03-24T11:53:17.621045+0000","last_active":"2024-03-24T11:53:47.166307+0000","last_peered":"2024-03-24T11:53:47.166307+0000","last_clean":"2024-03-24T11:53:47.166307+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:53:47.166307+0000","last_undegraded":"2024-03-24T11:53:47.166307+0000","last_fullsized":"2024-03-24T11:53:47.166307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:17.620911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:17.620911+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:20.912812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618332+0000","last_change":"2024-03-24T11:53:37.743905+0000","last_active":"2024-03-24T11:53:46.618332+0000","last_peered":"2024-03-24T11:53:46.618332+0000","last_clean":"2024-03-24T11:53:46.618332+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:53:46.618332+0000","last_undegraded":"2024-03-24T11:53:46.618332+0000","last_fullsized":"2024-03-24T11:53:46.618332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.743809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.743809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.308954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166575+0000","last_change":"2024-03-24T11:52:19.382663+0000","last_active":"2024-03-24T11:53:47.166575+0000","last_peered":"2024-03-24T11:53:47.166575+0000","last_clean":"2024-03-24T11:53:47.166575+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:53:47.166575+0000","last_undegraded":"2024-03-24T11:53:47.166575+0000","last_fullsized":"2024-03-24T11:53:47.166575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.250149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618397+0000","last_change":"2024-03-24T11:53:30.739186+0000","last_active":"2024-03-24T11:53:46.618397+0000","last_peered":"2024-03-24T11:53:46.618397+0000","last_clean":"2024-03-24T11:53:46.618397+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:53:46.618397+0000","last_undegraded":"2024-03-24T11:53:46.618397+0000","last_fullsized":"2024-03-24T11:53:46.618397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:30.739065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.739065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.055226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":115,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618980+0000","last_change":"2024-03-24T11:53:44.635166+0000","last_active":"2024-03-24T11:53:46.618980+0000","last_peered":"2024-03-24T11:53:46.618980+0000","last_clean":"2024-03-24T11:53:46.618980+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:53:46.618980+0000","last_undegraded":"2024-03-24T11:53:46.618980+0000","last_fullsized":"2024-03-24T11:53:46.618980+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:53:44.635073+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:07.128363+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166857+0000","last_change":"2024-03-24T11:53:27.651110+0000","last_active":"2024-03-24T11:53:47.166857+0000","last_peered":"2024-03-24T11:53:47.166857+0000","last_clean":"2024-03-24T11:53:47.166857+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:53:47.166857+0000","last_undegraded":"2024-03-24T11:53:47.166857+0000","last_fullsized":"2024-03-24T11:53:47.166857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:27.650990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:27.650990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:31.773205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617945+0000","last_change":"2024-03-24T11:53:42.989742+0000","last_active":"2024-03-24T11:53:46.617945+0000","last_peered":"2024-03-24T11:53:46.617945+0000","last_clean":"2024-03-24T11:53:46.617945+0000","last_became_active":"2024-03-24T11:52:19.384249+0000","last_became_peered":"2024-03-24T11:52:19.384249+0000","last_unstale":"2024-03-24T11:53:46.617945+0000","last_undegraded":"2024-03-24T11:53:46.617945+0000","last_fullsized":"2024-03-24T11:53:46.617945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:42.989664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:42.989664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:56.543730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619768+0000","last_change":"2024-03-24T11:53:33.356957+0000","last_active":"2024-03-24T11:53:46.619768+0000","last_peered":"2024-03-24T11:53:46.619768+0000","last_clean":"2024-03-24T11:53:46.619768+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:53:46.619768+0000","last_undegraded":"2024-03-24T11:53:46.619768+0000","last_fullsized":"2024-03-24T11:53:46.619768+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:33.356847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:33.356847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.122844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499590+0000","last_change":"2024-03-24T11:53:36.510517+0000","last_active":"2024-03-24T11:53:47.499590+0000","last_peered":"2024-03-24T11:53:47.499590+0000","last_clean":"2024-03-24T11:53:47.499590+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:53:47.499590+0000","last_undegraded":"2024-03-24T11:53:47.499590+0000","last_fullsized":"2024-03-24T11:53:47.499590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:36.510415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.510415+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.640197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617452+0000","last_change":"2024-03-24T11:53:36.945667+0000","last_active":"2024-03-24T11:53:46.617452+0000","last_peered":"2024-03-24T11:53:46.617452+0000","last_clean":"2024-03-24T11:53:46.617452+0000","last_became_active":"2024-03-24T11:52:21.390764+0000","last_became_peered":"2024-03-24T11:52:21.390764+0000","last_unstale":"2024-03-24T11:53:46.617452+0000","last_undegraded":"2024-03-24T11:53:46.617452+0000","last_fullsized":"2024-03-24T11:53:46.617452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:36.945578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.945578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:05.683323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.304037+0000","last_change":"2024-03-24T11:53:21.941429+0000","last_active":"2024-03-24T11:53:47.304037+0000","last_peered":"2024-03-24T11:53:47.304037+0000","last_clean":"2024-03-24T11:53:47.304037+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:53:47.304037+0000","last_undegraded":"2024-03-24T11:53:47.304037+0000","last_fullsized":"2024-03-24T11:53:47.304037+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_clean_scrub_stamp":"2024-03-24T11:53:21.941323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:31.533370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":66,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617005+0000","last_change":"2024-03-24T11:52:15.136259+0000","last_active":"2024-03-24T11:53:46.617005+0000","last_peered":"2024-03-24T11:53:46.617005+0000","last_clean":"2024-03-24T11:53:46.617005+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:53:46.617005+0000","last_undegraded":"2024-03-24T11:53:46.617005+0000","last_fullsized":"2024-03-24T11:53:46.617005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619780+0000","last_change":"2024-03-24T11:53:29.274282+0000","last_active":"2024-03-24T11:53:46.619780+0000","last_peered":"2024-03-24T11:53:46.619780+0000","last_clean":"2024-03-24T11:53:46.619780+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:53:46.619780+0000","last_undegraded":"2024-03-24T11:53:46.619780+0000","last_fullsized":"2024-03-24T11:53:46.619780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:53:29.274127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.274127+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:43.553788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332306+0000","last_change":"2024-03-24T11:53:39.602043+0000","last_active":"2024-03-24T11:53:47.332306+0000","last_peered":"2024-03-24T11:53:47.332306+0000","last_clean":"2024-03-24T11:53:47.332306+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:53:47.332306+0000","last_undegraded":"2024-03-24T11:53:47.332306+0000","last_fullsized":"2024-03-24T11:53:47.332306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:53:39.601937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.601937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.250197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.334525+0000","last_change":"2024-03-24T11:53:47.334525+0000","last_active":"2024-03-24T11:53:47.334525+0000","last_peered":"2024-03-24T11:53:47.334525+0000","last_clean":"2024-03-24T11:53:47.334525+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:53:47.334525+0000","last_undegraded":"2024-03-24T11:53:47.334525+0000","last_fullsized":"2024-03-24T11:53:47.334525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:47.334412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.334412+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:05.508632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748292+0000","last_change":"2024-03-24T11:53:35.814141+0000","last_active":"2024-03-24T11:53:46.748292+0000","last_peered":"2024-03-24T11:53:46.748292+0000","last_clean":"2024-03-24T11:53:46.748292+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:53:46.748292+0000","last_undegraded":"2024-03-24T11:53:46.748292+0000","last_fullsized":"2024-03-24T11:53:46.748292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.814032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.814032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.550457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.304025+0000","last_change":"2024-03-24T11:53:46.014153+0000","last_active":"2024-03-24T11:53:47.304025+0000","last_peered":"2024-03-24T11:53:47.304025+0000","last_clean":"2024-03-24T11:53:47.304025+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:53:47.304025+0000","last_undegraded":"2024-03-24T11:53:47.304025+0000","last_fullsized":"2024-03-24T11:53:47.304025+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_clean_scrub_stamp":"2024-03-24T11:53:46.014065+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:06.936760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617603+0000","last_change":"2024-03-24T11:52:21.854322+0000","last_active":"2024-03-24T11:53:46.617603+0000","last_peered":"2024-03-24T11:53:46.617603+0000","last_clean":"2024-03-24T11:53:46.617603+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:53:46.617603+0000","last_undegraded":"2024-03-24T11:53:46.617603+0000","last_fullsized":"2024-03-24T11:53:46.617603+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332115+0000","last_change":"2024-03-24T11:52:19.385213+0000","last_active":"2024-03-24T11:53:47.332115+0000","last_peered":"2024-03-24T11:53:47.332115+0000","last_clean":"2024-03-24T11:53:47.332115+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:53:47.332115+0000","last_undegraded":"2024-03-24T11:53:47.332115+0000","last_fullsized":"2024-03-24T11:53:47.332115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.620013+0000","last_change":"2024-03-24T11:53:35.343688+0000","last_active":"2024-03-24T11:53:46.620013+0000","last_peered":"2024-03-24T11:53:46.620013+0000","last_clean":"2024-03-24T11:53:46.620013+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:53:46.620013+0000","last_undegraded":"2024-03-24T11:53:46.620013+0000","last_fullsized":"2024-03-24T11:53:46.620013+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.343594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.343594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.531863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617313+0000","last_change":"2024-03-24T11:52:17.423884+0000","last_active":"2024-03-24T11:53:46.617313+0000","last_peered":"2024-03-24T11:53:46.617313+0000","last_clean":"2024-03-24T11:53:46.617313+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:53:46.617313+0000","last_undegraded":"2024-03-24T11:53:46.617313+0000","last_fullsized":"2024-03-24T11:53:46.617313+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331641+0000","last_change":"2024-03-24T11:52:21.390345+0000","last_active":"2024-03-24T11:53:47.331641+0000","last_peered":"2024-03-24T11:53:47.331641+0000","last_clean":"2024-03-24T11:53:47.331641+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:53:47.331641+0000","last_undegraded":"2024-03-24T11:53:47.331641+0000","last_fullsized":"2024-03-24T11:53:47.331641+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619196+0000","last_change":"2024-03-24T11:53:30.303601+0000","last_active":"2024-03-24T11:53:46.619196+0000","last_peered":"2024-03-24T11:53:46.619196+0000","last_clean":"2024-03-24T11:53:46.619196+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:53:46.619196+0000","last_undegraded":"2024-03-24T11:53:46.619196+0000","last_fullsized":"2024-03-24T11:53:46.619196+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.303505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.303505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.216822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748823+0000","last_change":"2024-03-24T11:52:15.142488+0000","last_active":"2024-03-24T11:53:46.748823+0000","last_peered":"2024-03-24T11:53:46.748823+0000","last_clean":"2024-03-24T11:53:46.748823+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:53:46.748823+0000","last_undegraded":"2024-03-24T11:53:46.748823+0000","last_fullsized":"2024-03-24T11:53:46.748823+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.616853+0000","last_change":"2024-03-24T11:53:39.743799+0000","last_active":"2024-03-24T11:53:46.616853+0000","last_peered":"2024-03-24T11:53:46.616853+0000","last_clean":"2024-03-24T11:53:46.616853+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:53:46.616853+0000","last_undegraded":"2024-03-24T11:53:46.616853+0000","last_fullsized":"2024-03-24T11:53:46.616853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T11:53:39.743685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.743685+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.765998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.616774+0000","last_change":"2024-03-24T11:52:21.381608+0000","last_active":"2024-03-24T11:53:46.616774+0000","last_peered":"2024-03-24T11:53:46.616774+0000","last_clean":"2024-03-24T11:53:46.616774+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:53:46.616774+0000","last_undegraded":"2024-03-24T11:53:46.616774+0000","last_fullsized":"2024-03-24T11:53:46.616774+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:47.376283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618307+0000","last_change":"2024-03-24T11:53:44.948231+0000","last_active":"2024-03-24T11:53:46.618307+0000","last_peered":"2024-03-24T11:53:46.618307+0000","last_clean":"2024-03-24T11:53:46.618307+0000","last_became_active":"2024-03-24T11:52:19.377484+0000","last_became_peered":"2024-03-24T11:52:19.377484+0000","last_unstale":"2024-03-24T11:53:46.618307+0000","last_undegraded":"2024-03-24T11:53:46.618307+0000","last_fullsized":"2024-03-24T11:53:46.618307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:53:44.948146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:13.027078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617651+0000","last_change":"2024-03-24T11:53:32.981301+0000","last_active":"2024-03-24T11:53:46.617651+0000","last_peered":"2024-03-24T11:53:46.617651+0000","last_clean":"2024-03-24T11:53:46.617651+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:53:46.617651+0000","last_undegraded":"2024-03-24T11:53:46.617651+0000","last_fullsized":"2024-03-24T11:53:46.617651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:32.981156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:32.981156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.401524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166570+0000","last_change":"2024-03-24T11:52:17.359275+0000","last_active":"2024-03-24T11:53:47.166570+0000","last_peered":"2024-03-24T11:53:47.166570+0000","last_clean":"2024-03-24T11:53:47.166570+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:53:47.166570+0000","last_undegraded":"2024-03-24T11:53:47.166570+0000","last_fullsized":"2024-03-24T11:53:47.166570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303237+0000","last_change":"2024-03-24T11:53:37.904794+0000","last_active":"2024-03-24T11:53:47.303237+0000","last_peered":"2024-03-24T11:53:47.303237+0000","last_clean":"2024-03-24T11:53:47.303237+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:53:47.303237+0000","last_undegraded":"2024-03-24T11:53:47.303237+0000","last_fullsized":"2024-03-24T11:53:47.303237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.904622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.904622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.565966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617416+0000","last_change":"2024-03-24T11:52:19.378422+0000","last_active":"2024-03-24T11:53:46.617416+0000","last_peered":"2024-03-24T11:53:46.617416+0000","last_clean":"2024-03-24T11:53:46.617416+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:53:46.617416+0000","last_undegraded":"2024-03-24T11:53:46.617416+0000","last_fullsized":"2024-03-24T11:53:46.617416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619437+0000","last_change":"2024-03-24T11:53:43.360101+0000","last_active":"2024-03-24T11:53:46.619437+0000","last_peered":"2024-03-24T11:53:46.619437+0000","last_clean":"2024-03-24T11:53:46.619437+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:53:46.619437+0000","last_undegraded":"2024-03-24T11:53:46.619437+0000","last_fullsized":"2024-03-24T11:53:46.619437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:43.359979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:43.359979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:53.166274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618371+0000","last_change":"2024-03-24T11:53:25.921423+0000","last_active":"2024-03-24T11:53:46.618371+0000","last_peered":"2024-03-24T11:53:46.618371+0000","last_clean":"2024-03-24T11:53:46.618371+0000","last_became_active":"2024-03-24T11:52:17.355956+0000","last_became_peered":"2024-03-24T11:52:17.355956+0000","last_unstale":"2024-03-24T11:53:46.618371+0000","last_undegraded":"2024-03-24T11:53:46.618371+0000","last_fullsized":"2024-03-24T11:53:46.618371+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:25.921326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.921326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:33.718763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.167048+0000","last_change":"2024-03-24T11:53:44.686755+0000","last_active":"2024-03-24T11:53:47.167048+0000","last_peered":"2024-03-24T11:53:47.167048+0000","last_clean":"2024-03-24T11:53:47.167048+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:53:47.167048+0000","last_undegraded":"2024-03-24T11:53:47.167048+0000","last_fullsized":"2024-03-24T11:53:47.167048+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:44.686619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:44.686619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:56.785834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617551+0000","last_change":"2024-03-24T11:53:28.755945+0000","last_active":"2024-03-24T11:53:46.617551+0000","last_peered":"2024-03-24T11:53:46.617551+0000","last_clean":"2024-03-24T11:53:46.617551+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:53:46.617551+0000","last_undegraded":"2024-03-24T11:53:46.617551+0000","last_fullsized":"2024-03-24T11:53:46.617551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:28.755826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:28.755826+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.854085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303678+0000","last_change":"2024-03-24T11:52:15.135724+0000","last_active":"2024-03-24T11:53:47.303678+0000","last_peered":"2024-03-24T11:53:47.303678+0000","last_clean":"2024-03-24T11:53:47.303678+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:53:47.303678+0000","last_undegraded":"2024-03-24T11:53:47.303678+0000","last_fullsized":"2024-03-24T11:53:47.303678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.836732+0000","last_change":"2024-03-24T11:53:47.836766+0000","last_active":"2024-03-24T11:53:47.836732+0000","last_peered":"2024-03-24T11:53:47.836732+0000","last_clean":"2024-03-24T11:53:47.836732+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:53:47.836732+0000","last_undegraded":"2024-03-24T11:53:47.836732+0000","last_fullsized":"2024-03-24T11:53:47.836732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:47.836657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.836657+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.133633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303779+0000","last_change":"2024-03-24T11:52:21.382650+0000","last_active":"2024-03-24T11:53:47.303779+0000","last_peered":"2024-03-24T11:53:47.303779+0000","last_clean":"2024-03-24T11:53:47.303779+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:53:47.303779+0000","last_undegraded":"2024-03-24T11:53:47.303779+0000","last_fullsized":"2024-03-24T11:53:47.303779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748950+0000","last_change":"2024-03-24T11:53:32.812255+0000","last_active":"2024-03-24T11:53:46.748950+0000","last_peered":"2024-03-24T11:53:46.748950+0000","last_clean":"2024-03-24T11:53:46.748950+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:53:46.748950+0000","last_undegraded":"2024-03-24T11:53:46.748950+0000","last_fullsized":"2024-03-24T11:53:46.748950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:32.812047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:32.812047+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:50.393185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748435+0000","last_change":"2024-03-24T11:52:15.138567+0000","last_active":"2024-03-24T11:53:46.748435+0000","last_peered":"2024-03-24T11:53:46.748435+0000","last_clean":"2024-03-24T11:53:46.748435+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:53:46.748435+0000","last_undegraded":"2024-03-24T11:53:46.748435+0000","last_fullsized":"2024-03-24T11:53:46.748435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.583092+0000","last_change":"2024-03-24T11:53:47.583124+0000","last_active":"2024-03-24T11:53:47.583092+0000","last_peered":"2024-03-24T11:53:47.583092+0000","last_clean":"2024-03-24T11:53:47.583092+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:53:47.583092+0000","last_undegraded":"2024-03-24T11:53:47.583092+0000","last_fullsized":"2024-03-24T11:53:47.583092+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.583025+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.607356+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331963+0000","last_change":"2024-03-24T11:53:42.480919+0000","last_active":"2024-03-24T11:53:47.331963+0000","last_peered":"2024-03-24T11:53:47.331963+0000","last_clean":"2024-03-24T11:53:47.331963+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:53:47.331963+0000","last_undegraded":"2024-03-24T11:53:47.331963+0000","last_fullsized":"2024-03-24T11:53:47.331963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:42.480832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:42.480832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.968593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748758+0000","last_change":"2024-03-24T11:52:19.379744+0000","last_active":"2024-03-24T11:53:46.748758+0000","last_peered":"2024-03-24T11:53:46.748758+0000","last_clean":"2024-03-24T11:53:46.748758+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:53:46.748758+0000","last_undegraded":"2024-03-24T11:53:46.748758+0000","last_fullsized":"2024-03-24T11:53:46.748758+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619228+0000","last_change":"2024-03-24T11:53:21.338998+0000","last_active":"2024-03-24T11:53:46.619228+0000","last_peered":"2024-03-24T11:53:46.619228+0000","last_clean":"2024-03-24T11:53:46.619228+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:53:46.619228+0000","last_undegraded":"2024-03-24T11:53:46.619228+0000","last_fullsized":"2024-03-24T11:53:46.619228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:21.338907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:21.338907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:27.393056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618226+0000","last_change":"2024-03-24T11:53:22.834856+0000","last_active":"2024-03-24T11:53:46.618226+0000","last_peered":"2024-03-24T11:53:46.618226+0000","last_clean":"2024-03-24T11:53:46.618226+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:53:46.618226+0000","last_undegraded":"2024-03-24T11:53:46.618226+0000","last_fullsized":"2024-03-24T11:53:46.618226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:53:22.834770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:22.834770+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:35.505245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499383+0000","last_change":"2024-03-24T11:52:21.386021+0000","last_active":"2024-03-24T11:53:47.499383+0000","last_peered":"2024-03-24T11:53:47.499383+0000","last_clean":"2024-03-24T11:53:47.499383+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:53:47.499383+0000","last_undegraded":"2024-03-24T11:53:47.499383+0000","last_fullsized":"2024-03-24T11:53:47.499383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618651+0000","last_change":"2024-03-24T11:52:19.376888+0000","last_active":"2024-03-24T11:53:46.618651+0000","last_peered":"2024-03-24T11:53:46.618651+0000","last_clean":"2024-03-24T11:53:46.618651+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:53:46.618651+0000","last_undegraded":"2024-03-24T11:53:46.618651+0000","last_fullsized":"2024-03-24T11:53:46.618651+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499326+0000","last_change":"2024-03-24T11:53:38.430978+0000","last_active":"2024-03-24T11:53:47.499326+0000","last_peered":"2024-03-24T11:53:47.499326+0000","last_clean":"2024-03-24T11:53:47.499326+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:53:47.499326+0000","last_undegraded":"2024-03-24T11:53:47.499326+0000","last_fullsized":"2024-03-24T11:53:47.499326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:38.430896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:38.430896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:46.558321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303898+0000","last_change":"2024-03-24T11:53:38.899939+0000","last_active":"2024-03-24T11:53:47.303898+0000","last_peered":"2024-03-24T11:53:47.303898+0000","last_clean":"2024-03-24T11:53:47.303898+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:53:47.303898+0000","last_undegraded":"2024-03-24T11:53:47.303898+0000","last_fullsized":"2024-03-24T11:53:47.303898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:38.899823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:38.899823+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.812037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617503+0000","last_change":"2024-03-24T11:52:21.389019+0000","last_active":"2024-03-24T11:53:46.617503+0000","last_peered":"2024-03-24T11:53:46.617503+0000","last_clean":"2024-03-24T11:53:46.617503+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:53:46.617503+0000","last_undegraded":"2024-03-24T11:53:46.617503+0000","last_fullsized":"2024-03-24T11:53:46.617503+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166888+0000","last_change":"2024-03-24T11:53:35.720564+0000","last_active":"2024-03-24T11:53:47.166888+0000","last_peered":"2024-03-24T11:53:47.166888+0000","last_clean":"2024-03-24T11:53:47.166888+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:53:47.166888+0000","last_undegraded":"2024-03-24T11:53:47.166888+0000","last_fullsized":"2024-03-24T11:53:47.166888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:53:35.720425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.720425+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.137342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":108,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166371+0000","last_change":"2024-03-24T11:53:22.656282+0000","last_active":"2024-03-24T11:53:47.166371+0000","last_peered":"2024-03-24T11:53:47.166371+0000","last_clean":"2024-03-24T11:53:47.166371+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:53:47.166371+0000","last_undegraded":"2024-03-24T11:53:47.166371+0000","last_fullsized":"2024-03-24T11:53:47.166371+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:53:22.656153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:22.656153+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:23.354452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:50.425097+0000","last_change":"2024-03-24T11:53:50.425119+0000","last_active":"2024-03-24T11:53:50.425097+0000","last_peered":"2024-03-24T11:53:50.425097+0000","last_clean":"2024-03-24T11:53:50.425097+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:53:50.425097+0000","last_undegraded":"2024-03-24T11:53:50.425097+0000","last_fullsized":"2024-03-24T11:53:50.425097+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:50.425035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:50.425035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:06.064329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303241+0000","last_change":"2024-03-24T11:52:21.386419+0000","last_active":"2024-03-24T11:53:47.303241+0000","last_peered":"2024-03-24T11:53:47.303241+0000","last_clean":"2024-03-24T11:53:47.303241+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:53:47.303241+0000","last_undegraded":"2024-03-24T11:53:47.303241+0000","last_fullsized":"2024-03-24T11:53:47.303241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:50.950640+0000","last_change":"2024-03-24T11:53:50.950662+0000","last_active":"2024-03-24T11:53:50.950640+0000","last_peered":"2024-03-24T11:53:50.950640+0000","last_clean":"2024-03-24T11:53:50.950640+0000","last_became_active":"2024-03-24T11:52:15.138792+0000","last_became_peered":"2024-03-24T11:52:15.138792+0000","last_unstale":"2024-03-24T11:53:50.950640+0000","last_undegraded":"2024-03-24T11:53:50.950640+0000","last_fullsized":"2024-03-24T11:53:50.950640+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:53:50.950565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.098973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618201+0000","last_change":"2024-03-24T11:53:26.940720+0000","last_active":"2024-03-24T11:53:46.618201+0000","last_peered":"2024-03-24T11:53:46.618201+0000","last_clean":"2024-03-24T11:53:46.618201+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:53:46.618201+0000","last_undegraded":"2024-03-24T11:53:46.618201+0000","last_fullsized":"2024-03-24T11:53:46.618201+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:53:26.940542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:26.940542+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.977884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303852+0000","last_change":"2024-03-24T11:52:19.378322+0000","last_active":"2024-03-24T11:53:47.303852+0000","last_peered":"2024-03-24T11:53:47.303852+0000","last_clean":"2024-03-24T11:53:47.303852+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:53:47.303852+0000","last_undegraded":"2024-03-24T11:53:47.303852+0000","last_fullsized":"2024-03-24T11:53:47.303852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619305+0000","last_change":"2024-03-24T11:53:42.665773+0000","last_active":"2024-03-24T11:53:46.619305+0000","last_peered":"2024-03-24T11:53:46.619305+0000","last_clean":"2024-03-24T11:53:46.619305+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:53:46.619305+0000","last_undegraded":"2024-03-24T11:53:46.619305+0000","last_fullsized":"2024-03-24T11:53:46.619305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:42.665691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:42.665691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:03.119522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748529+0000","last_change":"2024-03-24T11:53:36.855528+0000","last_active":"2024-03-24T11:53:46.748529+0000","last_peered":"2024-03-24T11:53:46.748529+0000","last_clean":"2024-03-24T11:53:46.748529+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:53:46.748529+0000","last_undegraded":"2024-03-24T11:53:46.748529+0000","last_fullsized":"2024-03-24T11:53:46.748529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:36.855425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.855425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:37.779842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748779+0000","last_change":"2024-03-24T11:52:37.802552+0000","last_active":"2024-03-24T11:53:46.748779+0000","last_peered":"2024-03-24T11:53:46.748779+0000","last_clean":"2024-03-24T11:53:46.748779+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:53:46.748779+0000","last_undegraded":"2024-03-24T11:53:46.748779+0000","last_fullsized":"2024-03-24T11:53:46.748779+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.304055+0000","last_change":"2024-03-24T11:53:45.014753+0000","last_active":"2024-03-24T11:53:47.304055+0000","last_peered":"2024-03-24T11:53:47.304055+0000","last_clean":"2024-03-24T11:53:47.304055+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:53:47.304055+0000","last_undegraded":"2024-03-24T11:53:47.304055+0000","last_fullsized":"2024-03-24T11:53:47.304055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.014670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.014670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.454483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332229+0000","last_change":"2024-03-24T11:53:34.632333+0000","last_active":"2024-03-24T11:53:47.332229+0000","last_peered":"2024-03-24T11:53:47.332229+0000","last_clean":"2024-03-24T11:53:47.332229+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:53:47.332229+0000","last_undegraded":"2024-03-24T11:53:47.332229+0000","last_fullsized":"2024-03-24T11:53:47.332229+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:34.632228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:34.632228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.019688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":172,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:49.833771+0000","last_change":"2024-03-24T11:53:39.041508+0000","last_active":"2024-03-24T11:53:49.833771+0000","last_peered":"2024-03-24T11:53:49.833771+0000","last_clean":"2024-03-24T11:53:49.833771+0000","last_became_active":"2024-03-24T11:52:15.144954+0000","last_became_peered":"2024-03-24T11:52:15.144954+0000","last_unstale":"2024-03-24T11:53:49.833771+0000","last_undegraded":"2024-03-24T11:53:49.833771+0000","last_fullsized":"2024-03-24T11:53:49.833771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:53:39.041412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.041412+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:06.745360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499159+0000","last_change":"2024-03-24T11:53:29.429575+0000","last_active":"2024-03-24T11:53:47.499159+0000","last_peered":"2024-03-24T11:53:47.499159+0000","last_clean":"2024-03-24T11:53:47.499159+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:53:47.499159+0000","last_undegraded":"2024-03-24T11:53:47.499159+0000","last_fullsized":"2024-03-24T11:53:47.499159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.429483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.429483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.616922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618111+0000","last_change":"2024-03-24T11:52:17.358816+0000","last_active":"2024-03-24T11:53:46.618111+0000","last_peered":"2024-03-24T11:53:46.618111+0000","last_clean":"2024-03-24T11:53:46.618111+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:53:46.618111+0000","last_undegraded":"2024-03-24T11:53:46.618111+0000","last_fullsized":"2024-03-24T11:53:46.618111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499049+0000","last_change":"2024-03-24T11:52:19.383334+0000","last_active":"2024-03-24T11:53:47.499049+0000","last_peered":"2024-03-24T11:53:47.499049+0000","last_clean":"2024-03-24T11:53:47.499049+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:53:47.499049+0000","last_undegraded":"2024-03-24T11:53:47.499049+0000","last_fullsized":"2024-03-24T11:53:47.499049+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.749021+0000","last_change":"2024-03-24T11:53:41.822978+0000","last_active":"2024-03-24T11:53:46.749021+0000","last_peered":"2024-03-24T11:53:46.749021+0000","last_clean":"2024-03-24T11:53:46.749021+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:53:46.749021+0000","last_undegraded":"2024-03-24T11:53:46.749021+0000","last_fullsized":"2024-03-24T11:53:46.749021+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:53:41.822890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.481555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617065+0000","last_change":"2024-03-24T11:52:19.368496+0000","last_active":"2024-03-24T11:53:46.617065+0000","last_peered":"2024-03-24T11:53:46.617065+0000","last_clean":"2024-03-24T11:53:46.617065+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:53:46.617065+0000","last_undegraded":"2024-03-24T11:53:46.617065+0000","last_fullsized":"2024-03-24T11:53:46.617065+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:50.814817+0000","last_change":"2024-03-24T11:53:50.814854+0000","last_active":"2024-03-24T11:53:50.814817+0000","last_peered":"2024-03-24T11:53:50.814817+0000","last_clean":"2024-03-24T11:53:50.814817+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:53:50.814817+0000","last_undegraded":"2024-03-24T11:53:50.814817+0000","last_fullsized":"2024-03-24T11:53:50.814817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:50.814720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:50.814720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:05.078424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":319,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331593+0000","last_change":"2024-03-24T11:52:19.498367+0000","last_active":"2024-03-24T11:53:47.331593+0000","last_peered":"2024-03-24T11:53:47.331593+0000","last_clean":"2024-03-24T11:53:47.331593+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:53:47.331593+0000","last_undegraded":"2024-03-24T11:53:47.331593+0000","last_fullsized":"2024-03-24T11:53:47.331593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'119","last_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_deep_scrub":"71'119","last_deep_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_clean_scrub_stamp":"2024-03-24T11:52:19.498276+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619584+0000","last_change":"2024-03-24T11:53:41.364342+0000","last_active":"2024-03-24T11:53:46.619584+0000","last_peered":"2024-03-24T11:53:46.619584+0000","last_clean":"2024-03-24T11:53:46.619584+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:53:46.619584+0000","last_undegraded":"2024-03-24T11:53:46.619584+0000","last_fullsized":"2024-03-24T11:53:46.619584+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:53:41.364225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:41.364225+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.172024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332134+0000","last_change":"2024-03-24T11:53:29.691119+0000","last_active":"2024-03-24T11:53:47.332134+0000","last_peered":"2024-03-24T11:53:47.332134+0000","last_clean":"2024-03-24T11:53:47.332134+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:53:47.332134+0000","last_undegraded":"2024-03-24T11:53:47.332134+0000","last_fullsized":"2024-03-24T11:53:47.332134+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.691006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.691006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:54.100657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.960225+0000","last_change":"2024-03-24T11:53:47.960246+0000","last_active":"2024-03-24T11:53:47.960225+0000","last_peered":"2024-03-24T11:53:47.960225+0000","last_clean":"2024-03-24T11:53:47.960225+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:53:47.960225+0000","last_undegraded":"2024-03-24T11:53:47.960225+0000","last_fullsized":"2024-03-24T11:53:47.960225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:47.960166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.960166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.716011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303538+0000","last_change":"2024-03-24T11:52:19.366588+0000","last_active":"2024-03-24T11:53:47.303538+0000","last_peered":"2024-03-24T11:53:47.303538+0000","last_clean":"2024-03-24T11:53:47.303538+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:53:47.303538+0000","last_undegraded":"2024-03-24T11:53:47.303538+0000","last_fullsized":"2024-03-24T11:53:47.303538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331521+0000","last_change":"2024-03-24T11:53:24.679709+0000","last_active":"2024-03-24T11:53:47.331521+0000","last_peered":"2024-03-24T11:53:47.331521+0000","last_clean":"2024-03-24T11:53:47.331521+0000","last_became_active":"2024-03-24T11:52:15.146310+0000","last_became_peered":"2024-03-24T11:52:15.146310+0000","last_unstale":"2024-03-24T11:53:47.331521+0000","last_undegraded":"2024-03-24T11:53:47.331521+0000","last_fullsized":"2024-03-24T11:53:47.331521+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:24.679579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.679579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:29.622468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303870+0000","last_change":"2024-03-24T11:53:24.917322+0000","last_active":"2024-03-24T11:53:47.303870+0000","last_peered":"2024-03-24T11:53:47.303870+0000","last_clean":"2024-03-24T11:53:47.303870+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:53:47.303870+0000","last_undegraded":"2024-03-24T11:53:47.303870+0000","last_fullsized":"2024-03-24T11:53:47.303870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:53:24.917220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.917220+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.119999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748949+0000","last_change":"2024-03-24T11:52:19.387410+0000","last_active":"2024-03-24T11:53:46.748949+0000","last_peered":"2024-03-24T11:53:46.748949+0000","last_clean":"2024-03-24T11:53:46.748949+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:53:46.748949+0000","last_undegraded":"2024-03-24T11:53:46.748949+0000","last_fullsized":"2024-03-24T11:53:46.748949+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617788+0000","last_change":"2024-03-24T11:53:41.036691+0000","last_active":"2024-03-24T11:53:46.617788+0000","last_peered":"2024-03-24T11:53:46.617788+0000","last_clean":"2024-03-24T11:53:46.617788+0000","last_became_active":"2024-03-24T11:52:21.851069+0000","last_became_peered":"2024-03-24T11:52:21.851069+0000","last_unstale":"2024-03-24T11:53:46.617788+0000","last_undegraded":"2024-03-24T11:53:46.617788+0000","last_fullsized":"2024-03-24T11:53:46.617788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:41.036572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:41.036572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:42.293567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332329+0000","last_change":"2024-03-24T11:53:18.576655+0000","last_active":"2024-03-24T11:53:47.332329+0000","last_peered":"2024-03-24T11:53:47.332329+0000","last_clean":"2024-03-24T11:53:47.332329+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:53:47.332329+0000","last_undegraded":"2024-03-24T11:53:47.332329+0000","last_fullsized":"2024-03-24T11:53:47.332329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:18.576538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:18.576538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.141440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":104,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303673+0000","last_change":"2024-03-24T11:53:39.923840+0000","last_active":"2024-03-24T11:53:47.303673+0000","last_peered":"2024-03-24T11:53:47.303673+0000","last_clean":"2024-03-24T11:53:47.303673+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:53:47.303673+0000","last_undegraded":"2024-03-24T11:53:47.303673+0000","last_fullsized":"2024-03-24T11:53:47.303673+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:53:39.923699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.923699+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.413674+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:50.319937+0000","last_change":"2024-03-24T11:53:50.319962+0000","last_active":"2024-03-24T11:53:50.319937+0000","last_peered":"2024-03-24T11:53:50.319937+0000","last_clean":"2024-03-24T11:53:50.319937+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:53:50.319937+0000","last_undegraded":"2024-03-24T11:53:50.319937+0000","last_fullsized":"2024-03-24T11:53:50.319937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T11:53:50.319837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.251719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166668+0000","last_change":"2024-03-24T11:53:39.653196+0000","last_active":"2024-03-24T11:53:47.166668+0000","last_peered":"2024-03-24T11:53:47.166668+0000","last_clean":"2024-03-24T11:53:47.166668+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:53:47.166668+0000","last_undegraded":"2024-03-24T11:53:47.166668+0000","last_fullsized":"2024-03-24T11:53:47.166668+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:39.653110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.653110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.778426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617540+0000","last_change":"2024-03-24T11:53:40.019269+0000","last_active":"2024-03-24T11:53:46.617540+0000","last_peered":"2024-03-24T11:53:46.617540+0000","last_clean":"2024-03-24T11:53:46.617540+0000","last_became_active":"2024-03-24T11:52:15.142974+0000","last_became_peered":"2024-03-24T11:52:15.142974+0000","last_unstale":"2024-03-24T11:53:46.617540+0000","last_undegraded":"2024-03-24T11:53:46.617540+0000","last_fullsized":"2024-03-24T11:53:46.617540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:40.019165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:40.019165+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.538489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.616730+0000","last_change":"2024-03-24T11:53:35.726728+0000","last_active":"2024-03-24T11:53:46.616730+0000","last_peered":"2024-03-24T11:53:46.616730+0000","last_clean":"2024-03-24T11:53:46.616730+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:53:46.616730+0000","last_undegraded":"2024-03-24T11:53:46.616730+0000","last_fullsized":"2024-03-24T11:53:46.616730+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:53:35.726630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.726630+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:37.790720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166376+0000","last_change":"2024-03-24T11:52:19.389693+0000","last_active":"2024-03-24T11:53:47.166376+0000","last_peered":"2024-03-24T11:53:47.166376+0000","last_clean":"2024-03-24T11:53:47.166376+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:53:47.166376+0000","last_undegraded":"2024-03-24T11:53:47.166376+0000","last_fullsized":"2024-03-24T11:53:47.166376+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332057+0000","last_change":"2024-03-24T11:52:21.854938+0000","last_active":"2024-03-24T11:53:47.332057+0000","last_peered":"2024-03-24T11:53:47.332057+0000","last_clean":"2024-03-24T11:53:47.332057+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:53:47.332057+0000","last_undegraded":"2024-03-24T11:53:47.332057+0000","last_fullsized":"2024-03-24T11:53:47.332057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619904+0000","last_change":"2024-03-24T11:53:15.292341+0000","last_active":"2024-03-24T11:53:46.619904+0000","last_peered":"2024-03-24T11:53:46.619904+0000","last_clean":"2024-03-24T11:53:46.619904+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:53:46.619904+0000","last_undegraded":"2024-03-24T11:53:46.619904+0000","last_fullsized":"2024-03-24T11:53:46.619904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:53:15.292233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.790414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":95,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166236+0000","last_change":"2024-03-24T11:53:26.665675+0000","last_active":"2024-03-24T11:53:47.166236+0000","last_peered":"2024-03-24T11:53:47.166236+0000","last_clean":"2024-03-24T11:53:47.166236+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:53:47.166236+0000","last_undegraded":"2024-03-24T11:53:47.166236+0000","last_fullsized":"2024-03-24T11:53:47.166236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:53:26.665554+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:48.241747+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.616713+0000","last_change":"2024-03-24T11:52:19.382757+0000","last_active":"2024-03-24T11:53:46.616713+0000","last_peered":"2024-03-24T11:53:46.616713+0000","last_clean":"2024-03-24T11:53:46.616713+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:53:46.616713+0000","last_undegraded":"2024-03-24T11:53:46.616713+0000","last_fullsized":"2024-03-24T11:53:46.616713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617009+0000","last_change":"2024-03-24T11:52:21.851542+0000","last_active":"2024-03-24T11:53:46.617009+0000","last_peered":"2024-03-24T11:53:46.617009+0000","last_clean":"2024-03-24T11:53:46.617009+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:53:46.617009+0000","last_undegraded":"2024-03-24T11:53:46.617009+0000","last_fullsized":"2024-03-24T11:53:46.617009+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748923+0000","last_change":"2024-03-24T11:52:19.388558+0000","last_active":"2024-03-24T11:53:46.748923+0000","last_peered":"2024-03-24T11:53:46.748923+0000","last_clean":"2024-03-24T11:53:46.748923+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:53:46.748923+0000","last_undegraded":"2024-03-24T11:53:46.748923+0000","last_fullsized":"2024-03-24T11:53:46.748923+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331638+0000","last_change":"2024-03-24T11:53:30.710184+0000","last_active":"2024-03-24T11:53:47.331638+0000","last_peered":"2024-03-24T11:53:47.331638+0000","last_clean":"2024-03-24T11:53:47.331638+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:53:47.331638+0000","last_undegraded":"2024-03-24T11:53:47.331638+0000","last_fullsized":"2024-03-24T11:53:47.331638+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.710079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.710079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.148875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166948+0000","last_change":"2024-03-24T11:52:17.351215+0000","last_active":"2024-03-24T11:53:47.166948+0000","last_peered":"2024-03-24T11:53:47.166948+0000","last_clean":"2024-03-24T11:53:47.166948+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:53:47.166948+0000","last_undegraded":"2024-03-24T11:53:47.166948+0000","last_fullsized":"2024-03-24T11:53:47.166948+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:49.795398+0000","last_change":"2024-03-24T11:53:49.795423+0000","last_active":"2024-03-24T11:53:49.795398+0000","last_peered":"2024-03-24T11:53:49.795398+0000","last_clean":"2024-03-24T11:53:49.795398+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:53:49.795398+0000","last_undegraded":"2024-03-24T11:53:49.795398+0000","last_fullsized":"2024-03-24T11:53:49.795398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:49.795328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:49.795328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.881169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166762+0000","last_change":"2024-03-24T11:53:42.651953+0000","last_active":"2024-03-24T11:53:47.166762+0000","last_peered":"2024-03-24T11:53:47.166762+0000","last_clean":"2024-03-24T11:53:47.166762+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:53:47.166762+0000","last_undegraded":"2024-03-24T11:53:47.166762+0000","last_fullsized":"2024-03-24T11:53:47.166762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:42.651866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:42.651866+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.201095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617902+0000","last_change":"2024-03-24T11:53:45.930437+0000","last_active":"2024-03-24T11:53:46.617902+0000","last_peered":"2024-03-24T11:53:46.617902+0000","last_clean":"2024-03-24T11:53:46.617902+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:53:46.617902+0000","last_undegraded":"2024-03-24T11:53:46.617902+0000","last_fullsized":"2024-03-24T11:53:46.617902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.930342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.930342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:15.517071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617650+0000","last_change":"2024-03-24T11:53:31.957804+0000","last_active":"2024-03-24T11:53:46.617650+0000","last_peered":"2024-03-24T11:53:46.617650+0000","last_clean":"2024-03-24T11:53:46.617650+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:53:46.617650+0000","last_undegraded":"2024-03-24T11:53:46.617650+0000","last_fullsized":"2024-03-24T11:53:46.617650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:31.957706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.957706+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.121723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166721+0000","last_change":"2024-03-24T11:53:31.724380+0000","last_active":"2024-03-24T11:53:47.166721+0000","last_peered":"2024-03-24T11:53:47.166721+0000","last_clean":"2024-03-24T11:53:47.166721+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:53:47.166721+0000","last_undegraded":"2024-03-24T11:53:47.166721+0000","last_fullsized":"2024-03-24T11:53:47.166721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:53:31.724235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.724235+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.862511+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.498982+0000","last_change":"2024-03-24T11:53:19.389995+0000","last_active":"2024-03-24T11:53:47.498982+0000","last_peered":"2024-03-24T11:53:47.498982+0000","last_clean":"2024-03-24T11:53:47.498982+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:53:47.498982+0000","last_undegraded":"2024-03-24T11:53:47.498982+0000","last_fullsized":"2024-03-24T11:53:47.498982+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:19.389895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:19.389895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:38.977195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618068+0000","last_change":"2024-03-24T11:52:15.134671+0000","last_active":"2024-03-24T11:53:46.618068+0000","last_peered":"2024-03-24T11:53:46.618068+0000","last_clean":"2024-03-24T11:53:46.618068+0000","last_became_active":"2024-03-24T11:52:15.134326+0000","last_became_peered":"2024-03-24T11:52:15.134326+0000","last_unstale":"2024-03-24T11:53:46.618068+0000","last_undegraded":"2024-03-24T11:53:46.618068+0000","last_fullsized":"2024-03-24T11:53:46.618068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166455+0000","last_change":"2024-03-24T11:53:34.756186+0000","last_active":"2024-03-24T11:53:47.166455+0000","last_peered":"2024-03-24T11:53:47.166455+0000","last_clean":"2024-03-24T11:53:47.166455+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:53:47.166455+0000","last_undegraded":"2024-03-24T11:53:47.166455+0000","last_fullsized":"2024-03-24T11:53:47.166455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:53:34.756054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:34.756054+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.815024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499690+0000","last_change":"2024-03-24T11:52:21.388549+0000","last_active":"2024-03-24T11:53:47.499690+0000","last_peered":"2024-03-24T11:53:47.499690+0000","last_clean":"2024-03-24T11:53:47.499690+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:53:47.499690+0000","last_undegraded":"2024-03-24T11:53:47.499690+0000","last_fullsized":"2024-03-24T11:53:47.499690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618570+0000","last_change":"2024-03-24T11:52:19.387447+0000","last_active":"2024-03-24T11:53:46.618570+0000","last_peered":"2024-03-24T11:53:46.618570+0000","last_clean":"2024-03-24T11:53:46.618570+0000","last_became_active":"2024-03-24T11:52:19.387115+0000","last_became_peered":"2024-03-24T11:52:19.387115+0000","last_unstale":"2024-03-24T11:53:46.618570+0000","last_undegraded":"2024-03-24T11:53:46.618570+0000","last_fullsized":"2024-03-24T11:53:46.618570+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499368+0000","last_change":"2024-03-24T11:53:24.386409+0000","last_active":"2024-03-24T11:53:47.499368+0000","last_peered":"2024-03-24T11:53:47.499368+0000","last_clean":"2024-03-24T11:53:47.499368+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:53:47.499368+0000","last_undegraded":"2024-03-24T11:53:47.499368+0000","last_fullsized":"2024-03-24T11:53:47.499368+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:24.386208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.386208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:43.956194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":95,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166803+0000","last_change":"2024-03-24T11:53:43.683275+0000","last_active":"2024-03-24T11:53:47.166803+0000","last_peered":"2024-03-24T11:53:47.166803+0000","last_clean":"2024-03-24T11:53:47.166803+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:53:47.166803+0000","last_undegraded":"2024-03-24T11:53:47.166803+0000","last_fullsized":"2024-03-24T11:53:47.166803+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-24T11:53:43.683151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:43.683151+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.427314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:49.544662+0000","last_change":"2024-03-24T11:53:49.544698+0000","last_active":"2024-03-24T11:53:49.544662+0000","last_peered":"2024-03-24T11:53:49.544662+0000","last_clean":"2024-03-24T11:53:49.544662+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:53:49.544662+0000","last_undegraded":"2024-03-24T11:53:49.544662+0000","last_fullsized":"2024-03-24T11:53:49.544662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:49.544578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:49.544578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:03.719476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.498892+0000","last_change":"2024-03-24T11:53:37.463221+0000","last_active":"2024-03-24T11:53:47.498892+0000","last_peered":"2024-03-24T11:53:47.498892+0000","last_clean":"2024-03-24T11:53:47.498892+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:53:47.498892+0000","last_undegraded":"2024-03-24T11:53:47.498892+0000","last_fullsized":"2024-03-24T11:53:47.498892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.463110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.463110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.363432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499471+0000","last_change":"2024-03-24T11:53:30.449547+0000","last_active":"2024-03-24T11:53:47.499471+0000","last_peered":"2024-03-24T11:53:47.499471+0000","last_clean":"2024-03-24T11:53:47.499471+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:53:47.499471+0000","last_undegraded":"2024-03-24T11:53:47.499471+0000","last_fullsized":"2024-03-24T11:53:47.499471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.449442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.449442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:36.454836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":98,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619765+0000","last_change":"2024-03-24T11:53:45.332408+0000","last_active":"2024-03-24T11:53:46.619765+0000","last_peered":"2024-03-24T11:53:46.619765+0000","last_clean":"2024-03-24T11:53:46.619765+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:53:46.619765+0000","last_undegraded":"2024-03-24T11:53:46.619765+0000","last_fullsized":"2024-03-24T11:53:46.619765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:53:45.332300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.332300+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.325555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303326+0000","last_change":"2024-03-24T11:52:21.387355+0000","last_active":"2024-03-24T11:53:47.303326+0000","last_peered":"2024-03-24T11:53:47.303326+0000","last_clean":"2024-03-24T11:53:47.303326+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:53:47.303326+0000","last_undegraded":"2024-03-24T11:53:47.303326+0000","last_fullsized":"2024-03-24T11:53:47.303326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.688825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.498874+0000","last_change":"2024-03-24T11:52:19.384147+0000","last_active":"2024-03-24T11:53:47.498874+0000","last_peered":"2024-03-24T11:53:47.498874+0000","last_clean":"2024-03-24T11:53:47.498874+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:53:47.498874+0000","last_undegraded":"2024-03-24T11:53:47.498874+0000","last_fullsized":"2024-03-24T11:53:47.498874+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617828+0000","last_change":"2024-03-24T11:53:37.995353+0000","last_active":"2024-03-24T11:53:46.617828+0000","last_peered":"2024-03-24T11:53:46.617828+0000","last_clean":"2024-03-24T11:53:46.617828+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:53:46.617828+0000","last_undegraded":"2024-03-24T11:53:46.617828+0000","last_fullsized":"2024-03-24T11:53:46.617828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.995261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:56.380355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303797+0000","last_change":"2024-03-24T11:53:36.884179+0000","last_active":"2024-03-24T11:53:47.303797+0000","last_peered":"2024-03-24T11:53:47.303797+0000","last_clean":"2024-03-24T11:53:47.303797+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:53:47.303797+0000","last_undegraded":"2024-03-24T11:53:47.303797+0000","last_fullsized":"2024-03-24T11:53:47.303797+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:36.884086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.884086+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.784016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748400+0000","last_change":"2024-03-24T11:53:45.874634+0000","last_active":"2024-03-24T11:53:46.748400+0000","last_peered":"2024-03-24T11:53:46.748400+0000","last_clean":"2024-03-24T11:53:46.748400+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:53:46.748400+0000","last_undegraded":"2024-03-24T11:53:46.748400+0000","last_fullsized":"2024-03-24T11:53:46.748400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.874555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.944511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499666+0000","last_change":"2024-03-24T11:53:25.429178+0000","last_active":"2024-03-24T11:53:47.499666+0000","last_peered":"2024-03-24T11:53:47.499666+0000","last_clean":"2024-03-24T11:53:47.499666+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:53:47.499666+0000","last_undegraded":"2024-03-24T11:53:47.499666+0000","last_fullsized":"2024-03-24T11:53:47.499666+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:25.429098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.429098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:46.655256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748336+0000","last_change":"2024-03-24T11:53:37.846040+0000","last_active":"2024-03-24T11:53:46.748336+0000","last_peered":"2024-03-24T11:53:46.748336+0000","last_clean":"2024-03-24T11:53:46.748336+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:53:46.748336+0000","last_undegraded":"2024-03-24T11:53:46.748336+0000","last_fullsized":"2024-03-24T11:53:46.748336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:37.845952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.845952+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.539200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331869+0000","last_change":"2024-03-24T11:53:31.671828+0000","last_active":"2024-03-24T11:53:47.331869+0000","last_peered":"2024-03-24T11:53:47.331869+0000","last_clean":"2024-03-24T11:53:47.331869+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:53:47.331869+0000","last_undegraded":"2024-03-24T11:53:47.331869+0000","last_fullsized":"2024-03-24T11:53:47.331869+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:31.671692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.671692+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:49.989770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418,"ondisk_log_size":418,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005079,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":969680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332126,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712572,"statfs":{"total":95995035648,"available":95961673728,"internally_reserved":0,"allocated":1433600,"data_stored":969498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":378709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32580,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712572,"statfs":{"total":95995035648,"available":95961673728,"internally_reserved":0,"allocated":1437696,"data_stored":968547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":847872,"data_stored":379129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":379559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920807,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":856064,"data_stored":378744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659170,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":847872,"data_stored":378856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:53:52.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:53:52.898 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:53.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:53.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T11:53:53.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: Deploying daemon node-exporter.a on smithi064 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: from='client.14778 v1:172.21.15.64:0/4228564123' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: 5.1b scrub starts 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: 5.1b scrub ok 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: 2.a deep-scrub starts 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[30223]: 2.a deep-scrub ok 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: Deploying daemon node-exporter.a on smithi064 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: from='client.14778 v1:172.21.15.64:0/4228564123' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: 5.1b scrub starts 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: 5.1b scrub ok 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: 2.a deep-scrub starts 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:52 smithi064 ceph-mon[42243]: 2.a deep-scrub ok 2024-03-24T11:53:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T11:53:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T11:53:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: Deploying daemon node-exporter.a on smithi064 2024-03-24T11:53:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T11:53:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T11:53:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: from='client.14778 v1:172.21.15.64:0/4228564123' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: 5.1b scrub starts 2024-03-24T11:53:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: 5.1b scrub ok 2024-03-24T11:53:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: 2.a deep-scrub starts 2024-03-24T11:53:53.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-mon[33593]: 2.a deep-scrub ok 2024-03-24T11:53:53.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=infra.usagestats.collector t=2024-03-24T11:53:52.85995059Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-24T11:53:53.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=provisioning.datasources t=2024-03-24T11:53:52.860999624Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T11:53:53.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=provisioning.datasources t=2024-03-24T11:53:52.871994784Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-24T11:53:53.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=provisioning.alerting t=2024-03-24T11:53:52.882861466Z level=info msg="starting to provision alerting" 2024-03-24T11:53:53.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=provisioning.alerting t=2024-03-24T11:53:52.882880936Z level=info msg="finished to provision alerting" 2024-03-24T11:53:53.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=ngalert.state.manager t=2024-03-24T11:53:52.883016936Z level=info msg="Warming state cache for startup" 2024-03-24T11:53:53.337 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=ngalert.state.manager t=2024-03-24T11:53:52.88314894Z level=info msg="State cache has been initialized" states=0 duration=131.426µs 2024-03-24T11:53:53.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=ticker t=2024-03-24T11:53:52.883214494Z level=info msg=starting first_tick=2024-03-24T11:54:00Z 2024-03-24T11:53:53.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=ngalert.multiorg.alertmanager t=2024-03-24T11:53:52.883235893Z level=info msg="starting MultiOrg Alertmanager" 2024-03-24T11:53:53.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=grafanaStorageLogger t=2024-03-24T11:53:52.883804105Z level=info msg="storage starting" 2024-03-24T11:53:53.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=http.server t=2024-03-24T11:53:52.884796719Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T11:53:53.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=sqlstore.transactions t=2024-03-24T11:53:52.902219135Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:53:53.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=sqlstore.transactions t=2024-03-24T11:53:52.912467748Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:53:53.338 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 11:53:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=sqlstore.transactions t=2024-03-24T11:53:52.923453033Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:53:54.032 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:53 smithi064 ceph-mon[30223]: 5.16 scrub starts 2024-03-24T11:53:54.032 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:53 smithi064 ceph-mon[30223]: 5.16 scrub ok 2024-03-24T11:53:54.032 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:53 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T11:53:54.032 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:53 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T11:53:54.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:53 smithi064 ceph-mon[42243]: 5.16 scrub starts 2024-03-24T11:53:54.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:53 smithi064 ceph-mon[42243]: 5.16 scrub ok 2024-03-24T11:53:54.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:53 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T11:53:54.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:53 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T11:53:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:53 smithi193 ceph-mon[33593]: 5.16 scrub starts 2024-03-24T11:53:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:53 smithi193 ceph-mon[33593]: 5.16 scrub ok 2024-03-24T11:53:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:53 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T11:53:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:53 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T11:53:54.865 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:53:54.867 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:53:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[30223]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T11:53:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T11:53:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:53:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:53:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[42243]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T11:53:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T11:53:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:53:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:54 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:53:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:54 smithi193 ceph-mon[33593]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:54 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T11:53:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:54 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T11:53:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:54 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:53:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:54 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:53:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:55 smithi064 ceph-mon[30223]: 5.10 deep-scrub starts 2024-03-24T11:53:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 5.10 deep-scrub ok 2024-03-24T11:53:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: from='client.34667 v1:172.21.15.64:0/2153507299' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 2.12 deep-scrub starts 2024-03-24T11:53:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 2.12 deep-scrub ok 2024-03-24T11:53:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:53:56.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 3.2 scrub starts 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 3.2 scrub ok 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:55 smithi064 ceph-mon[42243]: 5.10 deep-scrub starts 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 5.10 deep-scrub ok 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: from='client.34667 v1:172.21.15.64:0/2153507299' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 2.12 deep-scrub starts 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 2.12 deep-scrub ok 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 3.2 scrub starts 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 3.2 scrub ok 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: 5.10 deep-scrub starts 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: 5.10 deep-scrub ok 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: from='client.34667 v1:172.21.15.64:0/2153507299' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: 2.12 deep-scrub starts 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: 2.12 deep-scrub ok 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: 3.2 scrub starts 2024-03-24T11:53:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:55 smithi193 ceph-mon[33593]: 3.2 scrub ok 2024-03-24T11:53:56.551 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-03-24T11:53:53.146475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1017,"num_read_kb":863,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":627,"ondisk_log_size":627,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245320,"kb_used_data":8380,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715896,"statfs":{"total":767960285184,"available":767709077504,"internally_reserved":0,"allocated":8581120,"data_stored":4802722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002229"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617008+0000","last_change":"2024-03-24T11:52:21.854369+0000","last_active":"2024-03-24T11:53:46.617008+0000","last_peered":"2024-03-24T11:53:46.617008+0000","last_clean":"2024-03-24T11:53:46.617008+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:53:46.617008+0000","last_undegraded":"2024-03-24T11:53:46.617008+0000","last_fullsized":"2024-03-24T11:53:46.617008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166063+0000","last_change":"2024-03-24T11:53:29.715820+0000","last_active":"2024-03-24T11:53:47.166063+0000","last_peered":"2024-03-24T11:53:47.166063+0000","last_clean":"2024-03-24T11:53:47.166063+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:53:47.166063+0000","last_undegraded":"2024-03-24T11:53:47.166063+0000","last_fullsized":"2024-03-24T11:53:47.166063+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.715710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:29.794324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.304010+0000","last_change":"2024-03-24T11:52:15.137687+0000","last_active":"2024-03-24T11:53:47.304010+0000","last_peered":"2024-03-24T11:53:47.304010+0000","last_clean":"2024-03-24T11:53:47.304010+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:53:47.304010+0000","last_undegraded":"2024-03-24T11:53:47.304010+0000","last_fullsized":"2024-03-24T11:53:47.304010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617878+0000","last_change":"2024-03-24T11:53:43.627728+0000","last_active":"2024-03-24T11:53:46.617878+0000","last_peered":"2024-03-24T11:53:46.617878+0000","last_clean":"2024-03-24T11:53:46.617878+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:53:46.617878+0000","last_undegraded":"2024-03-24T11:53:46.617878+0000","last_fullsized":"2024-03-24T11:53:46.617878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:43.627630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:43.627630+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:10.204254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166243+0000","last_change":"2024-03-24T11:53:30.710255+0000","last_active":"2024-03-24T11:53:47.166243+0000","last_peered":"2024-03-24T11:53:47.166243+0000","last_clean":"2024-03-24T11:53:47.166243+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:53:47.166243+0000","last_undegraded":"2024-03-24T11:53:47.166243+0000","last_fullsized":"2024-03-24T11:53:47.166243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.710143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.710143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.327391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617873+0000","last_change":"2024-03-24T11:52:19.378597+0000","last_active":"2024-03-24T11:53:46.617873+0000","last_peered":"2024-03-24T11:53:46.617873+0000","last_clean":"2024-03-24T11:53:46.617873+0000","last_became_active":"2024-03-24T11:52:19.378282+0000","last_became_peered":"2024-03-24T11:52:19.378282+0000","last_unstale":"2024-03-24T11:53:46.617873+0000","last_undegraded":"2024-03-24T11:53:46.617873+0000","last_fullsized":"2024-03-24T11:53:46.617873+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619200+0000","last_change":"2024-03-24T11:52:15.141672+0000","last_active":"2024-03-24T11:53:46.619200+0000","last_peered":"2024-03-24T11:53:46.619200+0000","last_clean":"2024-03-24T11:53:46.619200+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:53:46.619200+0000","last_undegraded":"2024-03-24T11:53:46.619200+0000","last_fullsized":"2024-03-24T11:53:46.619200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166240+0000","last_change":"2024-03-24T11:52:17.350943+0000","last_active":"2024-03-24T11:53:47.166240+0000","last_peered":"2024-03-24T11:53:47.166240+0000","last_clean":"2024-03-24T11:53:47.166240+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:53:47.166240+0000","last_undegraded":"2024-03-24T11:53:47.166240+0000","last_fullsized":"2024-03-24T11:53:47.166240+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748344+0000","last_change":"2024-03-24T11:53:25.779733+0000","last_active":"2024-03-24T11:53:46.748344+0000","last_peered":"2024-03-24T11:53:46.748344+0000","last_clean":"2024-03-24T11:53:46.748344+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:53:46.748344+0000","last_undegraded":"2024-03-24T11:53:46.748344+0000","last_fullsized":"2024-03-24T11:53:46.748344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.779615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:49.135209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331945+0000","last_change":"2024-03-24T11:53:45.463943+0000","last_active":"2024-03-24T11:53:47.331945+0000","last_peered":"2024-03-24T11:53:47.331945+0000","last_clean":"2024-03-24T11:53:47.331945+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:53:47.331945+0000","last_undegraded":"2024-03-24T11:53:47.331945+0000","last_fullsized":"2024-03-24T11:53:47.331945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.463814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.463814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:10.450722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332054+0000","last_change":"2024-03-24T11:53:35.593649+0000","last_active":"2024-03-24T11:53:47.332054+0000","last_peered":"2024-03-24T11:53:47.332054+0000","last_clean":"2024-03-24T11:53:47.332054+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:53:47.332054+0000","last_undegraded":"2024-03-24T11:53:47.332054+0000","last_fullsized":"2024-03-24T11:53:47.332054+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.593554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.593554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.653079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166350+0000","last_change":"2024-03-24T11:53:20.591717+0000","last_active":"2024-03-24T11:53:47.166350+0000","last_peered":"2024-03-24T11:53:47.166350+0000","last_clean":"2024-03-24T11:53:47.166350+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:53:47.166350+0000","last_undegraded":"2024-03-24T11:53:47.166350+0000","last_fullsized":"2024-03-24T11:53:47.166350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_clean_scrub_stamp":"2024-03-24T11:53:20.591632+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.910645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303458+0000","last_change":"2024-03-24T11:52:21.391890+0000","last_active":"2024-03-24T11:53:47.303458+0000","last_peered":"2024-03-24T11:53:47.303458+0000","last_clean":"2024-03-24T11:53:47.303458+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:53:47.303458+0000","last_undegraded":"2024-03-24T11:53:47.303458+0000","last_fullsized":"2024-03-24T11:53:47.303458+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.108281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748596+0000","last_change":"2024-03-24T11:52:19.381366+0000","last_active":"2024-03-24T11:53:46.748596+0000","last_peered":"2024-03-24T11:53:46.748596+0000","last_clean":"2024-03-24T11:53:46.748596+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:53:46.748596+0000","last_undegraded":"2024-03-24T11:53:46.748596+0000","last_fullsized":"2024-03-24T11:53:46.748596+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.508572+0000","last_change":"2024-03-24T11:53:47.508597+0000","last_active":"2024-03-24T11:53:47.508572+0000","last_peered":"2024-03-24T11:53:47.508572+0000","last_clean":"2024-03-24T11:53:47.508572+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:53:47.508572+0000","last_undegraded":"2024-03-24T11:53:47.508572+0000","last_fullsized":"2024-03-24T11:53:47.508572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:47.508490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.508490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:58.461301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166307+0000","last_change":"2024-03-24T11:53:17.621045+0000","last_active":"2024-03-24T11:53:47.166307+0000","last_peered":"2024-03-24T11:53:47.166307+0000","last_clean":"2024-03-24T11:53:47.166307+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:53:47.166307+0000","last_undegraded":"2024-03-24T11:53:47.166307+0000","last_fullsized":"2024-03-24T11:53:47.166307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:17.620911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:17.620911+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:20.912812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618332+0000","last_change":"2024-03-24T11:53:37.743905+0000","last_active":"2024-03-24T11:53:46.618332+0000","last_peered":"2024-03-24T11:53:46.618332+0000","last_clean":"2024-03-24T11:53:46.618332+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:53:46.618332+0000","last_undegraded":"2024-03-24T11:53:46.618332+0000","last_fullsized":"2024-03-24T11:53:46.618332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.743809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.743809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.308954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166575+0000","last_change":"2024-03-24T11:52:19.382663+0000","last_active":"2024-03-24T11:53:47.166575+0000","last_peered":"2024-03-24T11:53:47.166575+0000","last_clean":"2024-03-24T11:53:47.166575+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:53:47.166575+0000","last_undegraded":"2024-03-24T11:53:47.166575+0000","last_fullsized":"2024-03-24T11:53:47.166575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.250149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618397+0000","last_change":"2024-03-24T11:53:30.739186+0000","last_active":"2024-03-24T11:53:46.618397+0000","last_peered":"2024-03-24T11:53:46.618397+0000","last_clean":"2024-03-24T11:53:46.618397+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:53:46.618397+0000","last_undegraded":"2024-03-24T11:53:46.618397+0000","last_fullsized":"2024-03-24T11:53:46.618397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:30.739065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.739065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.055226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":115,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618980+0000","last_change":"2024-03-24T11:53:44.635166+0000","last_active":"2024-03-24T11:53:46.618980+0000","last_peered":"2024-03-24T11:53:46.618980+0000","last_clean":"2024-03-24T11:53:46.618980+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:53:46.618980+0000","last_undegraded":"2024-03-24T11:53:46.618980+0000","last_fullsized":"2024-03-24T11:53:46.618980+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:53:44.635073+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:07.128363+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166857+0000","last_change":"2024-03-24T11:53:27.651110+0000","last_active":"2024-03-24T11:53:47.166857+0000","last_peered":"2024-03-24T11:53:47.166857+0000","last_clean":"2024-03-24T11:53:47.166857+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:53:47.166857+0000","last_undegraded":"2024-03-24T11:53:47.166857+0000","last_fullsized":"2024-03-24T11:53:47.166857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:27.650990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:27.650990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:31.773205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617945+0000","last_change":"2024-03-24T11:53:42.989742+0000","last_active":"2024-03-24T11:53:46.617945+0000","last_peered":"2024-03-24T11:53:46.617945+0000","last_clean":"2024-03-24T11:53:46.617945+0000","last_became_active":"2024-03-24T11:52:19.384249+0000","last_became_peered":"2024-03-24T11:52:19.384249+0000","last_unstale":"2024-03-24T11:53:46.617945+0000","last_undegraded":"2024-03-24T11:53:46.617945+0000","last_fullsized":"2024-03-24T11:53:46.617945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:42.989664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:42.989664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:56.543730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619768+0000","last_change":"2024-03-24T11:53:33.356957+0000","last_active":"2024-03-24T11:53:46.619768+0000","last_peered":"2024-03-24T11:53:46.619768+0000","last_clean":"2024-03-24T11:53:46.619768+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:53:46.619768+0000","last_undegraded":"2024-03-24T11:53:46.619768+0000","last_fullsized":"2024-03-24T11:53:46.619768+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:33.356847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:33.356847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.122844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499590+0000","last_change":"2024-03-24T11:53:36.510517+0000","last_active":"2024-03-24T11:53:47.499590+0000","last_peered":"2024-03-24T11:53:47.499590+0000","last_clean":"2024-03-24T11:53:47.499590+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:53:47.499590+0000","last_undegraded":"2024-03-24T11:53:47.499590+0000","last_fullsized":"2024-03-24T11:53:47.499590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:36.510415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.510415+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.640197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617452+0000","last_change":"2024-03-24T11:53:36.945667+0000","last_active":"2024-03-24T11:53:46.617452+0000","last_peered":"2024-03-24T11:53:46.617452+0000","last_clean":"2024-03-24T11:53:46.617452+0000","last_became_active":"2024-03-24T11:52:21.390764+0000","last_became_peered":"2024-03-24T11:52:21.390764+0000","last_unstale":"2024-03-24T11:53:46.617452+0000","last_undegraded":"2024-03-24T11:53:46.617452+0000","last_fullsized":"2024-03-24T11:53:46.617452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:36.945578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.945578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:05.683323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.304037+0000","last_change":"2024-03-24T11:53:21.941429+0000","last_active":"2024-03-24T11:53:47.304037+0000","last_peered":"2024-03-24T11:53:47.304037+0000","last_clean":"2024-03-24T11:53:47.304037+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:53:47.304037+0000","last_undegraded":"2024-03-24T11:53:47.304037+0000","last_fullsized":"2024-03-24T11:53:47.304037+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_clean_scrub_stamp":"2024-03-24T11:53:21.941323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:31.533370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":66,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617005+0000","last_change":"2024-03-24T11:52:15.136259+0000","last_active":"2024-03-24T11:53:46.617005+0000","last_peered":"2024-03-24T11:53:46.617005+0000","last_clean":"2024-03-24T11:53:46.617005+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:53:46.617005+0000","last_undegraded":"2024-03-24T11:53:46.617005+0000","last_fullsized":"2024-03-24T11:53:46.617005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619780+0000","last_change":"2024-03-24T11:53:29.274282+0000","last_active":"2024-03-24T11:53:46.619780+0000","last_peered":"2024-03-24T11:53:46.619780+0000","last_clean":"2024-03-24T11:53:46.619780+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:53:46.619780+0000","last_undegraded":"2024-03-24T11:53:46.619780+0000","last_fullsized":"2024-03-24T11:53:46.619780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:53:29.274127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.274127+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:43.553788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332306+0000","last_change":"2024-03-24T11:53:39.602043+0000","last_active":"2024-03-24T11:53:47.332306+0000","last_peered":"2024-03-24T11:53:47.332306+0000","last_clean":"2024-03-24T11:53:47.332306+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:53:47.332306+0000","last_undegraded":"2024-03-24T11:53:47.332306+0000","last_fullsized":"2024-03-24T11:53:47.332306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:53:39.601937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.601937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.250197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.334525+0000","last_change":"2024-03-24T11:53:47.334525+0000","last_active":"2024-03-24T11:53:47.334525+0000","last_peered":"2024-03-24T11:53:47.334525+0000","last_clean":"2024-03-24T11:53:47.334525+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:53:47.334525+0000","last_undegraded":"2024-03-24T11:53:47.334525+0000","last_fullsized":"2024-03-24T11:53:47.334525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:47.334412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.334412+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:05.508632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748292+0000","last_change":"2024-03-24T11:53:35.814141+0000","last_active":"2024-03-24T11:53:46.748292+0000","last_peered":"2024-03-24T11:53:46.748292+0000","last_clean":"2024-03-24T11:53:46.748292+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:53:46.748292+0000","last_undegraded":"2024-03-24T11:53:46.748292+0000","last_fullsized":"2024-03-24T11:53:46.748292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.814032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.814032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.550457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.304025+0000","last_change":"2024-03-24T11:53:46.014153+0000","last_active":"2024-03-24T11:53:47.304025+0000","last_peered":"2024-03-24T11:53:47.304025+0000","last_clean":"2024-03-24T11:53:47.304025+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:53:47.304025+0000","last_undegraded":"2024-03-24T11:53:47.304025+0000","last_fullsized":"2024-03-24T11:53:47.304025+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_clean_scrub_stamp":"2024-03-24T11:53:46.014065+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:06.936760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617603+0000","last_change":"2024-03-24T11:52:21.854322+0000","last_active":"2024-03-24T11:53:46.617603+0000","last_peered":"2024-03-24T11:53:46.617603+0000","last_clean":"2024-03-24T11:53:46.617603+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:53:46.617603+0000","last_undegraded":"2024-03-24T11:53:46.617603+0000","last_fullsized":"2024-03-24T11:53:46.617603+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332115+0000","last_change":"2024-03-24T11:52:19.385213+0000","last_active":"2024-03-24T11:53:47.332115+0000","last_peered":"2024-03-24T11:53:47.332115+0000","last_clean":"2024-03-24T11:53:47.332115+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:53:47.332115+0000","last_undegraded":"2024-03-24T11:53:47.332115+0000","last_fullsized":"2024-03-24T11:53:47.332115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.620013+0000","last_change":"2024-03-24T11:53:35.343688+0000","last_active":"2024-03-24T11:53:46.620013+0000","last_peered":"2024-03-24T11:53:46.620013+0000","last_clean":"2024-03-24T11:53:46.620013+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:53:46.620013+0000","last_undegraded":"2024-03-24T11:53:46.620013+0000","last_fullsized":"2024-03-24T11:53:46.620013+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.343594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.343594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.531863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617313+0000","last_change":"2024-03-24T11:52:17.423884+0000","last_active":"2024-03-24T11:53:46.617313+0000","last_peered":"2024-03-24T11:53:46.617313+0000","last_clean":"2024-03-24T11:53:46.617313+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:53:46.617313+0000","last_undegraded":"2024-03-24T11:53:46.617313+0000","last_fullsized":"2024-03-24T11:53:46.617313+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331641+0000","last_change":"2024-03-24T11:52:21.390345+0000","last_active":"2024-03-24T11:53:47.331641+0000","last_peered":"2024-03-24T11:53:47.331641+0000","last_clean":"2024-03-24T11:53:47.331641+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:53:47.331641+0000","last_undegraded":"2024-03-24T11:53:47.331641+0000","last_fullsized":"2024-03-24T11:53:47.331641+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619196+0000","last_change":"2024-03-24T11:53:30.303601+0000","last_active":"2024-03-24T11:53:46.619196+0000","last_peered":"2024-03-24T11:53:46.619196+0000","last_clean":"2024-03-24T11:53:46.619196+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:53:46.619196+0000","last_undegraded":"2024-03-24T11:53:46.619196+0000","last_fullsized":"2024-03-24T11:53:46.619196+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.303505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.303505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.216822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748823+0000","last_change":"2024-03-24T11:52:15.142488+0000","last_active":"2024-03-24T11:53:46.748823+0000","last_peered":"2024-03-24T11:53:46.748823+0000","last_clean":"2024-03-24T11:53:46.748823+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:53:46.748823+0000","last_undegraded":"2024-03-24T11:53:46.748823+0000","last_fullsized":"2024-03-24T11:53:46.748823+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.616853+0000","last_change":"2024-03-24T11:53:39.743799+0000","last_active":"2024-03-24T11:53:46.616853+0000","last_peered":"2024-03-24T11:53:46.616853+0000","last_clean":"2024-03-24T11:53:46.616853+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:53:46.616853+0000","last_undegraded":"2024-03-24T11:53:46.616853+0000","last_fullsized":"2024-03-24T11:53:46.616853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T11:53:39.743685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.743685+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.765998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.616774+0000","last_change":"2024-03-24T11:52:21.381608+0000","last_active":"2024-03-24T11:53:46.616774+0000","last_peered":"2024-03-24T11:53:46.616774+0000","last_clean":"2024-03-24T11:53:46.616774+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:53:46.616774+0000","last_undegraded":"2024-03-24T11:53:46.616774+0000","last_fullsized":"2024-03-24T11:53:46.616774+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:47.376283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618307+0000","last_change":"2024-03-24T11:53:44.948231+0000","last_active":"2024-03-24T11:53:46.618307+0000","last_peered":"2024-03-24T11:53:46.618307+0000","last_clean":"2024-03-24T11:53:46.618307+0000","last_became_active":"2024-03-24T11:52:19.377484+0000","last_became_peered":"2024-03-24T11:52:19.377484+0000","last_unstale":"2024-03-24T11:53:46.618307+0000","last_undegraded":"2024-03-24T11:53:46.618307+0000","last_fullsized":"2024-03-24T11:53:46.618307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:53:44.948146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:13.027078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617651+0000","last_change":"2024-03-24T11:53:32.981301+0000","last_active":"2024-03-24T11:53:46.617651+0000","last_peered":"2024-03-24T11:53:46.617651+0000","last_clean":"2024-03-24T11:53:46.617651+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:53:46.617651+0000","last_undegraded":"2024-03-24T11:53:46.617651+0000","last_fullsized":"2024-03-24T11:53:46.617651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:32.981156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:32.981156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.401524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166570+0000","last_change":"2024-03-24T11:52:17.359275+0000","last_active":"2024-03-24T11:53:47.166570+0000","last_peered":"2024-03-24T11:53:47.166570+0000","last_clean":"2024-03-24T11:53:47.166570+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:53:47.166570+0000","last_undegraded":"2024-03-24T11:53:47.166570+0000","last_fullsized":"2024-03-24T11:53:47.166570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303237+0000","last_change":"2024-03-24T11:53:37.904794+0000","last_active":"2024-03-24T11:53:47.303237+0000","last_peered":"2024-03-24T11:53:47.303237+0000","last_clean":"2024-03-24T11:53:47.303237+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:53:47.303237+0000","last_undegraded":"2024-03-24T11:53:47.303237+0000","last_fullsized":"2024-03-24T11:53:47.303237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.904622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.904622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.565966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617416+0000","last_change":"2024-03-24T11:52:19.378422+0000","last_active":"2024-03-24T11:53:46.617416+0000","last_peered":"2024-03-24T11:53:46.617416+0000","last_clean":"2024-03-24T11:53:46.617416+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:53:46.617416+0000","last_undegraded":"2024-03-24T11:53:46.617416+0000","last_fullsized":"2024-03-24T11:53:46.617416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619437+0000","last_change":"2024-03-24T11:53:43.360101+0000","last_active":"2024-03-24T11:53:46.619437+0000","last_peered":"2024-03-24T11:53:46.619437+0000","last_clean":"2024-03-24T11:53:46.619437+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:53:46.619437+0000","last_undegraded":"2024-03-24T11:53:46.619437+0000","last_fullsized":"2024-03-24T11:53:46.619437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:43.359979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:43.359979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:53.166274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618371+0000","last_change":"2024-03-24T11:53:25.921423+0000","last_active":"2024-03-24T11:53:46.618371+0000","last_peered":"2024-03-24T11:53:46.618371+0000","last_clean":"2024-03-24T11:53:46.618371+0000","last_became_active":"2024-03-24T11:52:17.355956+0000","last_became_peered":"2024-03-24T11:52:17.355956+0000","last_unstale":"2024-03-24T11:53:46.618371+0000","last_undegraded":"2024-03-24T11:53:46.618371+0000","last_fullsized":"2024-03-24T11:53:46.618371+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:25.921326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.921326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:33.718763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.167048+0000","last_change":"2024-03-24T11:53:44.686755+0000","last_active":"2024-03-24T11:53:47.167048+0000","last_peered":"2024-03-24T11:53:47.167048+0000","last_clean":"2024-03-24T11:53:47.167048+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:53:47.167048+0000","last_undegraded":"2024-03-24T11:53:47.167048+0000","last_fullsized":"2024-03-24T11:53:47.167048+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:44.686619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:44.686619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:56.785834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617551+0000","last_change":"2024-03-24T11:53:28.755945+0000","last_active":"2024-03-24T11:53:46.617551+0000","last_peered":"2024-03-24T11:53:46.617551+0000","last_clean":"2024-03-24T11:53:46.617551+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:53:46.617551+0000","last_undegraded":"2024-03-24T11:53:46.617551+0000","last_fullsized":"2024-03-24T11:53:46.617551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:28.755826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:28.755826+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.854085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303678+0000","last_change":"2024-03-24T11:52:15.135724+0000","last_active":"2024-03-24T11:53:47.303678+0000","last_peered":"2024-03-24T11:53:47.303678+0000","last_clean":"2024-03-24T11:53:47.303678+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:53:47.303678+0000","last_undegraded":"2024-03-24T11:53:47.303678+0000","last_fullsized":"2024-03-24T11:53:47.303678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.836732+0000","last_change":"2024-03-24T11:53:47.836766+0000","last_active":"2024-03-24T11:53:47.836732+0000","last_peered":"2024-03-24T11:53:47.836732+0000","last_clean":"2024-03-24T11:53:47.836732+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:53:47.836732+0000","last_undegraded":"2024-03-24T11:53:47.836732+0000","last_fullsized":"2024-03-24T11:53:47.836732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:47.836657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.836657+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.133633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303779+0000","last_change":"2024-03-24T11:52:21.382650+0000","last_active":"2024-03-24T11:53:47.303779+0000","last_peered":"2024-03-24T11:53:47.303779+0000","last_clean":"2024-03-24T11:53:47.303779+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:53:47.303779+0000","last_undegraded":"2024-03-24T11:53:47.303779+0000","last_fullsized":"2024-03-24T11:53:47.303779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748950+0000","last_change":"2024-03-24T11:53:32.812255+0000","last_active":"2024-03-24T11:53:46.748950+0000","last_peered":"2024-03-24T11:53:46.748950+0000","last_clean":"2024-03-24T11:53:46.748950+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:53:46.748950+0000","last_undegraded":"2024-03-24T11:53:46.748950+0000","last_fullsized":"2024-03-24T11:53:46.748950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:32.812047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:32.812047+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:50.393185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748435+0000","last_change":"2024-03-24T11:52:15.138567+0000","last_active":"2024-03-24T11:53:46.748435+0000","last_peered":"2024-03-24T11:53:46.748435+0000","last_clean":"2024-03-24T11:53:46.748435+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:53:46.748435+0000","last_undegraded":"2024-03-24T11:53:46.748435+0000","last_fullsized":"2024-03-24T11:53:46.748435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.583092+0000","last_change":"2024-03-24T11:53:47.583124+0000","last_active":"2024-03-24T11:53:47.583092+0000","last_peered":"2024-03-24T11:53:47.583092+0000","last_clean":"2024-03-24T11:53:47.583092+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:53:47.583092+0000","last_undegraded":"2024-03-24T11:53:47.583092+0000","last_fullsized":"2024-03-24T11:53:47.583092+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.583025+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.607356+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331963+0000","last_change":"2024-03-24T11:53:42.480919+0000","last_active":"2024-03-24T11:53:47.331963+0000","last_peered":"2024-03-24T11:53:47.331963+0000","last_clean":"2024-03-24T11:53:47.331963+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:53:47.331963+0000","last_undegraded":"2024-03-24T11:53:47.331963+0000","last_fullsized":"2024-03-24T11:53:47.331963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:42.480832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:42.480832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.968593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748758+0000","last_change":"2024-03-24T11:52:19.379744+0000","last_active":"2024-03-24T11:53:46.748758+0000","last_peered":"2024-03-24T11:53:46.748758+0000","last_clean":"2024-03-24T11:53:46.748758+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:53:46.748758+0000","last_undegraded":"2024-03-24T11:53:46.748758+0000","last_fullsized":"2024-03-24T11:53:46.748758+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619228+0000","last_change":"2024-03-24T11:53:21.338998+0000","last_active":"2024-03-24T11:53:46.619228+0000","last_peered":"2024-03-24T11:53:46.619228+0000","last_clean":"2024-03-24T11:53:46.619228+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:53:46.619228+0000","last_undegraded":"2024-03-24T11:53:46.619228+0000","last_fullsized":"2024-03-24T11:53:46.619228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:21.338907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:21.338907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:27.393056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618226+0000","last_change":"2024-03-24T11:53:22.834856+0000","last_active":"2024-03-24T11:53:46.618226+0000","last_peered":"2024-03-24T11:53:46.618226+0000","last_clean":"2024-03-24T11:53:46.618226+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:53:46.618226+0000","last_undegraded":"2024-03-24T11:53:46.618226+0000","last_fullsized":"2024-03-24T11:53:46.618226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:53:22.834770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:22.834770+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:35.505245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499383+0000","last_change":"2024-03-24T11:52:21.386021+0000","last_active":"2024-03-24T11:53:47.499383+0000","last_peered":"2024-03-24T11:53:47.499383+0000","last_clean":"2024-03-24T11:53:47.499383+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:53:47.499383+0000","last_undegraded":"2024-03-24T11:53:47.499383+0000","last_fullsized":"2024-03-24T11:53:47.499383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618651+0000","last_change":"2024-03-24T11:52:19.376888+0000","last_active":"2024-03-24T11:53:46.618651+0000","last_peered":"2024-03-24T11:53:46.618651+0000","last_clean":"2024-03-24T11:53:46.618651+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:53:46.618651+0000","last_undegraded":"2024-03-24T11:53:46.618651+0000","last_fullsized":"2024-03-24T11:53:46.618651+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499326+0000","last_change":"2024-03-24T11:53:38.430978+0000","last_active":"2024-03-24T11:53:47.499326+0000","last_peered":"2024-03-24T11:53:47.499326+0000","last_clean":"2024-03-24T11:53:47.499326+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:53:47.499326+0000","last_undegraded":"2024-03-24T11:53:47.499326+0000","last_fullsized":"2024-03-24T11:53:47.499326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:38.430896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:38.430896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:46.558321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303898+0000","last_change":"2024-03-24T11:53:38.899939+0000","last_active":"2024-03-24T11:53:47.303898+0000","last_peered":"2024-03-24T11:53:47.303898+0000","last_clean":"2024-03-24T11:53:47.303898+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:53:47.303898+0000","last_undegraded":"2024-03-24T11:53:47.303898+0000","last_fullsized":"2024-03-24T11:53:47.303898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:38.899823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:38.899823+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.812037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617503+0000","last_change":"2024-03-24T11:52:21.389019+0000","last_active":"2024-03-24T11:53:46.617503+0000","last_peered":"2024-03-24T11:53:46.617503+0000","last_clean":"2024-03-24T11:53:46.617503+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:53:46.617503+0000","last_undegraded":"2024-03-24T11:53:46.617503+0000","last_fullsized":"2024-03-24T11:53:46.617503+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166888+0000","last_change":"2024-03-24T11:53:35.720564+0000","last_active":"2024-03-24T11:53:47.166888+0000","last_peered":"2024-03-24T11:53:47.166888+0000","last_clean":"2024-03-24T11:53:47.166888+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:53:47.166888+0000","last_undegraded":"2024-03-24T11:53:47.166888+0000","last_fullsized":"2024-03-24T11:53:47.166888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:53:35.720425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.720425+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.137342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":108,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166371+0000","last_change":"2024-03-24T11:53:22.656282+0000","last_active":"2024-03-24T11:53:47.166371+0000","last_peered":"2024-03-24T11:53:47.166371+0000","last_clean":"2024-03-24T11:53:47.166371+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:53:47.166371+0000","last_undegraded":"2024-03-24T11:53:47.166371+0000","last_fullsized":"2024-03-24T11:53:47.166371+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:53:22.656153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:22.656153+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:23.354452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:50.425097+0000","last_change":"2024-03-24T11:53:50.425119+0000","last_active":"2024-03-24T11:53:50.425097+0000","last_peered":"2024-03-24T11:53:50.425097+0000","last_clean":"2024-03-24T11:53:50.425097+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:53:50.425097+0000","last_undegraded":"2024-03-24T11:53:50.425097+0000","last_fullsized":"2024-03-24T11:53:50.425097+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:50.425035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:50.425035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:06.064329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303241+0000","last_change":"2024-03-24T11:52:21.386419+0000","last_active":"2024-03-24T11:53:47.303241+0000","last_peered":"2024-03-24T11:53:47.303241+0000","last_clean":"2024-03-24T11:53:47.303241+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:53:47.303241+0000","last_undegraded":"2024-03-24T11:53:47.303241+0000","last_fullsized":"2024-03-24T11:53:47.303241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:50.950640+0000","last_change":"2024-03-24T11:53:50.950662+0000","last_active":"2024-03-24T11:53:50.950640+0000","last_peered":"2024-03-24T11:53:50.950640+0000","last_clean":"2024-03-24T11:53:50.950640+0000","last_became_active":"2024-03-24T11:52:15.138792+0000","last_became_peered":"2024-03-24T11:52:15.138792+0000","last_unstale":"2024-03-24T11:53:50.950640+0000","last_undegraded":"2024-03-24T11:53:50.950640+0000","last_fullsized":"2024-03-24T11:53:50.950640+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:53:50.950565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.098973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618201+0000","last_change":"2024-03-24T11:53:26.940720+0000","last_active":"2024-03-24T11:53:46.618201+0000","last_peered":"2024-03-24T11:53:46.618201+0000","last_clean":"2024-03-24T11:53:46.618201+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:53:46.618201+0000","last_undegraded":"2024-03-24T11:53:46.618201+0000","last_fullsized":"2024-03-24T11:53:46.618201+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:53:26.940542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:26.940542+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.977884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303852+0000","last_change":"2024-03-24T11:52:19.378322+0000","last_active":"2024-03-24T11:53:47.303852+0000","last_peered":"2024-03-24T11:53:47.303852+0000","last_clean":"2024-03-24T11:53:47.303852+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:53:47.303852+0000","last_undegraded":"2024-03-24T11:53:47.303852+0000","last_fullsized":"2024-03-24T11:53:47.303852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619305+0000","last_change":"2024-03-24T11:53:42.665773+0000","last_active":"2024-03-24T11:53:46.619305+0000","last_peered":"2024-03-24T11:53:46.619305+0000","last_clean":"2024-03-24T11:53:46.619305+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:53:46.619305+0000","last_undegraded":"2024-03-24T11:53:46.619305+0000","last_fullsized":"2024-03-24T11:53:46.619305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:42.665691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:42.665691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:03.119522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748529+0000","last_change":"2024-03-24T11:53:36.855528+0000","last_active":"2024-03-24T11:53:46.748529+0000","last_peered":"2024-03-24T11:53:46.748529+0000","last_clean":"2024-03-24T11:53:46.748529+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:53:46.748529+0000","last_undegraded":"2024-03-24T11:53:46.748529+0000","last_fullsized":"2024-03-24T11:53:46.748529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:36.855425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.855425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:37.779842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748779+0000","last_change":"2024-03-24T11:52:37.802552+0000","last_active":"2024-03-24T11:53:46.748779+0000","last_peered":"2024-03-24T11:53:46.748779+0000","last_clean":"2024-03-24T11:53:46.748779+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:53:46.748779+0000","last_undegraded":"2024-03-24T11:53:46.748779+0000","last_fullsized":"2024-03-24T11:53:46.748779+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.304055+0000","last_change":"2024-03-24T11:53:45.014753+0000","last_active":"2024-03-24T11:53:47.304055+0000","last_peered":"2024-03-24T11:53:47.304055+0000","last_clean":"2024-03-24T11:53:47.304055+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:53:47.304055+0000","last_undegraded":"2024-03-24T11:53:47.304055+0000","last_fullsized":"2024-03-24T11:53:47.304055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.014670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.014670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.454483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332229+0000","last_change":"2024-03-24T11:53:34.632333+0000","last_active":"2024-03-24T11:53:47.332229+0000","last_peered":"2024-03-24T11:53:47.332229+0000","last_clean":"2024-03-24T11:53:47.332229+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:53:47.332229+0000","last_undegraded":"2024-03-24T11:53:47.332229+0000","last_fullsized":"2024-03-24T11:53:47.332229+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:34.632228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:34.632228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.019688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":172,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:49.833771+0000","last_change":"2024-03-24T11:53:39.041508+0000","last_active":"2024-03-24T11:53:49.833771+0000","last_peered":"2024-03-24T11:53:49.833771+0000","last_clean":"2024-03-24T11:53:49.833771+0000","last_became_active":"2024-03-24T11:52:15.144954+0000","last_became_peered":"2024-03-24T11:52:15.144954+0000","last_unstale":"2024-03-24T11:53:49.833771+0000","last_undegraded":"2024-03-24T11:53:49.833771+0000","last_fullsized":"2024-03-24T11:53:49.833771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:53:39.041412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.041412+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:06.745360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499159+0000","last_change":"2024-03-24T11:53:29.429575+0000","last_active":"2024-03-24T11:53:47.499159+0000","last_peered":"2024-03-24T11:53:47.499159+0000","last_clean":"2024-03-24T11:53:47.499159+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:53:47.499159+0000","last_undegraded":"2024-03-24T11:53:47.499159+0000","last_fullsized":"2024-03-24T11:53:47.499159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.429483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.429483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.616922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618111+0000","last_change":"2024-03-24T11:52:17.358816+0000","last_active":"2024-03-24T11:53:46.618111+0000","last_peered":"2024-03-24T11:53:46.618111+0000","last_clean":"2024-03-24T11:53:46.618111+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:53:46.618111+0000","last_undegraded":"2024-03-24T11:53:46.618111+0000","last_fullsized":"2024-03-24T11:53:46.618111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499049+0000","last_change":"2024-03-24T11:52:19.383334+0000","last_active":"2024-03-24T11:53:47.499049+0000","last_peered":"2024-03-24T11:53:47.499049+0000","last_clean":"2024-03-24T11:53:47.499049+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:53:47.499049+0000","last_undegraded":"2024-03-24T11:53:47.499049+0000","last_fullsized":"2024-03-24T11:53:47.499049+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.749021+0000","last_change":"2024-03-24T11:53:41.822978+0000","last_active":"2024-03-24T11:53:46.749021+0000","last_peered":"2024-03-24T11:53:46.749021+0000","last_clean":"2024-03-24T11:53:46.749021+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:53:46.749021+0000","last_undegraded":"2024-03-24T11:53:46.749021+0000","last_fullsized":"2024-03-24T11:53:46.749021+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:53:41.822890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.481555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617065+0000","last_change":"2024-03-24T11:52:19.368496+0000","last_active":"2024-03-24T11:53:46.617065+0000","last_peered":"2024-03-24T11:53:46.617065+0000","last_clean":"2024-03-24T11:53:46.617065+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:53:46.617065+0000","last_undegraded":"2024-03-24T11:53:46.617065+0000","last_fullsized":"2024-03-24T11:53:46.617065+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:50.814817+0000","last_change":"2024-03-24T11:53:50.814854+0000","last_active":"2024-03-24T11:53:50.814817+0000","last_peered":"2024-03-24T11:53:50.814817+0000","last_clean":"2024-03-24T11:53:50.814817+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:53:50.814817+0000","last_undegraded":"2024-03-24T11:53:50.814817+0000","last_fullsized":"2024-03-24T11:53:50.814817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:50.814720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:50.814720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:05.078424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":319,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331593+0000","last_change":"2024-03-24T11:52:19.498367+0000","last_active":"2024-03-24T11:53:47.331593+0000","last_peered":"2024-03-24T11:53:47.331593+0000","last_clean":"2024-03-24T11:53:47.331593+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:53:47.331593+0000","last_undegraded":"2024-03-24T11:53:47.331593+0000","last_fullsized":"2024-03-24T11:53:47.331593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'119","last_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_deep_scrub":"71'119","last_deep_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_clean_scrub_stamp":"2024-03-24T11:52:19.498276+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619584+0000","last_change":"2024-03-24T11:53:41.364342+0000","last_active":"2024-03-24T11:53:46.619584+0000","last_peered":"2024-03-24T11:53:46.619584+0000","last_clean":"2024-03-24T11:53:46.619584+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:53:46.619584+0000","last_undegraded":"2024-03-24T11:53:46.619584+0000","last_fullsized":"2024-03-24T11:53:46.619584+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:53:41.364225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:41.364225+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.172024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332134+0000","last_change":"2024-03-24T11:53:29.691119+0000","last_active":"2024-03-24T11:53:47.332134+0000","last_peered":"2024-03-24T11:53:47.332134+0000","last_clean":"2024-03-24T11:53:47.332134+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:53:47.332134+0000","last_undegraded":"2024-03-24T11:53:47.332134+0000","last_fullsized":"2024-03-24T11:53:47.332134+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:29.691006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:29.691006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:54.100657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.960225+0000","last_change":"2024-03-24T11:53:47.960246+0000","last_active":"2024-03-24T11:53:47.960225+0000","last_peered":"2024-03-24T11:53:47.960225+0000","last_clean":"2024-03-24T11:53:47.960225+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:53:47.960225+0000","last_undegraded":"2024-03-24T11:53:47.960225+0000","last_fullsized":"2024-03-24T11:53:47.960225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:47.960166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.960166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.716011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303538+0000","last_change":"2024-03-24T11:52:19.366588+0000","last_active":"2024-03-24T11:53:47.303538+0000","last_peered":"2024-03-24T11:53:47.303538+0000","last_clean":"2024-03-24T11:53:47.303538+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:53:47.303538+0000","last_undegraded":"2024-03-24T11:53:47.303538+0000","last_fullsized":"2024-03-24T11:53:47.303538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331521+0000","last_change":"2024-03-24T11:53:24.679709+0000","last_active":"2024-03-24T11:53:47.331521+0000","last_peered":"2024-03-24T11:53:47.331521+0000","last_clean":"2024-03-24T11:53:47.331521+0000","last_became_active":"2024-03-24T11:52:15.146310+0000","last_became_peered":"2024-03-24T11:52:15.146310+0000","last_unstale":"2024-03-24T11:53:47.331521+0000","last_undegraded":"2024-03-24T11:53:47.331521+0000","last_fullsized":"2024-03-24T11:53:47.331521+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:24.679579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.679579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:29.622468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303870+0000","last_change":"2024-03-24T11:53:24.917322+0000","last_active":"2024-03-24T11:53:47.303870+0000","last_peered":"2024-03-24T11:53:47.303870+0000","last_clean":"2024-03-24T11:53:47.303870+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:53:47.303870+0000","last_undegraded":"2024-03-24T11:53:47.303870+0000","last_fullsized":"2024-03-24T11:53:47.303870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:53:24.917220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.917220+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.119999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748949+0000","last_change":"2024-03-24T11:52:19.387410+0000","last_active":"2024-03-24T11:53:46.748949+0000","last_peered":"2024-03-24T11:53:46.748949+0000","last_clean":"2024-03-24T11:53:46.748949+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:53:46.748949+0000","last_undegraded":"2024-03-24T11:53:46.748949+0000","last_fullsized":"2024-03-24T11:53:46.748949+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617788+0000","last_change":"2024-03-24T11:53:41.036691+0000","last_active":"2024-03-24T11:53:46.617788+0000","last_peered":"2024-03-24T11:53:46.617788+0000","last_clean":"2024-03-24T11:53:46.617788+0000","last_became_active":"2024-03-24T11:52:21.851069+0000","last_became_peered":"2024-03-24T11:52:21.851069+0000","last_unstale":"2024-03-24T11:53:46.617788+0000","last_undegraded":"2024-03-24T11:53:46.617788+0000","last_fullsized":"2024-03-24T11:53:46.617788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:41.036572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:41.036572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:42.293567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332329+0000","last_change":"2024-03-24T11:53:18.576655+0000","last_active":"2024-03-24T11:53:47.332329+0000","last_peered":"2024-03-24T11:53:47.332329+0000","last_clean":"2024-03-24T11:53:47.332329+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:53:47.332329+0000","last_undegraded":"2024-03-24T11:53:47.332329+0000","last_fullsized":"2024-03-24T11:53:47.332329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:18.576538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:18.576538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.141440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":104,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303673+0000","last_change":"2024-03-24T11:53:39.923840+0000","last_active":"2024-03-24T11:53:47.303673+0000","last_peered":"2024-03-24T11:53:47.303673+0000","last_clean":"2024-03-24T11:53:47.303673+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:53:47.303673+0000","last_undegraded":"2024-03-24T11:53:47.303673+0000","last_fullsized":"2024-03-24T11:53:47.303673+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:53:39.923699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.923699+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.413674+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:50.319937+0000","last_change":"2024-03-24T11:53:50.319962+0000","last_active":"2024-03-24T11:53:50.319937+0000","last_peered":"2024-03-24T11:53:50.319937+0000","last_clean":"2024-03-24T11:53:50.319937+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:53:50.319937+0000","last_undegraded":"2024-03-24T11:53:50.319937+0000","last_fullsized":"2024-03-24T11:53:50.319937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T11:53:50.319837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.251719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166668+0000","last_change":"2024-03-24T11:53:39.653196+0000","last_active":"2024-03-24T11:53:47.166668+0000","last_peered":"2024-03-24T11:53:47.166668+0000","last_clean":"2024-03-24T11:53:47.166668+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:53:47.166668+0000","last_undegraded":"2024-03-24T11:53:47.166668+0000","last_fullsized":"2024-03-24T11:53:47.166668+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:39.653110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.653110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.778426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617540+0000","last_change":"2024-03-24T11:53:40.019269+0000","last_active":"2024-03-24T11:53:46.617540+0000","last_peered":"2024-03-24T11:53:46.617540+0000","last_clean":"2024-03-24T11:53:46.617540+0000","last_became_active":"2024-03-24T11:52:15.142974+0000","last_became_peered":"2024-03-24T11:52:15.142974+0000","last_unstale":"2024-03-24T11:53:46.617540+0000","last_undegraded":"2024-03-24T11:53:46.617540+0000","last_fullsized":"2024-03-24T11:53:46.617540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:40.019165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:40.019165+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:45.538489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.616730+0000","last_change":"2024-03-24T11:53:35.726728+0000","last_active":"2024-03-24T11:53:46.616730+0000","last_peered":"2024-03-24T11:53:46.616730+0000","last_clean":"2024-03-24T11:53:46.616730+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:53:46.616730+0000","last_undegraded":"2024-03-24T11:53:46.616730+0000","last_fullsized":"2024-03-24T11:53:46.616730+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:53:35.726630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.726630+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:37.790720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166376+0000","last_change":"2024-03-24T11:52:19.389693+0000","last_active":"2024-03-24T11:53:47.166376+0000","last_peered":"2024-03-24T11:53:47.166376+0000","last_clean":"2024-03-24T11:53:47.166376+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:53:47.166376+0000","last_undegraded":"2024-03-24T11:53:47.166376+0000","last_fullsized":"2024-03-24T11:53:47.166376+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.332057+0000","last_change":"2024-03-24T11:52:21.854938+0000","last_active":"2024-03-24T11:53:47.332057+0000","last_peered":"2024-03-24T11:53:47.332057+0000","last_clean":"2024-03-24T11:53:47.332057+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:53:47.332057+0000","last_undegraded":"2024-03-24T11:53:47.332057+0000","last_fullsized":"2024-03-24T11:53:47.332057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619904+0000","last_change":"2024-03-24T11:53:15.292341+0000","last_active":"2024-03-24T11:53:46.619904+0000","last_peered":"2024-03-24T11:53:46.619904+0000","last_clean":"2024-03-24T11:53:46.619904+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:53:46.619904+0000","last_undegraded":"2024-03-24T11:53:46.619904+0000","last_fullsized":"2024-03-24T11:53:46.619904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:53:15.292233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:41.790414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":95,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166236+0000","last_change":"2024-03-24T11:53:26.665675+0000","last_active":"2024-03-24T11:53:47.166236+0000","last_peered":"2024-03-24T11:53:47.166236+0000","last_clean":"2024-03-24T11:53:47.166236+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:53:47.166236+0000","last_undegraded":"2024-03-24T11:53:47.166236+0000","last_fullsized":"2024-03-24T11:53:47.166236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:53:26.665554+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:48.241747+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.616713+0000","last_change":"2024-03-24T11:52:19.382757+0000","last_active":"2024-03-24T11:53:46.616713+0000","last_peered":"2024-03-24T11:53:46.616713+0000","last_clean":"2024-03-24T11:53:46.616713+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:53:46.616713+0000","last_undegraded":"2024-03-24T11:53:46.616713+0000","last_fullsized":"2024-03-24T11:53:46.616713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617009+0000","last_change":"2024-03-24T11:52:21.851542+0000","last_active":"2024-03-24T11:53:46.617009+0000","last_peered":"2024-03-24T11:53:46.617009+0000","last_clean":"2024-03-24T11:53:46.617009+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:53:46.617009+0000","last_undegraded":"2024-03-24T11:53:46.617009+0000","last_fullsized":"2024-03-24T11:53:46.617009+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748923+0000","last_change":"2024-03-24T11:52:19.388558+0000","last_active":"2024-03-24T11:53:46.748923+0000","last_peered":"2024-03-24T11:53:46.748923+0000","last_clean":"2024-03-24T11:53:46.748923+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:53:46.748923+0000","last_undegraded":"2024-03-24T11:53:46.748923+0000","last_fullsized":"2024-03-24T11:53:46.748923+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331638+0000","last_change":"2024-03-24T11:53:30.710184+0000","last_active":"2024-03-24T11:53:47.331638+0000","last_peered":"2024-03-24T11:53:47.331638+0000","last_clean":"2024-03-24T11:53:47.331638+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:53:47.331638+0000","last_undegraded":"2024-03-24T11:53:47.331638+0000","last_fullsized":"2024-03-24T11:53:47.331638+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.710079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.710079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:39.148875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166948+0000","last_change":"2024-03-24T11:52:17.351215+0000","last_active":"2024-03-24T11:53:47.166948+0000","last_peered":"2024-03-24T11:53:47.166948+0000","last_clean":"2024-03-24T11:53:47.166948+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:53:47.166948+0000","last_undegraded":"2024-03-24T11:53:47.166948+0000","last_fullsized":"2024-03-24T11:53:47.166948+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:52:16.324438+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:49.795398+0000","last_change":"2024-03-24T11:53:49.795423+0000","last_active":"2024-03-24T11:53:49.795398+0000","last_peered":"2024-03-24T11:53:49.795398+0000","last_clean":"2024-03-24T11:53:49.795398+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:53:49.795398+0000","last_undegraded":"2024-03-24T11:53:49.795398+0000","last_fullsized":"2024-03-24T11:53:49.795398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:49.795328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:49.795328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:55.881169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166762+0000","last_change":"2024-03-24T11:53:42.651953+0000","last_active":"2024-03-24T11:53:47.166762+0000","last_peered":"2024-03-24T11:53:47.166762+0000","last_clean":"2024-03-24T11:53:47.166762+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:53:47.166762+0000","last_undegraded":"2024-03-24T11:53:47.166762+0000","last_fullsized":"2024-03-24T11:53:47.166762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:42.651866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:42.651866+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:47.201095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617902+0000","last_change":"2024-03-24T11:53:45.930437+0000","last_active":"2024-03-24T11:53:46.617902+0000","last_peered":"2024-03-24T11:53:46.617902+0000","last_clean":"2024-03-24T11:53:46.617902+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:53:46.617902+0000","last_undegraded":"2024-03-24T11:53:46.617902+0000","last_fullsized":"2024-03-24T11:53:46.617902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.930342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.930342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:15.517071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617650+0000","last_change":"2024-03-24T11:53:31.957804+0000","last_active":"2024-03-24T11:53:46.617650+0000","last_peered":"2024-03-24T11:53:46.617650+0000","last_clean":"2024-03-24T11:53:46.617650+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:53:46.617650+0000","last_undegraded":"2024-03-24T11:53:46.617650+0000","last_fullsized":"2024-03-24T11:53:46.617650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:31.957706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.957706+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.121723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166721+0000","last_change":"2024-03-24T11:53:31.724380+0000","last_active":"2024-03-24T11:53:47.166721+0000","last_peered":"2024-03-24T11:53:47.166721+0000","last_clean":"2024-03-24T11:53:47.166721+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:53:47.166721+0000","last_undegraded":"2024-03-24T11:53:47.166721+0000","last_fullsized":"2024-03-24T11:53:47.166721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:53:31.724235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.724235+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:44.862511+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.498982+0000","last_change":"2024-03-24T11:53:19.389995+0000","last_active":"2024-03-24T11:53:47.498982+0000","last_peered":"2024-03-24T11:53:47.498982+0000","last_clean":"2024-03-24T11:53:47.498982+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:53:47.498982+0000","last_undegraded":"2024-03-24T11:53:47.498982+0000","last_fullsized":"2024-03-24T11:53:47.498982+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:19.389895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:19.389895+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:38.977195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618068+0000","last_change":"2024-03-24T11:52:15.134671+0000","last_active":"2024-03-24T11:53:46.618068+0000","last_peered":"2024-03-24T11:53:46.618068+0000","last_clean":"2024-03-24T11:53:46.618068+0000","last_became_active":"2024-03-24T11:52:15.134326+0000","last_became_peered":"2024-03-24T11:52:15.134326+0000","last_unstale":"2024-03-24T11:53:46.618068+0000","last_undegraded":"2024-03-24T11:53:46.618068+0000","last_fullsized":"2024-03-24T11:53:46.618068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:52:14.117087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166455+0000","last_change":"2024-03-24T11:53:34.756186+0000","last_active":"2024-03-24T11:53:47.166455+0000","last_peered":"2024-03-24T11:53:47.166455+0000","last_clean":"2024-03-24T11:53:47.166455+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:53:47.166455+0000","last_undegraded":"2024-03-24T11:53:47.166455+0000","last_fullsized":"2024-03-24T11:53:47.166455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:53:34.756054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:34.756054+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:59.815024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499690+0000","last_change":"2024-03-24T11:52:21.388549+0000","last_active":"2024-03-24T11:53:47.499690+0000","last_peered":"2024-03-24T11:53:47.499690+0000","last_clean":"2024-03-24T11:53:47.499690+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:53:47.499690+0000","last_undegraded":"2024-03-24T11:53:47.499690+0000","last_fullsized":"2024-03-24T11:53:47.499690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.618570+0000","last_change":"2024-03-24T11:52:19.387447+0000","last_active":"2024-03-24T11:53:46.618570+0000","last_peered":"2024-03-24T11:53:46.618570+0000","last_clean":"2024-03-24T11:53:46.618570+0000","last_became_active":"2024-03-24T11:52:19.387115+0000","last_became_peered":"2024-03-24T11:52:19.387115+0000","last_unstale":"2024-03-24T11:53:46.618570+0000","last_undegraded":"2024-03-24T11:53:46.618570+0000","last_fullsized":"2024-03-24T11:53:46.618570+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499368+0000","last_change":"2024-03-24T11:53:24.386409+0000","last_active":"2024-03-24T11:53:47.499368+0000","last_peered":"2024-03-24T11:53:47.499368+0000","last_clean":"2024-03-24T11:53:47.499368+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:53:47.499368+0000","last_undegraded":"2024-03-24T11:53:47.499368+0000","last_fullsized":"2024-03-24T11:53:47.499368+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:53:24.386208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:24.386208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:43.956194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":95,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.166803+0000","last_change":"2024-03-24T11:53:43.683275+0000","last_active":"2024-03-24T11:53:47.166803+0000","last_peered":"2024-03-24T11:53:47.166803+0000","last_clean":"2024-03-24T11:53:47.166803+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:53:47.166803+0000","last_undegraded":"2024-03-24T11:53:47.166803+0000","last_fullsized":"2024-03-24T11:53:47.166803+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-24T11:53:43.683151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:43.683151+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:01.427314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:49.544662+0000","last_change":"2024-03-24T11:53:49.544698+0000","last_active":"2024-03-24T11:53:49.544662+0000","last_peered":"2024-03-24T11:53:49.544662+0000","last_clean":"2024-03-24T11:53:49.544662+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:53:49.544662+0000","last_undegraded":"2024-03-24T11:53:49.544662+0000","last_fullsized":"2024-03-24T11:53:49.544662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:49.544578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:49.544578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:03.719476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.498892+0000","last_change":"2024-03-24T11:53:37.463221+0000","last_active":"2024-03-24T11:53:47.498892+0000","last_peered":"2024-03-24T11:53:47.498892+0000","last_clean":"2024-03-24T11:53:47.498892+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:53:47.498892+0000","last_undegraded":"2024-03-24T11:53:47.498892+0000","last_fullsized":"2024-03-24T11:53:47.498892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.463110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.463110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.363432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499471+0000","last_change":"2024-03-24T11:53:30.449547+0000","last_active":"2024-03-24T11:53:47.499471+0000","last_peered":"2024-03-24T11:53:47.499471+0000","last_clean":"2024-03-24T11:53:47.499471+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:53:47.499471+0000","last_undegraded":"2024-03-24T11:53:47.499471+0000","last_fullsized":"2024-03-24T11:53:47.499471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:30.449442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:30.449442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:36.454836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":98,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.619765+0000","last_change":"2024-03-24T11:53:45.332408+0000","last_active":"2024-03-24T11:53:46.619765+0000","last_peered":"2024-03-24T11:53:46.619765+0000","last_clean":"2024-03-24T11:53:46.619765+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:53:46.619765+0000","last_undegraded":"2024-03-24T11:53:46.619765+0000","last_fullsized":"2024-03-24T11:53:46.619765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:53:45.332300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.332300+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.325555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303326+0000","last_change":"2024-03-24T11:52:21.387355+0000","last_active":"2024-03-24T11:53:47.303326+0000","last_peered":"2024-03-24T11:53:47.303326+0000","last_clean":"2024-03-24T11:53:47.303326+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:53:47.303326+0000","last_undegraded":"2024-03-24T11:53:47.303326+0000","last_fullsized":"2024-03-24T11:53:47.303326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:52:20.361608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:48.688825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.498874+0000","last_change":"2024-03-24T11:52:19.384147+0000","last_active":"2024-03-24T11:53:47.498874+0000","last_peered":"2024-03-24T11:53:47.498874+0000","last_clean":"2024-03-24T11:53:47.498874+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:53:47.498874+0000","last_undegraded":"2024-03-24T11:53:47.498874+0000","last_fullsized":"2024-03-24T11:53:47.498874+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:52:18.347513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.617828+0000","last_change":"2024-03-24T11:53:37.995353+0000","last_active":"2024-03-24T11:53:46.617828+0000","last_peered":"2024-03-24T11:53:46.617828+0000","last_clean":"2024-03-24T11:53:46.617828+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:53:46.617828+0000","last_undegraded":"2024-03-24T11:53:46.617828+0000","last_fullsized":"2024-03-24T11:53:46.617828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.995261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:56.380355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.303797+0000","last_change":"2024-03-24T11:53:36.884179+0000","last_active":"2024-03-24T11:53:47.303797+0000","last_peered":"2024-03-24T11:53:47.303797+0000","last_clean":"2024-03-24T11:53:47.303797+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:53:47.303797+0000","last_undegraded":"2024-03-24T11:53:47.303797+0000","last_fullsized":"2024-03-24T11:53:47.303797+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:36.884086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.884086+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:52.784016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748400+0000","last_change":"2024-03-24T11:53:45.874634+0000","last_active":"2024-03-24T11:53:46.748400+0000","last_peered":"2024-03-24T11:53:46.748400+0000","last_clean":"2024-03-24T11:53:46.748400+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:53:46.748400+0000","last_undegraded":"2024-03-24T11:53:46.748400+0000","last_fullsized":"2024-03-24T11:53:46.748400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.874555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:00.944511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.499666+0000","last_change":"2024-03-24T11:53:25.429178+0000","last_active":"2024-03-24T11:53:47.499666+0000","last_peered":"2024-03-24T11:53:47.499666+0000","last_clean":"2024-03-24T11:53:47.499666+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:53:47.499666+0000","last_undegraded":"2024-03-24T11:53:47.499666+0000","last_fullsized":"2024-03-24T11:53:47.499666+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:25.429098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:25.429098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:46.655256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:46.748336+0000","last_change":"2024-03-24T11:53:37.846040+0000","last_active":"2024-03-24T11:53:46.748336+0000","last_peered":"2024-03-24T11:53:46.748336+0000","last_clean":"2024-03-24T11:53:46.748336+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:53:46.748336+0000","last_undegraded":"2024-03-24T11:53:46.748336+0000","last_fullsized":"2024-03-24T11:53:46.748336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:53:37.845952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:37.845952+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:04.539200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T11:53:47.331869+0000","last_change":"2024-03-24T11:53:31.671828+0000","last_active":"2024-03-24T11:53:47.331869+0000","last_peered":"2024-03-24T11:53:47.331869+0000","last_clean":"2024-03-24T11:53:47.331869+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:53:47.331869+0000","last_undegraded":"2024-03-24T11:53:47.331869+0000","last_fullsized":"2024-03-24T11:53:47.331869+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:31.671692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:31.671692+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:49.989770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418,"ondisk_log_size":418,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005079,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":969680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332126,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712572,"statfs":{"total":95995035648,"available":95961673728,"internally_reserved":0,"allocated":1433600,"data_stored":969498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":378709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32580,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712572,"statfs":{"total":95995035648,"available":95961673728,"internally_reserved":0,"allocated":1437696,"data_stored":968547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":847872,"data_stored":379129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":379559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920807,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":856064,"data_stored":378744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659170,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":847872,"data_stored":378856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:53:56.557 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:53:56.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump 2024-03-24T11:53:56.802 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:56 smithi064 bash[81084]: Writing manifest to image destination 2024-03-24T11:53:57.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:53:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:53:57.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:56 smithi193 ceph-mon[33593]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-24T11:53:57.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:56 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T11:53:57.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:56 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T11:53:57.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:56 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T11:53:57.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:56 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T11:53:57.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:56 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T11:53:57.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:56 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T11:53:57.281 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 podman[81084]: 2024-03-24 11:53:56.940000402 +0000 UTC m=+2.850454091 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:53:57.281 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 podman[81084]: 2024-03-24 11:53:57.063450122 +0000 UTC m=+2.973908438 container create c7cbb8fc5d4b7512ed65850bc9a27a13fa8eb6f14bc5295e83f4b92a06f20ba5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:53:57.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T11:53:57.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T11:53:57.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T11:53:57.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T11:53:57.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:56 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T11:53:57.401 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:53:57.532 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:53:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 podman[81084]: 2024-03-24 11:53:57.282050517 +0000 UTC m=+3.192504197 container init c7cbb8fc5d4b7512ed65850bc9a27a13fa8eb6f14bc5295e83f4b92a06f20ba5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 podman[81084]: 2024-03-24 11:53:57.287460735 +0000 UTC m=+3.197914405 container start c7cbb8fc5d4b7512ed65850bc9a27a13fa8eb6f14bc5295e83f4b92a06f20ba5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.292Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.292Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.292Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.292Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:53:57.533 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:53:57.534 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:53:57.534 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:53:57.534 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:53:57.534 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:53:57.534 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:53:57.534 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:53:57.535 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:53:57.536 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:53:57.537 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:53:57.537 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:53:57.537 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:53:57.537 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:53:57.537 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a[81263]: ts=2024-03-24T11:53:57.293Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:53:57.537 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 bash[81084]: c7cbb8fc5d4b7512ed65850bc9a27a13fa8eb6f14bc5295e83f4b92a06f20ba5 2024-03-24T11:53:57.537 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 11:53:57 smithi064 systemd[1]: Started Ceph node-exporter.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:53:58.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T11:53:58.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T11:53:58.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T11:53:58.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T11:53:58.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:58.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T11:53:58.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T11:53:58.258 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T11:53:58.258 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T11:53:58.258 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T11:53:58.258 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T11:53:58.258 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:58.259 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.259 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.259 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.259 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T11:53:58.259 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:57 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T11:53:58.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T11:53:58.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T11:53:58.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T11:53:58.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T11:53:58.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:58.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:53:58.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T11:53:58.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:57 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T11:53:58.586 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:53:58 smithi193 systemd[1]: Starting Ceph node-exporter.b for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:53:58.958 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:53:58 smithi193 bash[67213]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T11:53:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[30223]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T11:53:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T11:53:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[30223]: Deploying daemon node-exporter.b on smithi193 2024-03-24T11:53:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T11:53:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T11:53:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[30223]: 6.b scrub starts 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[30223]: 6.b scrub ok 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[42243]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[42243]: Deploying daemon node-exporter.b on smithi193 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[42243]: 6.b scrub starts 2024-03-24T11:53:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:58 smithi064 ceph-mon[42243]: 6.b scrub ok 2024-03-24T11:53:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:58 smithi193 ceph-mon[33593]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:58 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T11:53:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:58 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T11:53:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:58 smithi193 ceph-mon[33593]: Deploying daemon node-exporter.b on smithi193 2024-03-24T11:53:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:58 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T11:53:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:58 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T11:53:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:58 smithi193 ceph-mon[33593]: 6.b scrub starts 2024-03-24T11:53:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:58 smithi193 ceph-mon[33593]: 6.b scrub ok 2024-03-24T11:53:59.337 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:53:58 smithi193 bash[67213]: Getting image source signatures 2024-03-24T11:53:59.337 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:53:58 smithi193 bash[67213]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T11:53:59.337 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:53:58 smithi193 bash[67213]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T11:53:59.337 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:53:58 smithi193 bash[67213]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T11:53:59.362 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 95 2024-03-24T11:53:59.362 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T11:53:59.362 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-03-24T11:46:40.657065+0000 2024-03-24T11:53:59.362 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2024-03-24T11:53:54.938657+0000 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:crush_version 18 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:full_ratio 0.95 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:backfillfull_ratio 0.9 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:nearfull_ratio 0.85 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:require_min_compat_client luminous 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:min_compat_client luminous 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:require_osd_release squid 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:stretch_mode_enabled false 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:pool 3 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 71 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-24T11:53:59.363 INFO:teuthology.orchestra.run.smithi064.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:max_osd 8 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6801/2675992571 v1:172.21.15.64:6802/2675992571 exists,up bc14a64f-9e6c-41f1-901e-c49fabffb53d 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6805/841667271 v1:172.21.15.64:6806/841667271 exists,up 74587332-f0b2-4905-9888-7bdb604cc0e9 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 up in weight 1 up_from 18 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6809/435147390 v1:172.21.15.64:6810/435147390 exists,up a0c0ed85-9f33-4198-a1f2-447eb17c364b 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 up in weight 1 up_from 27 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6813/3355318038 v1:172.21.15.64:6814/3355318038 exists,up 1396f66c-3e0c-4a01-b0a7-6f763989a479 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 up in weight 1 up_from 33 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6800/3851339176 v1:172.21.15.193:6801/3851339176 exists,up 50d65d8b-3e90-46a4-b19d-50c3015af40b 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 up in weight 1 up_from 41 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6804/1625306337 v1:172.21.15.193:6805/1625306337 exists,up 50c5d50e-7e68-487c-9f68-1abb8c927ff6 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 up in weight 1 up_from 51 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6808/2408237278 v1:172.21.15.193:6809/2408237278 exists,up 7e74fe39-5707-4c90-ad38-1083cd7a6996 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 up in weight 1 up_from 61 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6812/2649989609 v1:172.21.15.193:6813/2649989609 exists,up e568df64-cc9a-43f8-a432-8628d1da1015 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.b [7,2] 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.4 [1,5] 2024-03-24T11:53:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.9 [1,2] 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.1d [1,2] 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/3538503390 expires 2024-03-25T11:52:55.117233+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/1277626303 expires 2024-03-25T11:52:55.117233+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3407451558 expires 2024-03-25T11:52:55.117233+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3753895773 expires 2024-03-25T11:52:55.117233+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/842059688 expires 2024-03-25T11:47:36.035658+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/1610565583 expires 2024-03-25T11:47:36.035658+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/2730170969 expires 2024-03-25T11:47:08.661692+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3257265142 expires 2024-03-25T11:47:08.661692+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/1459540658 expires 2024-03-25T11:52:55.117233+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/629947599 expires 2024-03-25T11:47:08.661692+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/520335082 expires 2024-03-25T11:47:08.661692+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3351953651 expires 2024-03-25T11:52:55.117233+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/4027721305 expires 2024-03-25T11:47:36.035658+0000 2024-03-24T11:53:59.365 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/4028269665 expires 2024-03-25T11:47:36.035658+0000 2024-03-24T11:54:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T11:54:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T11:54:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T11:54:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T11:54:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[30223]: 5.c scrub starts 2024-03-24T11:54:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[30223]: 5.c scrub ok 2024-03-24T11:54:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4043370003' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:54:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T11:54:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T11:54:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T11:54:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T11:54:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[42243]: 5.c scrub starts 2024-03-24T11:54:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[42243]: 5.c scrub ok 2024-03-24T11:54:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:53:59 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4043370003' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:54:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:59 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T11:54:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:59 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T11:54:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:59 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T11:54:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:59 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T11:54:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:59 smithi193 ceph-mon[33593]: 5.c scrub starts 2024-03-24T11:54:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:59 smithi193 ceph-mon[33593]: 5.c scrub ok 2024-03-24T11:54:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:53:59 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4043370003' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:54:00.375 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6801/2675992571 v1:172.21.15.64:6802/2675992571 exists,up bc14a64f-9e6c-41f1-901e-c49fabffb53d', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6805/841667271 v1:172.21.15.64:6806/841667271 exists,up 74587332-f0b2-4905-9888-7bdb604cc0e9', 'osd.2 up in weight 1 up_from 18 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6809/435147390 v1:172.21.15.64:6810/435147390 exists,up a0c0ed85-9f33-4198-a1f2-447eb17c364b', 'osd.3 up in weight 1 up_from 27 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6813/3355318038 v1:172.21.15.64:6814/3355318038 exists,up 1396f66c-3e0c-4a01-b0a7-6f763989a479', 'osd.4 up in weight 1 up_from 33 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6800/3851339176 v1:172.21.15.193:6801/3851339176 exists,up 50d65d8b-3e90-46a4-b19d-50c3015af40b', 'osd.5 up in weight 1 up_from 41 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6804/1625306337 v1:172.21.15.193:6805/1625306337 exists,up 50c5d50e-7e68-487c-9f68-1abb8c927ff6', 'osd.6 up in weight 1 up_from 51 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6808/2408237278 v1:172.21.15.193:6809/2408237278 exists,up 7e74fe39-5707-4c90-ad38-1083cd7a6996', 'osd.7 up in weight 1 up_from 61 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6812/2649989609 v1:172.21.15.193:6813/2649989609 exists,up e568df64-cc9a-43f8-a432-8628d1da1015'] 2024-03-24T11:54:00.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon mon.a config show 2024-03-24T11:54:00.633 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:00.964 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:00 smithi193 bash[67213]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T11:54:00.964 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:00 smithi193 bash[67213]: Writing manifest to image destination 2024-03-24T11:54:01.232 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 podman[67213]: 2024-03-24 11:54:01.021206603 +0000 UTC m=+2.483739852 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:54:01.232 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 podman[67213]: 2024-03-24 11:54:01.094810359 +0000 UTC m=+2.557343597 container create c4156adfae44ce2bba092e3ad6a3e60118733451d274fe444fbd208dabeff47a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:54:01.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:00 smithi193 ceph-mon[33593]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:01.233 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:00 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T11:54:01.233 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:00 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T11:54:01.233 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:00 smithi193 ceph-mon[33593]: 2.16 scrub starts 2024-03-24T11:54:01.233 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:00 smithi193 ceph-mon[33593]: 2.16 scrub ok 2024-03-24T11:54:01.233 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:00 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T11:54:01.233 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:00 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T11:54:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[42243]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T11:54:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T11:54:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[42243]: 2.16 scrub starts 2024-03-24T11:54:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[42243]: 2.16 scrub ok 2024-03-24T11:54:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T11:54:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T11:54:01.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[30223]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:01.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T11:54:01.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T11:54:01.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[30223]: 2.16 scrub starts 2024-03-24T11:54:01.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[30223]: 2.16 scrub ok 2024-03-24T11:54:01.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T11:54:01.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:00 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T11:54:01.586 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 podman[67213]: 2024-03-24 11:54:01.322173676 +0000 UTC m=+2.784706925 container init c4156adfae44ce2bba092e3ad6a3e60118733451d274fe444fbd208dabeff47a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 podman[67213]: 2024-03-24 11:54:01.32764869 +0000 UTC m=+2.790181928 container start c4156adfae44ce2bba092e3ad6a3e60118733451d274fe444fbd208dabeff47a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.331Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.331Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.332Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.332Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.332Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.332Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:54:01.587 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:54:01.588 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:54:01.589 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:54:01.590 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:54:01.591 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:54:01.591 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:54:01.591 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:54:01.591 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:54:01.591 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:54:01.591 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b[67262]: ts=2024-03-24T11:54:01.333Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:54:01.591 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 bash[67213]: c4156adfae44ce2bba092e3ad6a3e60118733451d274fe444fbd208dabeff47a 2024-03-24T11:54:01.591 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 11:54:01 smithi193 systemd[1]: Started Ceph node-exporter.b for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mon.a", 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster": "ceph", 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "admin_socket_mode": "", 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_client_required": "cephx, none", 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_cluster_required": "cephx", 2024-03-24T11:54:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_debug": "true", 2024-03-24T11:54:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-24T11:54:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-24T11:54:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_service_required": "cephx", 2024-03-24T11:54:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-24T11:54:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_supported": "", 2024-03-24T11:54:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio": "true", 2024-03-24T11:54:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-24T11:54:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_poll_ms": "250", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_reap_max": "16", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_async_discard": "false", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_block_size": "4096", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio": "false", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_inflight_ios": "false", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_enable_discard": "false", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_flock_retry": "3", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_inject_crash": "0", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-24T11:54:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring": "false", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring_hipri": "false", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_type": "", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_alloc_size": "1048576", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_allocator": "hybrid", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_buffered_io": "true", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_for_zeros": "false", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-24T11:54:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_compact_log_sync": "false", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_max_log_runway": "4194304", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_max_prefetch": "1048576", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_min_flush_size": "524288", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_min_log_runway": "1048576", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_replay_recovery": "false", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_sync_write": "false", 2024-03-24T11:54:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_allocation_from_file": "true", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_allocator": "hybrid", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_create": "true", 2024-03-24T11:54:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_create": "false", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_path": "", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_size": "0", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_path": "", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_preallocate_file": "false", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_size": "107374182400", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_create": "false", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_path": "", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_size": "100663296", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs": "true", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-24T11:54:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_autotune": "true", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size": "0", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-24T11:54:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_type": "2q", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_clone_cow": "true", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_mode": "none", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_csum_type": "crc32c", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_freelist": "false", 2024-03-24T11:54:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_small_allocations": "0", 2024-03-24T11:54:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_default_buffered_read": "true", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_default_buffered_write": "false", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fail_eio": "false", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-24T11:54:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mount": "false", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_umount": "false", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-24T11:54:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_ignore_data_csum": "false", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_op_age": "5.000000", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_alloc_size": "0", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size": "0", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_deferred_txc": "32", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size": "0", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-24T11:54:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_nid_prealloc": "1024", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_retry_disk_reads": "3", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_cf": "true", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-24T11:54:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_mem": "512", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_tracing": "false", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-24T11:54:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_zero_block_detection": "false", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-24T11:54:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_cluster_require_signatures": "false", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_cluster_require_version": "2", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_require_signatures": "false", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_require_version": "2", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_service_require_signatures": "false", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_service_require_version": "2", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_sign_messages": "true", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "chdir": "", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_acl_type": "", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_asio_thread_count": "2", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_cache_mid": "0.750000", 2024-03-24T11:54:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_cache_size": "16384", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_caps_release_delay": "5", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_check_pool_perm": "true", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_force_sync_read": "false", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_getattr_caps": "false", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_inject_tick_delay": "0", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_die_on_failed_remount": "false", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_dirsize_rbytes": "true", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_force_lazyio": "false", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_fs": "", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_inject_release_failure": "false", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_max_inline_size": "4096", 2024-03-24T11:54:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mds_namespace": "", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_metadata": "", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_gid": "-1", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_timeout": "300", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_uid": "-1", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mountpoint": "/", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_notify_timeout": "10", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc": "true", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_dirty": "104857600", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_objects": "1000", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_size": "209715200", 2024-03-24T11:54:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_target_dirty": "8388608", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_permissions": "true", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_quota": "true", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_quota_df": "true", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_max_bytes": "0", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_max_periods": "4", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_min": "131072", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_reconnect_stale": "false", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_shutdown_timeout": "30", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_snapdir": ".snap", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_tick_interval": "1", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_trace": "", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_try_dentry_invalidate": "false", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_use_faked_inos": "false", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_use_random_mds": "false", 2024-03-24T11:54:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog": "false", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog_port": "12201", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_monitors": "default=true", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog": "false", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog_level": "info", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "-", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_network": "", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_network_interface": "", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_isal": "false", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_level": "5", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_winsize": "-15", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zstd_level": "1", 2024-03-24T11:54:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_alien_op_num_threads": "6", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_seastar_cpu_cores": "", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location_hook": "", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location_hook_timeout": "10", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "daemonize": "false", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "daos_pool": "tank", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-24T11:54:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_alienstore": "0/5", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_allow_any_pool_priority": "false", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asok": "1/5", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asok_assert_abort": "false", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asserts_on_shutdown": "true", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_auth": "1/5", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bdev": "1/3", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bluefs": "1/5", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bluestore": "1/5", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_buffer": "0/1", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ceph_exporter": "1/5", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cephfs_mirror": "0/5", 2024-03-24T11:54:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cephsqlite": "0/5", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_client": "0/5", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_compressor": "1/5", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_context": "0/1", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crimson_interrupt": "0/5", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crush": "1/1", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crypto": "1/5", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cyanstore": "0/5", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_deliberately_leak_memory": "false", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_disable_randomized_ping": "false", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_dpdk": "1/5", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_eventtrace": "1/5", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_filer": "0/1", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_filestore": "1/3", 2024-03-24T11:54:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_finisher": "1/1", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_fuse": "1/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_heartbeat_testing_span": "0", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_heartbeatmap": "1/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_javaclient": "1/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_journal": "1/3", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_journaler": "0/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_kstore": "1/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_lockdep": "0/1", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mclock": "1/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds": "1/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_balancer": "1/5", 2024-03-24T11:54:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_locker": "1/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_log": "1/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_log_expire": "1/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_migrator": "1/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_memstore": "1/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mgr": "2/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mgrc": "1/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mon": "20/20", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_monc": "0/10", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ms": "1/1", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_none": "0/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objclass": "0/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objectcacher": "0/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objecter": "0/1", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_optracker": "0/5", 2024-03-24T11:54:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_osd": "1/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_paxos": "20/20", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_perfcounter": "1/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_prioritycache": "1/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rados": "0/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd": "0/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_mirror": "0/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_pwl": "0/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_replay": "0/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_refs": "0/0", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_reserver": "1/1", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw": "1/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_access": "1/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_datacache": "1/5", 2024-03-24T11:54:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_dbstore": "1/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_flight": "1/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_sync": "1/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rocksdb": "4/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_backref": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_cache": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_cleaner": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_device": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_epm": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_journal": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_lba": "0/5", 2024-03-24T11:54:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_odata": "0/5", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_omap": "0/5", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_onode": "0/5", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_t": "0/5", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_tm": "0/5", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_striper": "0/1", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_test": "0/5", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_throttle": "1/1", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_timer": "0/1", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_tp": "0/5", 2024-03-24T11:54:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_trace": "1/5", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "device_failure_prediction_mode": "none", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_graylog": "false", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_journald": "false", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_stderr": "true", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_syslog": "false", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event_tracing": "false", 2024-03-24T11:54:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_addr": "0.0.0.0", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_http_port": "9926", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_prio_limit": "5", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_sort_metrics": "true", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_stats_period": "5", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "fake_statfs_for_testing": "0", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "fatal_signal_handlers": "true", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "filer_max_purge_ops": "10", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "filer_max_truncate_ops": "128", 2024-03-24T11:54:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_apply_finisher_threads": "1", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_blackhole": "false", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_btrfs_clone_range": "true", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_btrfs_snap": "true", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_caller_concurrency": "10", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_collect_device_partition_information": "true", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_commit_timeout": "600.000000", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_inject_read_err": "false", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_omap_check": "false", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_verify_split": "false", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_dump_file": "", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-24T11:54:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fadvise": "true", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fail_eio": "true", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fd_cache_shards": "16", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fd_cache_size": "128", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fiemap": "false", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fiemap_threshold": "4096", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_inject_stall": "0", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_parallel": "false", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_trailing": "false", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_writeahead": "false", 2024-03-24T11:54:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_kill_at": "0", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs": "0", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size": "0", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-24T11:54:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_merge_threshold": "-10", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_odsync_write": "false", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_backend": "rocksdb", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_backend_path": "", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_thread_timeout": "60", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_threads": "2", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_punch_hole": "false", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-24T11:54:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_ops": "50", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_seek_data_hole": "false", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_sloppy_crc": "false", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_splice": "false", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_split_multiple": "2", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_split_rand_factor": "20", 2024-03-24T11:54:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_update_to": "1000", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_enable": "true", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-24T11:54:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_xfs_extsize": "false", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_zfs_snap": "false", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fio_dir": "/tmp/fio", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_allow_other": "true", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_atomic_o_trunc": "true", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_big_writes": "true", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_debug": "false", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_default_permissions": "false", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_disable_pagecache": "false", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_max_write": "0", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_multithreaded": "true", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_require_active_mds": "true", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_set_user_groups": "true", 2024-03-24T11:54:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_move": "true", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_read": "true", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_write": "true", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_use_invalidate_cb": "true", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "gss_target_name": "ceph", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_file": "", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_inject_failure": "0", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_interval": "5", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "host": "smithi064", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-24T11:54:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "inject_early_sigterm": "false", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "jaeger_agent_port": "6799", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "jaeger_tracing_enable": "false", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_aio": "true", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_align_min_size": "65536", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_block_align": "true", 2024-03-24T11:54:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_block_size": "4096", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_dio": "true", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_discard": "false", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_force_aio": "false", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_ignore_corruption": "false", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_max_write_bytes": "10485760", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_max_write_entries": "100", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_replay_from": "0", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_write_header_frequency": "0", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_zero_on_create": "false", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_prefetch_periods": "10", 2024-03-24T11:54:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_prezero_periods": "5", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_write_head_interval": "15", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "key": "", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "keyfile": "", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_backend": "rocksdb", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_default_stripe_size": "65536", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_fsck_on_mount": "false", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_max_bytes": "67108864", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_max_ops": "512", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_nid_prealloc": "1024", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_onode_map_size": "1024", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-24T11:54:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_sync_submit_transaction": "false", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_sync_transaction": "false", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "librados_thread_count": "2", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "lockdep": "false", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "lockdep_force_backtrace": "false", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_coarse_timestamps": "true", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_flush_on_exit": "false", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_graylog_host": "127.0.0.1", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_graylog_port": "12201", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_max_new": "1000", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_max_recent": "10000", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_stderr_prefix": "", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_stop_at_utilization": "0.970000", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": "true", 2024-03-24T11:54:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_graylog": "false", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_journald": "false", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_stderr": "false", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_syslog": "false", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "max_rotating_auth_attempts": "10", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_action_on_write_error": "1", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_alternate_name_max": "8192", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_asio_thread_count": "2", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_export_pin": "true", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_dirs": "true", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_interval": "5", 2024-03-24T11:54:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_interval": "10", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_max": "-1", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_max_until": "-1", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_merge_size": "50", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_midchunk": "0.300000", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_min_start": "0.200000", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_minchunk": "0.001000", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_mode": "0", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_need_max": "1.200000", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_need_min": "0.800000", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_overload_epochs": "2", 2024-03-24T11:54:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_bits": "3", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_size": "10000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_target_decay": "10.000000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_grace": "15.000000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_interval": "4.000000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_mid": "0.700000", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_release_free_interval": "10", 2024-03-24T11:54:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_reservation": "0.050000", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_interval": "1", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_threshold": "262144", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_prealloc_inos": "1000", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_connect_bootstrapping": "false", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_damage_table_max_entries": "10000", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_auth_pins": "false", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_frag": "false", 2024-03-24T11:54:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_scatterstat": "false", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_subtrees": "false", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_decay_halflife": "5.000000", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_default_dir_hash": "2", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_defer_session_stale": "true", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_deny_all_reconnect": "false", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_keys_per_op": "16384", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_max_commit_size": "10", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_max_entries": "0", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_prefetch": "true", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_on_map": "false", 2024-03-24T11:54:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_early_reply": "true", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_enable_op_tracker": "true", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_enforce_unique_name": "true", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_random": "true", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-24T11:54:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_health_summarize_threshold": "10", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_health_dummy": "false", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_migrator_session_race": "false", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_join_fs": "", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_journal_format": "1", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_create_at": "0", 2024-03-24T11:54:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_export_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_import_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_expire_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_replay_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_link_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_mdstable_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_openc_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_rename_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_shutdown_at": "0", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_event_large_threshold": "524288", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_events_per_segment": "1024", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-24T11:54:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_max_events": "-1", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_max_segments": "128", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_pause": "false", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_segment_size": "0", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_threshold": "128", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_warn_factor": "2.000000", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_caps_per_client": "1048576", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_completed_flushes": "100000", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_completed_requests": "100000", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_export_size": "20971520", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_file_recover": "32", 2024-03-24T11:54:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_files": "64", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_ops": "8192", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_snaps_per_dir": "100", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_metrics_update_interval": "2", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_min_caps_per_client": "100", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_min_caps_working_set": "10000", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_numa_node": "-1", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_complaint_time": "30.000000", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_duration": "600", 2024-03-24T11:54:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_size": "20", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_slow_op_size": "20", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_log_threshold": "5", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_ping_grace": "15", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_ping_interval": "5", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_caps": "30000", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_warning_threshold": "262144", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-24T11:54:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_replay_interval": "1.000000", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_root_ino_gid": "0", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_root_ino_uid": "0", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-24T11:54:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_shutdown_check": "0", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_skip_ino": "0", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_max_uid": "4294967294", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_min_uid": "0", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_rstat": "false", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_standby_replay_damaged": "false", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_symlink_recovery": "true", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_thrash_exports": "0", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_thrash_fragments": "0", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_tick_interval": "5.000000", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_valgrind_exit": "false", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_verify_backtrace": "1", 2024-03-24T11:54:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_verify_scatter": "false", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_wipe_sessions": "false", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool_debug": "false", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_device_bytes": "1073741824", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_page_set": "false", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_page_size": "65536", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_bytes": "134217728", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_messages": "512", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-24T11:54:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_disabled_modules": "", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_max_pg_num_change": "128", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mds_bytes": "134217728", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mds_messages": "128", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mon_bytes": "134217728", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mon_messages": "128", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_osd_bytes": "536870912", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_osd_messages": "8192", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_pool": "true", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_standby_modules": "true", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_stats_period": "5", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_stats_threshold": "5", 2024-03-24T11:54:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_tick_period": "2", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_allow_pool_delete": "true", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_allow_pool_size_one": "false", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_auth_validate_all_caps": "true", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_bytes": "104857600", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_directed_command_retry": "2", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-24T11:54:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_parallel": "3", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_log_interval": "1.000000", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_ping_interval": "10.000000", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_target_rank": "-1", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_file_level": "debug", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_file": "true", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-24T11:54:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_journald": "true", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_bootstrap": "false", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_start": "false", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_trim": "true", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cpu_threads": "4", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_crush_min_required_version": "hammer", 2024-03-24T11:54:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_daemon_bytes": "419430400", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_avail_crit": "5", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_avail_warn": "5", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_size_warn": "16106127360", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_json": "false", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_transactions": "false", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_extra_checks": "false", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_reef": "false", 2024-03-24T11:54:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_squid": "false", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_down_mkfs_grace": "60", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_election_default_strategy": "1", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_election_timeout": "5.000000", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ping_divisor": "2", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_enable_op_tracker": "true", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_fake_pool_delete": "false", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_force_quorum_join": "false", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_globalid_prealloc": "10000", 2024-03-24T11:54:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_detail_to_clog": "true", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_log_update_period": "5", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_max_detail": "50", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog": "true", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog_interval": "600", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_host": "172.21.15.64:6789/0", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_host_override": "", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_initial_members": "", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease": "5.000000", 2024-03-24T11:54:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_full_interval": "50", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_max": "10000", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_max_summary": "50", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_log_epochs": "500", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_osd": "10000", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_pg_per_osd": "10000", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_pool_pg_num": "65536", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-24T11:54:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_force_trim_to": "0", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_skip_sanity": "false", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_memory_autotune": "true", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_memory_target": "2147483648", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_beacon_grace": "30", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_digest_period": "5", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_inactive_grace": "60", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_min_osdmap_epochs": "50", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_complaint_time": "30", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_duration": "600", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_size": "20", 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_slow_op_size": "20", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_log_threshold": "5", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_in": "false", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_cache_size": "500", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-24T11:54:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_down_out_interval": "600", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_force_trim_to": "0", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_down_reporters": "2", 2024-03-24T11:54:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_report_timeout": "900", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-24T11:54:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_stuck_threshold": "60", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_probe_timeout": "2.000000", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_max_change": "0.050000", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_max_osds": "4", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-24T11:54:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_interval": "86400", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_max_keys": "100", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_timeout": "300", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_session_timeout": "300", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_smart_report_timeout": "5", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stat_smooth_intervals": "6", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_pool_min_size": "2", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_pool_size": "4", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_debug": "false", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_provider_kill_at": "0", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_requester_kill_at": "0", 2024-03-24T11:54:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_timeout": "60.000000", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_target_pg_per_osd": "100", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_tick_interval": "5", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_timecheck_interval": "300.000000", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_use_min_delay_socket": "false", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_older_version_delay": "604800", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_misplaced": "false", 2024-03-24T11:54:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_older_version": "true", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": "", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-24T11:54:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_tracing_enabled": "false", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_op_threads": "3", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_cm": "false", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_device_name": "", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_dscp": "96", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_local_gid": "", 2024-03-24T11:54:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_port_num": "1", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_sl": "3", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_support_srq": "true", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_type": "ib", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_reap_threshold": "5", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_before_connect": "false", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_ipv4": "true", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_ipv6": "false", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_msgr1": "true", 2024-03-24T11:54:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_msgr2": "false", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_port_max": "7568", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_port_min": "6800", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_retry_count": "3", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_retry_delay": "5", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_client": "false", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mds": "false", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mgr": "false", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mon": "false", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_osd": "false", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_client_mode": "crc secure", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_cluster_mode": "crc secure", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_cluster_type": "", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_compress_secure": "false", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_connection_idle_timeout": "900", 2024-03-24T11:54:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_connection_ready_timeout": "10", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_crc_data": "true", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_crc_header": "true", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_bad_msg": "false", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_bug": "true", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_old_message": "true", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_skipped_message": "false", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_coremask": "0xF", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_enable_tso": "true", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hugepages": "", 2024-03-24T11:54:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_lro": "true", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_memory_channel": "4", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_pmd": "", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_port_id": "0", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dump_on_send": "false", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_initial_backoff": "0.200000", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_max": "1.000000", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-24T11:54:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_type": "", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_network_congestion": "0", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_socket_failures": "0", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_learn_addr_from_peer": "true", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_max_accept_failures": "4", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_max_backoff": "15.000000", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_client_mode": "secure crc", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_service_mode": "secure crc", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compress_min_size": "1024", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compress_mode": "none", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_pq_min_cost": "65536", 2024-03-24T11:54:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_public_type": "", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_service_mode": "crc secure", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_listen_backlog": "512", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_nodelay": "true", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_rcvbuf": "0", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_type": "async+posix", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "no_config_file": "false", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_completion_locks_per_session": "32", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inflight_ops": "1024", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_tick_interval": "5.000000", 2024-03-24T11:54:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_timeout": "10.000000", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "objectstore_blackhole": "false", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "openssl_engine_opts": "", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_delay_time": "5.000000", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_hist_halflife": "1000", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_max_low_ops": "2", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_max_ops": "4", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_slop": "0.020000", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_asio_thread_count": "2", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_async_recovery_min_cost": "100", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-24T11:54:02.469 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_scan_max": "512", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_scan_min": "64", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_degraded": "false", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_peering": "false", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_unfound": "true", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_beacon_report_interval": "300", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_duration": "30", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_max_block_size": "67108864", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_blkin_trace_all": "false", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-24T11:54:02.470 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_check_for_log_corruption": "false", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_update_on_start": "true", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_message_cap": "256", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_message_size_cap": "524288000", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_op_priority": "63", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_watch_timeout": "30", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_command_max_records": "256", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_compact_on_start": "false", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-24T11:54:02.471 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_update_on_start": "true", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_update_weight_set": "true", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_feed_pullee": "-1", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-24T11:54:02.472 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_misdirected_ops": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_no_acting_change": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_no_purge_strays": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_op_order": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_shutdown": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_trim_objects": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_snaps": "false", 2024-03-24T11:54:02.473 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_keys": "1024", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_stride": "524288", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_default_notify_timeout": "30", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep": "0.000000", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_discard_disconnected_ops": "true", 2024-03-24T11:54:02.474 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_enable_op_tracker": "true", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_info": "true", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown": "true", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_function_tracing": "false", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_grace": "20", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_interval": "6", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-24T11:54:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_peers": "10", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_size": "2000", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_stale": "600", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_max_size": "100000", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_min_size": "1000", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal_size": "5120", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_kill_backfill_at": "0", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-24T11:54:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_cache_size": "50", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_dedup": "true", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_message_max": "40", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_share_max_epochs": "40", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_attr_name_len": "100", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_attr_size": "0", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_backfills": "1", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_markdown_count": "5", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_markdown_period": "600", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_name_len": "2048", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_namespace_len": "256", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_size": "134217728", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-24T11:54:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_blocked_by": "16", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_log_entries": "10000", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pgls": "1024", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_push_cost": "8388608", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_push_objects": "10", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_scrubs": "3", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_trimming_pgs": "2", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_write_op_reply_len": "64", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_write_size": "90", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-24T11:54:02.478 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_profile": "balanced", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-24T11:54:02.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_base": "805306368", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_cache_min": "134217728", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target": "4294967296", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target_autotune": "false", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_min_pg_log_entries": "250", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_min_recovery_priority": "0", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_report_interval": "5", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_num_cache_shards": "32", 2024-03-24T11:54:02.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_num_op_tracker_shard": "32", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_auto_affinity": "true", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_node": "-1", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_prefer_iface": "true", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objecter_finishers": "1", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore": "bluestore", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_fuse": "false", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_tracing": "false", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_complaint_time": "30.000000", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_duration": "600", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_size": "20", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_slow_op_size": "20", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-24T11:54:02.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_log_threshold": "5", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards": "0", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards_hdd": "5", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards_ssd": "8", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_pq_min_cost": "65536", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_queue_cut_off": "high", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_thread_timeout": "15", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_open_classes_on_start": "true", 2024-03-24T11:54:02.482 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_os_flags": "0", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_peering_op_priority": "255", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_delete_cost": "1048576", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_delete_priority": "5", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_trim_max": "10000", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_trim_min": "100", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-24T11:54:02.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_crimson": "false", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flags": "0", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-24T11:54:02.484 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_min_size": "0", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pg_num": "32", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pgp_num": "0", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_read_ratio": "70", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_size": "3", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_type": "replicated", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_push_per_object_cost": "1000", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recover_clone_overlap": "true", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_cost": "20971520", 2024-03-24T11:54:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active": "0", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_single_start": "1", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_op_priority": "3", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_priority": "5", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep": "0.000000", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-24T11:54:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_repair_during_recovery": "false", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_requested_scrub_priority": "120", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_auto_repair": "false", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_begin_hour": "0", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_begin_week_day": "0", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_chunk_max": "25", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_chunk_min": "5", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_cost": "52428800", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_during_recovery": "false", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_end_hour": "0", 2024-03-24T11:54:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_end_week_day": "0", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_event_cost": "4096", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_invalid_stats": "true", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_max_preemptions": "5", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_priority": "5", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_sleep": "0.000000", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-24T11:54:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_skip_data_digest": "false", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_smart_report_timeout": "5", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_cost": "1048576", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_priority": "5", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_target_transaction_size": "30", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-24T11:54:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tracing": "false", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_use_stale_snap": "false", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osdc_blkin_trace_all": "false", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_kill_at": "0", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_max_join_drift": "10", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_min": "500", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_min_wait": "0.050000", 2024-03-24T11:54:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_propose_interval": "1.000000", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_max": "500", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_min": "10", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_stash_full_interval": "25", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_trim_max": "500", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_trim_min": "250", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "perf": "true", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "pid_file": "", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "-", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrv": "", 2024-03-24T11:54:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "public_bind_addr": "-", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "public_network": "", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "public_network_interface": "", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_enabled": "false", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_session_max_number": "256", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_mon_op_timeout": "0", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_osd_op_timeout": "0", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_tracing": "false", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_atime_update_interval": "60", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_balance_parent_reads": "false", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_balance_snap_reads": "false", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blkin_trace_all": "false", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-24T11:54:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache": "true", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_policy": "writearound", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_size": "33554432", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_clone_copy_on_read": "false", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_compression_hint": "none", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_concurrent_management_ops": "10", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-24T11:54:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_clone_format": "auto", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_data_pool": "", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_features": "61", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_format": "2", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_map_options": "", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_order": "22", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_pool": "rbd", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_stripe_count": "0", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_stripe_unit": "0", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_enable_alloc_hint": "true", 2024-03-24T11:54:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_io_scheduler": "simple", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_order": "24", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_pool": "", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_splay_width": "4", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_localize_parent_reads": "false", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_localize_snap_reads": "false", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-24T11:54:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-24T11:54:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mtime_update_interval": "60", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_non_blocking_aio": "true", 2024-03-24T11:54:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_op_thread_timeout": "60", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_op_threads": "1", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_parent_cache_enabled": "false", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_plugins": "", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_burst": "0", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_limit": "0", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_exclude_ops": "0", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_burst": "0", 2024-03-24T11:54:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-24T11:54:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_limit": "0", 2024-03-24T11:54:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-24T11:54:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-24T11:54:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-24T11:54:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_read_from_replica_policy": "default", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-24T11:54:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_skip_partial_discard": "true", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_tracing": "false", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_validate_names": "true", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_validate_pool": "true", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "restapi_base_url": "", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "restapi_log_level": "", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_acl_grants_max_num": "100", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_admin_entry": "admin", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_backend_store": "rados", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_barbican_url": "", 2024-03-24T11:54:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_beast_enable_async": "true", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_counters_cache": "false", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_enabled": "true", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_expiry_interval": "900", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_lru_size": "10000", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_config_store": "rados", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_content_length_compat": "false", 2024-03-24T11:54:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_copy_obj_progress": "true", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cors_rules_max_num": "100", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cross_domain_policy": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_addr": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_password": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_username": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_require_ssl": "true", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-24T11:54:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_addr": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_auth": "token", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_namespace": "", 2024-03-24T11:54:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_prefix": "", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_token_file": "", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_buffersize": "524288", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_low_speed_time": "300", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-24T11:54:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_port": "6379", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_changes_size": "1000", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_num_shards": "128", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_window": "30", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-24T11:54:02.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-24T11:54:02.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dns_name": "", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dns_s3website_name": "", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dynamic_resharding": "true", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_gc_threads": "true", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_lc_threads": "true", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_ops_log": "false", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_quota_threads": "true", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_static_website": "false", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_usage_log": "false", 2024-03-24T11:54:02.508 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enforce_swift_acls": "true", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_exit_timeout_secs": "120", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_expose_bucket": "false", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_extended_http_attrs": "", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_filter": "none", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_frontends": "beast port=7480", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_deferred": "50", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_objs": "32", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_processor_period": "3600", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-24T11:54:02.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_init_timeout": "300", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_connection_idle": "30", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_domain": "", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_password": "", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_password_path": "", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_project": "", 2024-03-24T11:54:02.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_tenant": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_token": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_token_path": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_user": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_api_version": "2", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_domain": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_password": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_project": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_user": "", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-24T11:54:02.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_url": "", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_debug_interval": "-1", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_lock_max_time": "90", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_objs": "32", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_rules": "1000", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_worker": "3", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_thread_delay": "0", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_dnattr": "uid", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_searchfilter": "", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-24T11:54:02.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_http_headers": "", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_object_name_utc": "false", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attr_name_len": "0", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attr_size": "0", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_chunk_size": "4194304", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-24T11:54:02.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_listing_results": "1000", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_notify_retries": "10", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_put_param_size": "1048576", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_put_size": "5368709120", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_slo_entries": "1000", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_md_log_max_shards": "64", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_mp_lock_max_time": "600", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-24T11:54:02.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_max_gc": "300", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_num_async_rados_threads": "32", 2024-03-24T11:54:02.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_num_control_oids": "8", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_numa_node": "-1", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_chunk_size": "100", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_gc_interval": "600", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_thread_timeout": "600", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_tracing": "false", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_token": "", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_url": "", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_verify_ssl": "true", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-24T11:54:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_rados": "false", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_socket_path": "", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_push_interval": "2.000000", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_lanes": "3", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_partitions": "3", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-24T11:54:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_print_continue": "true", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_tracing": "false", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm": "", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm_id": "", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_region": "", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-24T11:54:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_request_uri": "", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_batch_size": "64", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_max_aio": "128", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_num_logs": "16", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_thread_interval": "600", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_resolve_cname": "false", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_run_sync_thread": "true", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-24T11:54:02.519 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_scheduler_type": "throttler", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_script_uri": "", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_service_provider_name": "", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_client_id": "", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_client_secret": "", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_entry": "sts", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_key": "sts", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_min_session_duration": "900", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_token_introspection_url": "", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_account_in_url": "false", 2024-03-24T11:54:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_auth_entry": "auth", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_auth_url": "", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_custom_header": "", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_need_stats": "true", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_tenant_name": "", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_token_expiration": "86400", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_url": "", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_url_prefix": "swift", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_lease_period": "120", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-24T11:54:02.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_thread_pool_size": "512", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_comment": "", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_createby": "", 2024-03-24T11:54:02.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_encoding": "", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_flag": "false", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_origin": "", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_tracker": "", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_trust_forwarded_https": "false", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_max_shards": "32", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_max_user_shards": "1", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_use_opa_authz": "false", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_counters_cache": "false", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-24T11:54:02.523 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_max_buckets": "1000", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_policies_max_num": "100", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_unique_email": "true", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_verify_ssl": "true", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone": "", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone_id": "", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup": "", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup_id": "", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-24T11:54:02.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgwlc_auto_session_clear": "true", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_block_size": "4096", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_size": "536870912", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-24T11:54:02.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_index_type": "binary_search", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_partition_filters": "false", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_perf": "false", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "run_dir": "/var/run/ceph", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_block_create": "true", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_cache_lru_size": "67108864", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_default_max_object_size": "16777216", 2024-03-24T11:54:02.526 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_device_size": "53687091200", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_capacity": "16", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_main_device_type": "SSD", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_segment_size": "67108864", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "set_keepcaps": "false", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "setgroup": "ceph", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "setuser": "ceph", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "setuser_match_path": "", 2024-03-24T11:54:02.527 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-24T11:54:02.528 INFO:teuthology.orchestra.run.smithi064.stdout: "thp": "false", 2024-03-24T11:54:02.528 INFO:teuthology.orchestra.run.smithi064.stdout: "threadpool_default_timeout": "60", 2024-03-24T11:54:02.528 INFO:teuthology.orchestra.run.smithi064.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-24T11:54:02.528 INFO:teuthology.orchestra.run.smithi064.stdout: "throttler_perf_counter": "true" 2024-03-24T11:54:02.528 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: 4.e scrub starts 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[30223]: 4.e scrub ok 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: 4.e scrub starts 2024-03-24T11:54:02.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:02 smithi064 ceph-mon[42243]: 4.e scrub ok 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: 4.e scrub starts 2024-03-24T11:54:02.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:02 smithi193 ceph-mon[33593]: 4.e scrub ok 2024-03-24T11:54:03.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-24T11:54:04.287 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:04.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:04 smithi064 ceph-mon[30223]: 5.7 scrub starts 2024-03-24T11:54:04.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:04 smithi064 ceph-mon[30223]: 5.7 scrub ok 2024-03-24T11:54:04.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:04 smithi064 ceph-mon[30223]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:04.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:04 smithi064 ceph-mon[42243]: 5.7 scrub starts 2024-03-24T11:54:04.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:04 smithi064 ceph-mon[42243]: 5.7 scrub ok 2024-03-24T11:54:04.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:04 smithi064 ceph-mon[42243]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:04.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:04 smithi193 ceph-mon[33593]: 5.7 scrub starts 2024-03-24T11:54:04.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:04 smithi193 ceph-mon[33593]: 5.7 scrub ok 2024-03-24T11:54:04.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:04 smithi193 ceph-mon[33593]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:05.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T11:54:05.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T11:54:05.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[30223]: 6.0 scrub starts 2024-03-24T11:54:05.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[30223]: 6.0 scrub ok 2024-03-24T11:54:05.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:05.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T11:54:05.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T11:54:05.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[42243]: 6.0 scrub starts 2024-03-24T11:54:05.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[42243]: 6.0 scrub ok 2024-03-24T11:54:05.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:05 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:05.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:05 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T11:54:05.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:05 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T11:54:05.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:05 smithi193 ceph-mon[33593]: 6.0 scrub starts 2024-03-24T11:54:05.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:05 smithi193 ceph-mon[33593]: 6.0 scrub ok 2024-03-24T11:54:05.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:05 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:05.993 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a: {} 2024-03-24T11:54:06.004 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b: {} 2024-03-24T11:54:06.014 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c: {} 2024-03-24T11:54:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:54:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:54:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:54:06.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[30223]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3870075413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[42243]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3870075413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:54:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:06.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:06 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:06.746 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-24T11:54:06.757 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T11:54:06.758 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T11:54:06.758 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-24T11:54:06.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-24T11:54:06.759 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-24T11:54:06.759 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:54:06.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-24T11:54:06.759 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-24T11:54:06.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:06.760 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-24T11:54:06.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:54:06.761 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-24T11:54:06.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T11:54:06.762 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-24T11:54:06.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:54:06.782 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:06 smithi193 ceph-mon[33593]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-24T11:54:06.782 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:06 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T11:54:06.782 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:06 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T11:54:06.782 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:06 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3870075413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:54:06.782 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:06 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:06.782 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:06 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:06.799 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:06.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:06.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:06.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:07.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:07.069 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:07.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:07.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:54:06 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:54:07.170 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:07.195 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:07.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:07.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:07.321 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:07.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:07.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:07.437 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:07.454 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:07.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:07.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:54:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:07.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:07.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:07.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:07.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:07.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:07.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:07.830 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:07.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:07.837 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:07.838 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:07.839 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:07.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:07.955 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:07.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:07.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[30223]: 2.19 scrub starts 2024-03-24T11:54:07.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[30223]: 2.19 scrub ok 2024-03-24T11:54:07.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:07.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:54:07.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:54:07.964 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[42243]: 2.19 scrub starts 2024-03-24T11:54:07.964 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[42243]: 2.19 scrub ok 2024-03-24T11:54:07.964 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:07.964 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:54:07.964 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:07 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:54:08.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:08.080 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:08.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:08.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:07 smithi193 ceph-mon[33593]: 2.19 scrub starts 2024-03-24T11:54:08.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:07 smithi193 ceph-mon[33593]: 2.19 scrub ok 2024-03-24T11:54:08.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:07 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:08.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:07 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:54:08.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:07 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:54:08.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:08.206 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:08.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:08.307 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:08.336 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:08.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:08.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:08.464 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:08.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:08.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:08.588 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:08.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:08.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:08.714 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:08.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:08.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:08 smithi064 ceph-mon[30223]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:08.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:08 smithi064 ceph-mon[30223]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:54:08.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:08 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T11:54:08.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:08 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T11:54:08.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:08 smithi064 ceph-mon[42243]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:08.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:08 smithi064 ceph-mon[42243]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:54:08.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:08 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T11:54:08.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:08 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T11:54:08.815 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:08.841 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:08.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:08.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:08.972 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:08.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:09.073 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:09.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:08 smithi193 ceph-mon[33593]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:08 smithi193 ceph-mon[33593]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:54:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:08 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T11:54:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:08 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T11:54:09.101 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:09.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:09.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:09.230 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:09.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:09.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:09.360 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:09.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:09.462 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:09.489 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:09.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:09.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:09.616 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:09.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:09.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:09.743 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:09.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:09.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:09.872 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:09.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:09.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:10.000 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:10.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:10.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:09 smithi064 ceph-mon[30223]: 4.4 scrub starts 2024-03-24T11:54:10.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:09 smithi064 ceph-mon[30223]: 4.4 scrub ok 2024-03-24T11:54:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:09 smithi064 ceph-mon[42243]: 4.4 scrub starts 2024-03-24T11:54:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:09 smithi064 ceph-mon[42243]: 4.4 scrub ok 2024-03-24T11:54:10.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:09 smithi193 ceph-mon[33593]: 4.4 scrub starts 2024-03-24T11:54:10.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:09 smithi193 ceph-mon[33593]: 4.4 scrub ok 2024-03-24T11:54:10.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:10.125 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:10.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:10.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:10.255 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:10.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:10.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:10.383 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:10.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:10.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:10.510 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:10.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:10.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:10.637 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:10.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:10.739 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:10.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:10.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:10.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:10.893 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:10.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:10.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:11.019 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:11.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:11.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:10 smithi064 ceph-mon[30223]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:11.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:11.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:10 smithi064 ceph-mon[42243]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:11.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:11.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:10 smithi193 ceph-mon[33593]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:11.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:11.121 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:11.149 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:11.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:11.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:11.275 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:11.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:11.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:11.400 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:11.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:11.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:11.526 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:11.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:11.627 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:11.656 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:11.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:11.758 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:11.780 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:11.785 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:11.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:11.887 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:11.914 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:11.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:12.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:12.040 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:12.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:12.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:12.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:12.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:12.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:12.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[30223]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:12.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:12.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:12.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[30223]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:54:12.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[30223]: Reconfiguring daemon alertmanager.a on smithi064 2024-03-24T11:54:12.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[42243]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:12.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:12.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:12.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[42243]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:54:12.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:12 smithi064 ceph-mon[42243]: Reconfiguring daemon alertmanager.a on smithi064 2024-03-24T11:54:12.295 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:12.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:12.397 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:12.424 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:12.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:12.526 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:12.554 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:12.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:12.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:12 smithi193 ceph-mon[33593]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:12 smithi193 ceph-mon[33593]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:54:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:12 smithi193 ceph-mon[33593]: Reconfiguring daemon alertmanager.a on smithi064 2024-03-24T11:54:12.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:12.681 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:12.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:12.782 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:12.809 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:12.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:12.912 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:12.939 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:12.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:13.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:13.065 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:13.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:13.106 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:54:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-03-24T11:54:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-03-24T11:54:13.106 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:54:13.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:13.191 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:13.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:13.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:13.317 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:13.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:13.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:13.443 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:13.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:13.544 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:13.569 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:13.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:13.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:13.699 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:13.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:13.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:13.826 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:13.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:13.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:13.951 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:13.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:14.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:14.080 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:54:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:14.081 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:14.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:14.099 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:54:14.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:54:14.137 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:54:14.158 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:54:14.178 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:54:14.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:54:14.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:54:14.265 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:54:14.287 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:14.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:14.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:14.413 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:14.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:14.515 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:14.544 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:14.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[42243]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T11:54:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[42243]: 2.1f deep-scrub starts 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[42243]: 2.1f deep-scrub ok 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2457147454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[30223]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[30223]: 2.1f deep-scrub starts 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[30223]: 2.1f deep-scrub ok 2024-03-24T11:54:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2457147454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:14 smithi193 ceph-mon[33593]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:14 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T11:54:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:14 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T11:54:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:14 smithi193 ceph-mon[33593]: 2.1f deep-scrub starts 2024-03-24T11:54:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:14 smithi193 ceph-mon[33593]: 2.1f deep-scrub ok 2024-03-24T11:54:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2457147454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:14.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:14.673 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:14.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:14.775 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:14.803 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:14.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:14.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:14.928 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:14.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:15.030 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:15.058 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:15.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:15.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:15.186 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:15.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:15.229 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_0' created 2024-03-24T11:54:15.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:15.311 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:15.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:15.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:15.436 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:15.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:15.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:15 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1661641945' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:54:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:15 smithi064 ceph-mon[30223]: 4.11 scrub starts 2024-03-24T11:54:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:15 smithi064 ceph-mon[30223]: 4.11 scrub ok 2024-03-24T11:54:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:15 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1661641945' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:54:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:15 smithi064 ceph-mon[42243]: 4.11 scrub starts 2024-03-24T11:54:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:15 smithi064 ceph-mon[42243]: 4.11 scrub ok 2024-03-24T11:54:15.566 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:15.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:15 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1661641945' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:54:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:15 smithi193 ceph-mon[33593]: 4.11 scrub starts 2024-03-24T11:54:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:15 smithi193 ceph-mon[33593]: 4.11 scrub ok 2024-03-24T11:54:15.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:15.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:15.692 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:15.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:15.793 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:15.821 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:15.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:15.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:15.951 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:15.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:16.053 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:16.081 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:16.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:16.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:16.207 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:16.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:16.310 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:16.338 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:16.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:16.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:16.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:16.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:16.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[30223]: 5.1 scrub starts 2024-03-24T11:54:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[30223]: 5.1 scrub ok 2024-03-24T11:54:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[30223]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1661641945' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T11:54:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[30223]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:54:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[30223]: 5.17 scrub starts 2024-03-24T11:54:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[30223]: 5.17 scrub ok 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[42243]: 5.1 scrub starts 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[42243]: 5.1 scrub ok 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[42243]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1661641945' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[42243]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[42243]: 5.17 scrub starts 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:16 smithi064 ceph-mon[42243]: 5.17 scrub ok 2024-03-24T11:54:16.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:16 smithi193 ceph-mon[33593]: 5.1 scrub starts 2024-03-24T11:54:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:16 smithi193 ceph-mon[33593]: 5.1 scrub ok 2024-03-24T11:54:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:16 smithi193 ceph-mon[33593]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:16 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1661641945' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T11:54:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:16 smithi193 ceph-mon[33593]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:54:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:16 smithi193 ceph-mon[33593]: 5.17 scrub starts 2024-03-24T11:54:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:16 smithi193 ceph-mon[33593]: 5.17 scrub ok 2024-03-24T11:54:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:16.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:16.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:16.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:16.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:16.843 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:16.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:16.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:16.972 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:16.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:17.074 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:17.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:54:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:54:17.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:17.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:17.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:17.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:17.243 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[42243]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3093703516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T11:54:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T11:54:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:54:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:54:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[30223]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:54:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:17.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3093703516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:54:17.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:17.358 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:17.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:17.460 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:17.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:17.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:17 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T11:54:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:17 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T11:54:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:17 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:54:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:17 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:54:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:17 smithi193 ceph-mon[33593]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:54:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:17 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3093703516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:54:17.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:17.613 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:17.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:17.655 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:54:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:17.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:17.743 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:17.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:17.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:17.872 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:17.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:17.974 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:17.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T11:54:18.002 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:18.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:18.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:18.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:18.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:18.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:18.256 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:18.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:18.358 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:18.386 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:18.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:18.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:18.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[42243]: pgmap v61: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[42243]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3093703516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[42243]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:54:18.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[30223]: pgmap v61: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:18.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:18.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[30223]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:18.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3093703516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:54:18.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:18 smithi064 ceph-mon[30223]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:54:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:18 smithi193 ceph-mon[33593]: pgmap v61: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:18 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:18 smithi193 ceph-mon[33593]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3093703516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:54:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:18 smithi193 ceph-mon[33593]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:54:18.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:18.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:18.739 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:18.766 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:18.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:18.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:18.893 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:18.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:18.994 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:19.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:19.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:19.119 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:19.148 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:19.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:19.249 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-24T11:54:19.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:19.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:19.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:19.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:19.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-24T11:54:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:19.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:19.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:19.535 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:19.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:19 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/483471593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:54:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:19 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/483471593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:54:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:19 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/483471593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:54:19.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:19.663 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:19.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:19.765 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:19.789 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:19.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:19.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:19.914 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:19.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:20.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:20.044 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:20.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:20.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:20.169 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:20.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:20.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:20.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:20.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:20.418 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:20.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:20.520 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:20.545 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:20.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:20 smithi064 ceph-mon[30223]: pgmap v63: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:20 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/483471593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:54:20.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:20 smithi064 ceph-mon[30223]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:54:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:20 smithi064 ceph-mon[42243]: pgmap v63: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/483471593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:54:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:20 smithi064 ceph-mon[42243]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:54:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:20 smithi193 ceph-mon[33593]: pgmap v63: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:20 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/483471593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:54:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:20 smithi193 ceph-mon[33593]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:54:20.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:20.673 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:20.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:20.774 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:20.798 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:20.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:20.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:20.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:20.924 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:20.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:21.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:21.049 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:21.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:21.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:21.175 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:21.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:21.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:21.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:21.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:21.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:21.427 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:21.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:21.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:21 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:54:21.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:21 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:54:21.503 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:21 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:54:21.503 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:21 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:54:21.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-24T11:54:21.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:21.552 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:21.553 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:21.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:21 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:54:21.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:21 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:54:21.654 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:21.678 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:21.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:21.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:21.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:21.805 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:21.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:21.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:21.929 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:21.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:21.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:22.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:22.055 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:22.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:22.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:22.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:22.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:22.308 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:22.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:22.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:22.435 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:22.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:22.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:22 smithi064 ceph-mon[42243]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:54:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:22 smithi064 ceph-mon[42243]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:54:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:22 smithi064 ceph-mon[30223]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:54:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:22 smithi064 ceph-mon[30223]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:54:22.562 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:22.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:22 smithi193 ceph-mon[33593]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:54:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:22 smithi193 ceph-mon[33593]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:54:22.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:22.689 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:22.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:22.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:54:22.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:22.815 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:22.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:22.850 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:22 smithi064 systemd[1]: Stopping Ceph alertmanager.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:54:22.869 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:22.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:22.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:23.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:23.068 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:23.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:23.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:23.194 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:23.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:23.295 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:23.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:23.323 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:23.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:23.379 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:23.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:23.448 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:23.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:23.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:23.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:23.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:23.677 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:23.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:23.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:23.770 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:23 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[79579]: ts=2024-03-24T11:54:23.529Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:54:23.770 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:23 smithi064 podman[85069]: 2024-03-24 11:54:23.729067304 +0000 UTC m=+0.345940252 container died 99c8402f534fe712e26b28028754771ba6acc170b5c72a44e73578811d266bdc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:23.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:23.834 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:23.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:23.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:23.959 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:23.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:24.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:54:24.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:24.085 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:24.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:24.186 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:24.214 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:24.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:24 smithi064 ceph-mon[30223]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:24.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:24 smithi064 ceph-mon[42243]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:24.302 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:24 smithi064 podman[85069]: 2024-03-24 11:54:24.245995724 +0000 UTC m=+0.862868669 container cleanup 99c8402f534fe712e26b28028754771ba6acc170b5c72a44e73578811d266bdc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:24.303 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:24 smithi064 bash[85069]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:54:24.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:24.340 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:24.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:24.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:24.465 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:24.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:24.567 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:24 smithi193 ceph-mon[33593]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:24.595 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:24.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:24.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:24.721 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:24.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:24.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:24.846 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:24.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:24.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:24.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:24.949 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:24 smithi064 podman[85119]: 2024-03-24 11:54:24.866577082 +0000 UTC m=+1.319256073 container remove 99c8402f534fe712e26b28028754771ba6acc170b5c72a44e73578811d266bdc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:24.949 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:24 smithi064 podman[85119]: 2024-03-24 11:54:24.95059337 +0000 UTC m=+1.403272324 volume remove 181ba13240481972096c27e218ea48c4d671e9abf19fd0a55020606847ff9491 2024-03-24T11:54:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:24.977 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:24.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:25.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:25.101 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:25.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:25.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:25.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:25.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:25.357 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:25.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:25.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:25.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:25.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:25.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:25.616 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:25.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:25.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:25.741 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:25.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:25.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:25.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:25.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:25.996 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:25.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:26.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:26.122 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:26.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:26.223 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:26.251 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:26.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:26.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:26.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:26.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:26.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:26.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:26 smithi064 ceph-mon[30223]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:54:26.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:26.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:26.502 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:26 smithi064 ceph-mon[42243]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:54:26.502 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:26.503 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:26.504 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:26.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:26.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:26 smithi193 ceph-mon[33593]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:54:26.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:26.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:26.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:26.631 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:26.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:26.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:26.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:26.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:26.887 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:26.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:26.989 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:27.016 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:27.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:27.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:54:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:54:27.118 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:27.142 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:54:27.142 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:54:27.144 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:27.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:27.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:27.270 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:27.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:27.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:27.401 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:27.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:27.502 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:27.529 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:27.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:27.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:54:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:27.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:27.654 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:27.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:27.756 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:27.784 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:27.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:27.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:27.910 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:27.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:28.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:28.013 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:27 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:28.013 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:27 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:28.038 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:28.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:28.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T11:54:28.085 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:28.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:27 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:28.116 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:28.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:28.168 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:28.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:28.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:28.294 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:28.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:28.395 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:28.422 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:28.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:28.524 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:28.551 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:28.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:28.647 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-24T11:54:28.653 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:28.683 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:28.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:28.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:28.808 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:28.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:28.891 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:28 smithi064 ceph-mon[42243]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:28.891 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:28 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2562033997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:54:28.891 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:28 smithi064 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:54:28.891 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:28 smithi064 systemd[1]: Stopped Ceph alertmanager.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:54:28.892 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:28 smithi064 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.004s CPU time. 2024-03-24T11:54:28.892 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:28 smithi064 systemd[1]: Starting Ceph alertmanager.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:54:28.892 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:28 smithi064 ceph-mon[30223]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:28.892 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2562033997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:54:28.910 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:28.938 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:28.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:29.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:29.064 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:29.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:28 smithi193 ceph-mon[33593]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:28 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2562033997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:54:29.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:29.189 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:29.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:29.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:29.318 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:29.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:29.420 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:29.447 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:29.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:29.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:29.549 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:29.577 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:29.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:29.679 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:29.708 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:29.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:29.809 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:29.835 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:29.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:29.937 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:29.965 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:29.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:29 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2562033997' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:54:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:29 smithi064 ceph-mon[30223]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:54:30.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:29 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2562033997' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:54:30.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:29 smithi064 ceph-mon[42243]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:54:30.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:30.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:29 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2562033997' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:54:30.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:29 smithi193 ceph-mon[33593]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:54:30.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:30.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:30.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-24T11:54:30.196 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:30.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:30.233 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:30.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:30.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:30.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:30.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:30.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:30.489 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:30.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:30.592 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:30.620 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:30.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:30.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:30.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:30.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:30.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:30.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:30 smithi064 ceph-mon[30223]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:54:30.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:30 smithi064 ceph-mon[42243]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:54:30.757 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:30 smithi064 podman[86212]: 2024-03-24 11:54:30.44420382 +0000 UTC m=+0.207830692 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-24T11:54:30.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:30.872 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:30.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:30.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:30.997 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:30.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:30 smithi193 ceph-mon[33593]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:54:31.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:31.123 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:31.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:31.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:31.247 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:31.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:31.349 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:31.377 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:31.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:31.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:31.508 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:31.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:31.609 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:31.636 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:31.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:31.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:31.763 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:31.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:31.797 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2631103038' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:31.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2631103038' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:31.865 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:31.893 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:31.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:31.995 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:32.021 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:32.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:32.052 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:31 smithi064 podman[86212]: 2024-03-24 11:54:31.797531546 +0000 UTC m=+1.561158390 volume create d83ecb71e1e373788c59d31a059cfbcaeac710b6abbb09b82e6a83db31e8d4f7 2024-03-24T11:54:32.052 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:31 smithi064 podman[86212]: 2024-03-24 11:54:31.898117567 +0000 UTC m=+1.661744411 container create 9864e48d66ff759409e7aadc63e6e1d24752f545b830d3f82a75e2ee65bcf817 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:32.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2631103038' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:32.123 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:32.149 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:32.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:32.182 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-24T11:54:32.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:32.276 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:32.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:32.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:32.403 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:32.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:32.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:32.530 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:32.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:32.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:32.657 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:32.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:32.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:32.783 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:32.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:32 smithi064 ceph-mon[30223]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:54:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:32 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2631103038' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:32 smithi064 ceph-mon[30223]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:54:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:32 smithi064 ceph-mon[42243]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:54:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:32 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2631103038' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:32 smithi064 ceph-mon[42243]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:54:32.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:32.909 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:32.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:33.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:33.036 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:33.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:32 smithi193 ceph-mon[33593]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:54:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:32 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2631103038' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:32 smithi193 ceph-mon[33593]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:54:33.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:33.164 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:33.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:33.208 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 podman[86212]: 2024-03-24 11:54:33.106398987 +0000 UTC m=+2.870025849 container init 9864e48d66ff759409e7aadc63e6e1d24752f545b830d3f82a75e2ee65bcf817 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:33.208 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 podman[86212]: 2024-03-24 11:54:33.112194121 +0000 UTC m=+2.875820966 container start 9864e48d66ff759409e7aadc63e6e1d24752f545b830d3f82a75e2ee65bcf817 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:33.208 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:33.124Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T11:54:33.208 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:33.124Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T11:54:33.208 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:33.125Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.64 port=9094 2024-03-24T11:54:33.209 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:33.126Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:54:33.209 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:33.149Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:54:33.209 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:33.150Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:54:33.209 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:33.152Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T11:54:33.209 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:33.152Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T11:54:33.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:33.290 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:33.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:33.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:33.415 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:33.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:33.473 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 bash[86212]: 9864e48d66ff759409e7aadc63e6e1d24752f545b830d3f82a75e2ee65bcf817 2024-03-24T11:54:33.473 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:33 smithi064 systemd[1]: Started Ceph alertmanager.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:54:33.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:33.541 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:33.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:33.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:33.666 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:33.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:33.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[30223]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[42243]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:33.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:33 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:33.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:33.793 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:33.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:33.894 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:33.919 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:33.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:34.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:34.044 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:34.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:34.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:33 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:54:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:33 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:54:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:33 smithi193 ceph-mon[33593]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:54:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:33 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:33 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:34.123 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:54:34.143 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:54:34.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:34.162 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:54:34.169 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:34.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:34.182 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:54:34.202 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:54:34.222 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:54:34.242 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:54:34.262 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:54:34.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:54:34.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:54:34.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:54:34.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:54:34.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:54:34.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:54:34.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:54:34.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:54:34.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:34.295 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:34.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:34.397 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:34.423 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:34.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:34.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:34.550 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:34.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:34.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:34.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:34.678 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T11:54:34.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:34.803 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:34.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:34.904 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:34.930 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:34.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:35.032 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:35.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:34 smithi193 ceph-mon[33593]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:35.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:34 smithi193 ceph-mon[33593]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:54:35.035 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:34 smithi193 ceph-mon[33593]: Reconfiguring daemon prometheus.a on smithi193 2024-03-24T11:54:35.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:34 smithi193 ceph-mon[33593]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:54:35.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:34 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/612849094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:35.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:34 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2230871151' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:54:35.036 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:34 smithi193 ceph-mon[33593]: from='client.34727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:54:35.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[30223]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:35.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[30223]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:54:35.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[30223]: Reconfiguring daemon prometheus.a on smithi193 2024-03-24T11:54:35.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[30223]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:54:35.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/612849094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2230871151' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[30223]: from='client.34727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[42243]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[42243]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[42243]: Reconfiguring daemon prometheus.a on smithi193 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[42243]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/612849094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2230871151' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:54:35.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:34 smithi064 ceph-mon[42243]: from='client.34727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:54:35.057 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:35.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:35.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:35.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:35.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:35.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:35.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:35.311 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:35.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:35.389 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:35 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:35.126Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.0001488s 2024-03-24T11:54:35.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:35.436 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:35.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:35.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:35.561 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:35.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:35.620 DEBUG:teuthology.orchestra.run.smithi064:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-24T11:54:35.623 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T11:54:35.623 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T11:54:35.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:35.689 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:35.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:35.699 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:hints = 1 2024-03-24T11:54:35.700 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T11:54:35.700 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object prefix: benchmark_data_smithi064_87013 2024-03-24T11:54:35.700 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:54:35.700 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 0 0 0 0 0 0 - 0 2024-03-24T11:54:35.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:35.834 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:35.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:35.907 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:35 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:35.907 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:35 smithi193 ceph-mon[33593]: from='client.34727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:54:35.907 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:35 smithi193 ceph-mon[33593]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:54:35.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:35.980 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:35.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:35.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:35 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:35.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:35 smithi064 ceph-mon[30223]: from='client.34727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:54:35.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:35 smithi064 ceph-mon[30223]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:54:35.988 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:35 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:35.988 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:35 smithi064 ceph-mon[42243]: from='client.34727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:54:35.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:35 smithi064 ceph-mon[42243]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:54:36.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:36.121 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:36.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:36.223 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:36.258 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:36.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:36.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:36.411 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:36.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:36.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:36.556 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:36.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:36.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:36.699 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:36.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:36.699 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 1 16 3379 3363 210.176 210.188 0.00281793 0.00473891 2024-03-24T11:54:36.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:36.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:36 smithi193 ceph-mon[33593]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:36.843 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:36.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:36.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:36 smithi064 ceph-mon[30223]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:36.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:36 smithi064 ceph-mon[42243]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:54:36.940 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:36.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:36.976 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:36.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:37.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:37.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:54:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:54:37.086 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 systemd[1]: Stopping Ceph prometheus.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:54:37.120 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:37.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:37.222 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:37.260 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:37.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:37.362 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:37.398 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:37.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:37.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:54:37.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:37.543 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:37.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:37.645 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:37.695 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:37.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:37.700 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 2 16 6351 6335 197.953 185.75 0.00620498 0.0050283 2024-03-24T11:54:37.704 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:54:37.704 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:54:37.704 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:54:37.704 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:54:37.704 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:54:37.705 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T11:54:37.705 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:54:37.705 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T11:54:37.705 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T11:54:37.705 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:54:37.705 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.375Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T11:54:37.705 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[64838]: ts=2024-03-24T11:54:37.376Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T11:54:37.705 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 podman[70178]: 2024-03-24 11:54:37.386370674 +0000 UTC m=+0.159202546 container died e39a1ce87a77071da96a4227ef42f3e9ba882ffdb77049b4dd147b828347881f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:37.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:37.813 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:54:37.813 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T11:54:37.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T11:54:37.813 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T11:54:37.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.813 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:2282 7.4 7:28776a0b:::benchmark_data_smithi064_87013_object2281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.813 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:36.362895+0000", 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.446386575, 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010660927000000001, 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2282 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.362895+0000", 2024-03-24T11:54:37.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.362895+0000", 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.362896+0000", 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.423e-06 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.362919+0000", 2024-03-24T11:54:37.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.283e-05 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.362927+0000", 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4000000000000003e-06 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.362934+0000", 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0330000000000003e-06 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.365014+0000", 2024-03-24T11:54:37.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020802360000000001 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.365252+0000", 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023779500000000001 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.365382+0000", 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013045999999999999 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.367889+0000", 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025070230000000002 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.368263+0000", 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037398299999999999 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.373466+0000", 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052029729999999996 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.373514+0000", 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7620000000000001e-05 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.373556+0000", 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2150999999999997e-05 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.819 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:4134 7.4 7:27ee364d:::benchmark_data_smithi064_87013_object4133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:36.928624+0000", 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.88065785299999999, 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010535577000000001, 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4134 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.928624+0000", 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.928624+0000", 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.928625+0000", 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6400000000000001e-07 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.928649+0000", 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4335e-05 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.928656+0000", 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.576e-06 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.928663+0000", 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4470000000000003e-06 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.930589+0000", 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019262039999999999 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.930751+0000", 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016138700000000001 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.930898+0000", 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014711499999999999 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.932854+0000", 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001956565 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.934973+0000", 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002118586 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939027+0000", 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040539299999999999 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939103+0000", 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6423999999999999e-05 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939159+0000", 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6144000000000003e-05 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:4157 7.f 7:fd766592:::benchmark_data_smithi064_87013_object4156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.825 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:36.937963+0000", 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.87131808899999996, 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013535080999999999, 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4157 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.937963+0000", 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.937963+0000", 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.937966+0000", 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0650000000000001e-06 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.937988+0000", 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2279000000000001e-05 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.937999+0000", 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1644999999999999e-05 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.938008+0000", 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2160000000000006e-06 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.938086+0000", 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8659999999999996e-05 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.938246+0000", 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015979500000000001 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.940034+0000", 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017874200000000001 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.941376+0000", 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001342394 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.945269+0000", 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038926479999999999 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.951350+0000", 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060813209999999998 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.951445+0000", 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.951499+0000", 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3774999999999999e-05 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:4158 7.9 7:9678b322:::benchmark_data_smithi064_87013_object4157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.831 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:36.939315+0000", 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.86996642300000004, 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010137297, 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4158 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939315+0000", 2024-03-24T11:54:37.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939315+0000", 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939317+0000", 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.824e-06 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939381+0000", 2024-03-24T11:54:37.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4411000000000004e-05 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939404+0000", 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2248000000000001e-05 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.939427+0000", 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3101000000000001e-05 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.940100+0000", 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067301500000000001 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.940242+0000", 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141858 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.940380+0000", 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138737 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.945703+0000", 2024-03-24T11:54:37.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053222210000000002 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.947779+0000", 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002076952 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.949350+0000", 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015708639999999999 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.949407+0000", 2024-03-24T11:54:37.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6746000000000002e-05 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:36.949452+0000", 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.532e-05 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:5035 7.9 7:95518dda:::benchmark_data_smithi064_87013_object5034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.223558+0000", 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.58572358199999996, 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01039843, 2024-03-24T11:54:37.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5035 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.223558+0000", 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.223558+0000", 2024-03-24T11:54:37.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.223559+0000", 2024-03-24T11:54:37.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0100000000000001e-06 2024-03-24T11:54:37.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.223585+0000", 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6503e-05 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.223593+0000", 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8599999999999993e-06 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.223600+0000", 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5560000000000004e-06 2024-03-24T11:54:37.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.224731+0000", 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011309880000000001 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.224845+0000", 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114349 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.224994+0000", 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014859699999999999 2024-03-24T11:54:37.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.227922+0000", 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029283310000000002 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.227935+0000", 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3244999999999999e-05 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.233854+0000", 2024-03-24T11:54:37.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059183650000000001 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.233915+0000", 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1451000000000005e-05 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.233956+0000", 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1174999999999998e-05 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6077 7.6 7:6421b764:::benchmark_data_smithi064_87013_object6076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.579492+0000", 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.229789455, 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011100241, 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6077 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.579492+0000", 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.579492+0000", 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.579493+0000", 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5300000000000003e-07 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.579539+0000", 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6025000000000001e-05 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.579546+0000", 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9369999999999998e-06 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.579552+0000", 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1149999999999999e-06 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.580282+0000", 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072986800000000005 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.580398+0000", 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116558 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.580535+0000", 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013630900000000001 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.581032+0000", 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049703699999999998 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.582295+0000", 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012634370000000001 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590498+0000", 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082026640000000001 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590550+0000", 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2516000000000003e-05 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590592+0000", 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1922000000000003e-05 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6106 7.4 7:2b50039d:::benchmark_data_smithi064_87013_object6105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.590401+0000", 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.21888060400000001, 2024-03-24T11:54:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012643474, 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6106 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590401+0000", 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590401+0000", 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590403+0000", 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.001e-06 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590424+0000", 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1401999999999999e-05 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590433+0000", 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.473e-06 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590438+0000", 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4519999999999996e-06 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590657+0000", 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021830999999999999 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590785+0000", 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128257 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.590940+0000", 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015509700000000001 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592038+0000", 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010977070000000001 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592294+0000", 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000256603 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.602949+0000", 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010655029999999999 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.603001+0000", 2024-03-24T11:54:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1276000000000002e-05 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.603044+0000", 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3866000000000003e-05 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6107 7.9 7:970ebfc0:::benchmark_data_smithi064_87013_object6106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.591428+0000", 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.21785332900000001, 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012374227999999999, 2024-03-24T11:54:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6107 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591428+0000", 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591428+0000", 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591429+0000", 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.139e-06 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591455+0000", 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5406e-05 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591484+0000", 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9054e-05 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591498+0000", 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4134000000000001e-05 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591580+0000", 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1734e-05 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591701+0000", 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012086700000000001 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.591829+0000", 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012843500000000001 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592466+0000", 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063691900000000001 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.593664+0000", 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011984980000000001 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.603693+0000", 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010028877 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.603752+0000", 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8693999999999997e-05 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.603802+0000", 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0470999999999998e-05 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6112 7.4 7:25e5d280:::benchmark_data_smithi064_87013_object6111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.592266+0000", 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.21701584900000001, 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011636275999999999, 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6112 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592266+0000", 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592266+0000", 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592267+0000", 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592289+0000", 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1749000000000001e-05 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592297+0000", 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3610000000000001e-06 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592304+0000", 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2779999999999996e-06 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592346+0000", 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2246999999999997e-05 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.592457+0000", 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011082 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.593062+0000", 2024-03-24T11:54:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000604555 2024-03-24T11:54:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.594102+0000", 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001040175 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.594416+0000", 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031450199999999998 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.603808+0000", 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093919490000000001 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.603857+0000", 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9039999999999998e-05 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.603902+0000", 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4496999999999998e-05 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6116 7.9 7:9a4781ca:::benchmark_data_smithi064_87013_object6115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.596567+0000", 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.21271483599999999, 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011173984, 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6116 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596567+0000", 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596567+0000", 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596569+0000", 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1890000000000001e-06 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596596+0000", 2024-03-24T11:54:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6741e-05 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596612+0000", 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6087e-05 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596620+0000", 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0979999999999994e-06 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596703+0000", 2024-03-24T11:54:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2850999999999997e-05 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596819+0000", 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116309 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.596967+0000", 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014803599999999999 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.597938+0000", 2024-03-24T11:54:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097096300000000003 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.598959+0000", 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001020789 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.607656+0000", 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086970699999999995 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.607703+0000", 2024-03-24T11:54:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7493000000000001e-05 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.607741+0000", 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7357999999999997e-05 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6147 7.4 7:23ac6145:::benchmark_data_smithi064_87013_object6146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.609650+0000", 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.19963146500000001, 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010242618, 2024-03-24T11:54:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6147 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.609650+0000", 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.609650+0000", 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.609652+0000", 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.517e-06 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.609673+0000", 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1691999999999998e-05 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.609684+0000", 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0278000000000001e-05 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.609691+0000", 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6310000000000003e-06 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.610322+0000", 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000630604 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.610460+0000", 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137949 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.610624+0000", 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016391100000000001 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.614682+0000", 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040579120000000003 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.614694+0000", 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2802999999999999e-05 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.619773+0000", 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005079114 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:37.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:37.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.619832+0000", 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8696000000000002e-05 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.619893+0000", 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0510999999999997e-05 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6231 7.f 7:fe6f2ea5:::benchmark_data_smithi064_87013_object6230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.643077+0000", 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.16620412300000001, 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013327953, 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6231 2024-03-24T11:54:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643077+0000", 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643077+0000", 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643079+0000", 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.956e-06 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643101+0000", 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2036999999999999e-05 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643111+0000", 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0013999999999999e-05 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643120+0000", 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2300000000000008e-06 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643238+0000", 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118361 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643333+0000", 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5470000000000006e-05 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.643425+0000", 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1728e-05 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.645493+0000", 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020675490000000001 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.647113+0000", 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001620387 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.656309+0000", 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091961630000000003 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.656361+0000", 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1468000000000003e-05 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.656405+0000", 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4589999999999998e-05 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6244 7.f 7:f8d4c9e4:::benchmark_data_smithi064_87013_object6243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.647109+0000", 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.162172448, 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010523357000000001, 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6244 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.647109+0000", 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.647109+0000", 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.647111+0000", 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2180000000000002e-06 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.647136+0000", 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4834999999999999e-05 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.647149+0000", 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2999999999999999e-05 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.647158+0000", 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0289999999999994e-06 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.647436+0000", 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000277991 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.652850+0000", 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054139030000000003 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.653011+0000", 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016047300000000001 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.656417+0000", 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003406147 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.656457+0000", 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0083000000000002e-05 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.657455+0000", 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000998229 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.657525+0000", 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0332000000000004e-05 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.657632+0000", 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107117 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6316 7.f 7:f188c3b3:::benchmark_data_smithi064_87013_object6315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.679149+0000", 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.130132367, 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010094786999999999, 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6316 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.679149+0000", 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.679149+0000", 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.679150+0000", 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8999999999999996e-07 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.679170+0000", 2024-03-24T11:54:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9891e-05 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.679176+0000", 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7320000000000001e-06 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.679182+0000", 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4779999999999999e-06 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.681234+0000", 2024-03-24T11:54:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020521379999999998 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.681347+0000", 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011255 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.682621+0000", 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001274744 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.684744+0000", 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021222720000000001 2024-03-24T11:54:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.687147+0000", 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024030800000000001 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.689147+0000", 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020003909999999998 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.689203+0000", 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6076999999999999e-05 2024-03-24T11:54:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.689244+0000", 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0744000000000002e-05 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:6321 7.e e105/97 7:7edcd848:::benchmark_data_smithi064_87013_object6320:head v 105'361, mlcod=105'359)", 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.685604+0000", 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.123677538, 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992212, 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.685604+0000", 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.685604+0000", 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.685605+0000", 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.685631+0000", 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6676999999999999e-05 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.685639+0000", 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7440000000000007e-06 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.685645+0000", 2024-03-24T11:54:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0859999999999998e-06 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.688153+0000", 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025077680000000001 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.688172+0000", 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8669e-05 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.697544+0000", 2024-03-24T11:54:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093719219999999995 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.697596+0000", 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2652999999999999e-05 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6331 7.6 7:6ed53ab4:::benchmark_data_smithi064_87013_object6330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.692915+0000", 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.11636703800000001, 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014022985999999999, 2024-03-24T11:54:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6331 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692915+0000", 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692915+0000", 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692917+0000", 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7099999999999999e-06 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692947+0000", 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0216e-05 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692965+0000", 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7487999999999999e-05 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692974+0000", 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8240000000000006e-06 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.694739+0000", 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017652449999999999 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.694860+0000", 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012111 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.695007+0000", 2024-03-24T11:54:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147136 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.698509+0000", 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035013869999999999 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.698920+0000", 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041103999999999999 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.706761+0000", 2024-03-24T11:54:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078416790000000007 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.706840+0000", 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8929000000000003e-05 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.706937+0000", 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7221999999999999e-05 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6333 7.6 7:65614870:::benchmark_data_smithi064_87013_object6332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.692988+0000", 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.116293224, 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018682544999999998, 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6333 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692988+0000", 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692988+0000", 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.692989+0000", 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8400000000000003e-07 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.693012+0000", 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2578999999999998e-05 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.693018+0000", 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.693024+0000", 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4960000000000004e-06 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.697748+0000", 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004724242 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.697907+0000", 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158951 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.698053+0000", 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014573 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700760+0000", 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027074379999999999 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.703210+0000", 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024496650000000002 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711571+0000", 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083612740000000001 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711624+0000", 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2410999999999997e-05 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711671+0000", 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7255000000000001e-05 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6347 7.f 7:f09e82dc:::benchmark_data_smithi064_87013_object6346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.700194+0000", 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.109087421, 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011478136999999999, 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6347 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700194+0000", 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700194+0000", 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700196+0000", 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0269999999999998e-06 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700220+0000", 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4000000000000001e-05 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700235+0000", 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4691999999999999e-05 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700243+0000", 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3459999999999994e-06 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700305+0000", 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2132999999999996e-05 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700445+0000", 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000139245 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.702474+0000", 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020291710000000002 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.710251+0000", 2024-03-24T11:54:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077777569999999997 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.710817+0000", 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056594800000000004 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711572+0000", 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075479799999999999 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711621+0000", 2024-03-24T11:54:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9038e-05 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711672+0000", 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0982000000000001e-05 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6349 7.6 7:6f6fff88:::benchmark_data_smithi064_87013_object6348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.700415+0000", 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.10886647000000001, 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011389099999999999, 2024-03-24T11:54:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6349 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700415+0000", 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700415+0000", 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700417+0000", 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.561e-06 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700439+0000", 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1891000000000001e-05 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700450+0000", 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.152e-05 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700458+0000", 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.028e-06 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700795+0000", 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033645600000000001 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.700933+0000", 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013869500000000001 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.701088+0000", 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473900000000001 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.704497+0000", 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034086490000000001 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.704510+0000", 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3617e-05 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711720+0000", 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072098889999999997 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711766+0000", 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.634e-05 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.711804+0000", 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7715000000000001e-05 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:6371 7.f 7:f55c7d60:::benchmark_data_smithi064_87013_object6370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:37.716534+0000", 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.092747766999999995, 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011455737000000001, 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:54:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6371 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.716534+0000", 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.716534+0000", 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.716534+0000", 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-07 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.716554+0000", 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9777000000000001e-05 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.716558+0000", 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1339999999999997e-06 2024-03-24T11:54:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.716562+0000", 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6569999999999999e-06 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.716605+0000", 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2833000000000001e-05 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.716748+0000", 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143144 2024-03-24T11:54:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.717954+0000", 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012055569999999999 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.720358+0000", 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024046470000000002 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.727824+0000", 2024-03-24T11:54:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074660639999999997 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.727880+0000", 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5926e-05 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.727944+0000", 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3973000000000006e-05 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:37.727990+0000", 2024-03-24T11:54:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5393999999999999e-05 2024-03-24T11:54:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:37.932 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:54:37.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:38.013 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:38.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:37 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:37 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:38.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:37 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:38.086 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 podman[70178]: 2024-03-24 11:54:37.804207646 +0000 UTC m=+0.577039513 container cleanup e39a1ce87a77071da96a4227ef42f3e9ba882ffdb77049b4dd147b828347881f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:38.086 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:37 smithi193 bash[70178]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:54:38.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:38.156 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:38.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:38.257 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:38.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:38.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:38.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:38.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:38.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:38.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:38.578 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:38.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:38.586 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:38 smithi193 podman[70225]: 2024-03-24 11:54:38.256526591 +0000 UTC m=+0.869578826 container remove e39a1ce87a77071da96a4227ef42f3e9ba882ffdb77049b4dd147b828347881f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:38.586 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:38 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T11:54:38.586 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:38 smithi193 systemd[1]: Stopped Ceph prometheus.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:54:38.586 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:38 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.350s CPU time. 2024-03-24T11:54:38.586 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:38 smithi193 systemd[1]: Starting Ceph prometheus.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T11:54:38.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:38.700 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 3 16 9033 9017 187.813 167.625 0.00518045 0.00531554 2024-03-24T11:54:38.721 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:38.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:38.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:38.856 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:38.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:38.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:38.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:54:39.000 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:39.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:39.027 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:38 smithi193 ceph-mon[33593]: pgmap v78: 148 pgs: 148 active+clean; 95 MiB data, 314 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 16 MiB/s wr, 252 op/s 2024-03-24T11:54:39.027 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:38 smithi193 podman[70356]: 2024-03-24 11:54:38.637968513 +0000 UTC m=+0.032880191 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T11:54:39.028 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:38 smithi193 podman[70356]: 2024-03-24 11:54:38.758410866 +0000 UTC m=+0.153322526 container create 01e74f4670046fc440ec4da06892b58b9ec3b208214e13a0328060135425a389 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:39.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:38 smithi064 ceph-mon[30223]: pgmap v78: 148 pgs: 148 active+clean; 95 MiB data, 314 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 16 MiB/s wr, 252 op/s 2024-03-24T11:54:39.042 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:38 smithi064 ceph-mon[42243]: pgmap v78: 148 pgs: 148 active+clean; 95 MiB data, 314 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 16 MiB/s wr, 252 op/s 2024-03-24T11:54:39.102 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:39.134 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:39.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:39.235 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:39.266 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:39.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:39.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:39.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STOPPING 2024-03-24T11:54:39.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:54:39.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STOPPED 2024-03-24T11:54:39.303 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STARTING 2024-03-24T11:54:39.336 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 podman[70356]: 2024-03-24 11:54:39.028252188 +0000 UTC m=+0.423163861 container init 01e74f4670046fc440ec4da06892b58b9ec3b208214e13a0328060135425a389 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:39.336 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 podman[70356]: 2024-03-24 11:54:39.035398783 +0000 UTC m=+0.430310441 container start 01e74f4670046fc440ec4da06892b58b9ec3b208214e13a0328060135425a389 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.071Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.071Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.071Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi193 (none))" 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.071Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.071Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.074Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.076Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.078Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.079Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.080Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.080Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.978µs 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.080Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:54:39.337 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.080Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.081Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.081Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.958µs wal_replay_duration=814.819µs wbl_replay_duration=155ns total_replay_duration=873.258µs 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.082Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.082Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.082Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.109Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=27.276646ms db_storage=980ns remote_storage=1.372µs web_handler=518ns query_engine=749ns scrape=636.374µs scrape_sd=117.374µs notify=16.947µs notify_sd=13.615µs rules=26.153314ms tracing=8.11µs 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.109Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T11:54:39.109Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 bash[70356]: 01e74f4670046fc440ec4da06892b58b9ec3b208214e13a0328060135425a389 2024-03-24T11:54:39.338 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 11:54:39 smithi193 systemd[1]: Started Ceph prometheus.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T11:54:39.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:39.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T11:54:39.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:39.417 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:39.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:39.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:39.549 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:39.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:39.595 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Serving on http://:::9283 2024-03-24T11:54:39.595 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STARTED 2024-03-24T11:54:39.595 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STOPPING 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STOPPED 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STARTING 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Serving on http://:::9283 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STARTED 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STOPPING 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STOPPED 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STARTING 2024-03-24T11:54:39.596 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Serving on http://:::9283 2024-03-24T11:54:39.652 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:39.682 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:39.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:39.700 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 4 16 11573 11557 180.544 158.75 0.00666944 0.0055298 2024-03-24T11:54:39.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:39.889 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:39.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:39.933 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:39.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:40.036 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:39 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:11:54:39] ENGINE Bus STARTED 2024-03-24T11:54:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:40.046 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:40.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:40.098 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:40.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:40.180 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:40.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:40.282 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:40.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: pgmap v79: 148 pgs: 148 active+clean; 95 MiB data, 314 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 13 MiB/s wr, 201 op/s 2024-03-24T11:54:40.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: pgmap v79: 148 pgs: 148 active+clean; 95 MiB data, 314 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 13 MiB/s wr, 201 op/s 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:40.315 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:40.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:40.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: pgmap v79: 148 pgs: 148 active+clean; 95 MiB data, 314 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 13 MiB/s wr, 201 op/s 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:54:40.449 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:54:40.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:54:40.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:54:40.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:54:40.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mon.0 v1:172.21.15.64:0/1953277302' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:54:40.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:40.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:40.462 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:40.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:40.565 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:40.601 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:40.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:40.701 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 5 16 13963 13947 174.308 149.375 0.00642552 0.00572759 2024-03-24T11:54:40.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:40.749 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:40.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:40.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:40.892 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:40.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:40.993 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:41.031 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:41.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:41.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:41.176 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:41.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:41.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:41.319 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:41.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:41.420 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:41.452 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:41.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:41.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:41.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:41.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:41.589 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:41.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:41.630 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:41.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:41.701 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 6 16 16273 16257 169.309 144.375 0.00425163 0.00590005 2024-03-24T11:54:41.739 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:41.739 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:41.739 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:41.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:41.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:41.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:41.884 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:41.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:41.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:42.018 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:42.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:42.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:42.158 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:42.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:42.260 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:42.293 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:42.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:42.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:42.434 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:42.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:42.536 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:42.568 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:42.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:42.657 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-24T11:54:42.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:42.701 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 7 16 18786 18770 167.558 157.062 0.00409493 0.00596063 2024-03-24T11:54:42.709 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:42.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:42.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:42.845 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:42.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:42.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:42.983 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:42.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:42 smithi064 ceph-mon[30223]: pgmap v80: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:54:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/958149544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:54:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:42 smithi064 ceph-mon[30223]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:54:43.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:42 smithi064 ceph-mon[42243]: pgmap v80: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:54:43.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/958149544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:54:43.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:42 smithi064 ceph-mon[42243]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:54:43.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:42 smithi193 ceph-mon[33593]: pgmap v80: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:54:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/958149544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:54:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:42 smithi193 ceph-mon[33593]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:54:43.126 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:43.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:43.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:43.262 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:43.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:43.365 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:43.399 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:43.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:43.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:43.538 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:43.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:43.551 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 11:54:43 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T11:54:43.130Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004037843s 2024-03-24T11:54:43.639 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:43.662 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:43.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:43.701 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 8 16 20932 20916 163.37 134.125 0.00789655 0.00611467 2024-03-24T11:54:43.764 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:43.796 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:43.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:43.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:43 smithi064 ceph-mon[30223]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:54:43.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:43 smithi064 ceph-mon[30223]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:54:43.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:43 smithi064 ceph-mon[42243]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:54:43.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:43 smithi064 ceph-mon[42243]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:54:43.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:43.929 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:43.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:44.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:44.079 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:44.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:43 smithi193 ceph-mon[33593]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:54:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:43 smithi193 ceph-mon[33593]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:54:44.179 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:44.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:44.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:44.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:44.355 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:44.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:44.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:44.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:44.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:44.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:44.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:44.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:44.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 9 16 23329 23313 161.863 149.812 0.00367196 0.00617099 2024-03-24T11:54:44.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:44.765 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:44.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:44.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-24T11:54:44.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:44.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:45.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:44 smithi064 ceph-mon[30223]: pgmap v82: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:54:45.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:44 smithi064 ceph-mon[42243]: pgmap v82: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:54:45.011 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:45.043 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:45.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:45.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:44 smithi193 ceph-mon[33593]: pgmap v82: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:54:45.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:45.191 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:45.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:45.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:45.326 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:45.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:45.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:45.468 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:45.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:45.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:45.601 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:45.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:45.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 10 16 25415 25399 158.713 130.375 0.0034263 0.00629247 2024-03-24T11:54:45.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:45.734 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:45.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:45.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:45.880 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:45.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:45.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:46.014 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:46.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:46.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:46.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:46.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:46.301 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:46.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:46.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:46.428 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:46.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:46.530 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:46.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:46.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:46.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:46.690 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:46.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:46.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 11 16 27203 27187 154.443 111.75 0.030916 0.00645381 2024-03-24T11:54:46.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:46.833 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:46.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:46.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:46.967 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:46.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:54:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:54:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[30223]: pgmap v83: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-24T11:54:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[30223]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:54:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:47.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:54:47.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:54:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[42243]: pgmap v83: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-24T11:54:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[42243]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:54:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:47.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:46 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:47.069 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:47.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:54:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:54:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:46 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:54:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:46 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:54:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:46 smithi193 ceph-mon[33593]: pgmap v83: 148 pgs: 148 active+clean; 940 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-24T11:54:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:46 smithi193 ceph-mon[33593]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:54:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:46 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:46 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:54:47.100 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:47.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:47.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:47.227 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:47.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:47.328 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:47.358 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:47.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:47.460 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:47.489 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:47.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:47.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:47.616 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:47.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:47.686 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:54:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:47.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 12 16 28608 28592 148.89 87.8125 0.00253492 0.00662597 2024-03-24T11:54:47.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:47.757 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:47.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:47.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:47.894 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:47.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:47.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:47.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:48.021 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:48.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:48.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:47 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:47 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/434997228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:54:48.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:47 smithi064 ceph-mon[42243]: from='client.34754 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:54:48.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:47 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:48.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:47 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/434997228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:54:48.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:47 smithi064 ceph-mon[30223]: from='client.34754 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:54:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:47 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:47 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/434997228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:54:48.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:47 smithi193 ceph-mon[33593]: from='client.34754 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:54:48.124 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:48.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:48.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:48.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:48.290 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:48.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:48.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:48.421 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:48.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:48.523 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:48.554 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:48.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:48.655 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:48.686 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:48.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:48.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 13 16 30086 30070 144.543 92.375 0.0120814 0.00691225 2024-03-24T11:54:48.788 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:48.811 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:48.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:48.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:48.953 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:48.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[30223]: pgmap v85: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 194 MiB/s wr, 3.10k op/s 2024-03-24T11:54:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[30223]: from='client.34754 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:54:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[30223]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:54:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T11:54:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T11:54:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T11:54:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T11:54:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[42243]: pgmap v85: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 194 MiB/s wr, 3.10k op/s 2024-03-24T11:54:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[42243]: from='client.34754 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:54:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[42243]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:54:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T11:54:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T11:54:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T11:54:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:48 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T11:54:49.055 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:49.083 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:48 smithi193 ceph-mon[33593]: pgmap v85: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 194 MiB/s wr, 3.10k op/s 2024-03-24T11:54:49.083 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:48 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:54:49.083 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:48 smithi193 ceph-mon[33593]: from='client.34754 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:54:49.083 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:48 smithi193 ceph-mon[33593]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:54:49.083 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:48 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T11:54:49.083 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:48 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T11:54:49.083 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:48 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T11:54:49.083 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:48 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T11:54:49.089 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:49.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:49.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:49.215 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:49.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:49.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:49.349 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:49.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:49.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:49.480 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:49.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:49.581 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:49.618 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:49.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:49.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 14 16 31694 31678 141.397 100.5 0.0126472 0.00706576 2024-03-24T11:54:49.719 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:49.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:49.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:49.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:49.882 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:49.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:49.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:50.006 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:50.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:50.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T11:54:50.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T11:54:50.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T11:54:50.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T11:54:50.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T11:54:50.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T11:54:50.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T11:54:50.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T11:54:50.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:49 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T11:54:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:49 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T11:54:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:49 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T11:54:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:49 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T11:54:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:49 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T11:54:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:49 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T11:54:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:49 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T11:54:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:49 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T11:54:50.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:49 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T11:54:50.108 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:50.143 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:50.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:50.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:50.269 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:50.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:50.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:50.406 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:50.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:50.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:50.546 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:50.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:50.648 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:50.681 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:50.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:50.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 15 15 33148 33133 138.032 90.9375 0.00542963 0.0072399 2024-03-24T11:54:50.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:50.824 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:50.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:50.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:50.952 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:50.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[30223]: pgmap v87: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 786 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-24T11:54:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[30223]: 6.13 scrub starts 2024-03-24T11:54:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[30223]: 6.13 scrub ok 2024-03-24T11:54:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T11:54:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T11:54:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T11:54:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T11:54:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[42243]: pgmap v87: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 786 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-24T11:54:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[42243]: 6.13 scrub starts 2024-03-24T11:54:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[42243]: 6.13 scrub ok 2024-03-24T11:54:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T11:54:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T11:54:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T11:54:51.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:50 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T11:54:51.053 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:51.083 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:51.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:50 smithi193 ceph-mon[33593]: pgmap v87: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 786 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-24T11:54:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:50 smithi193 ceph-mon[33593]: 6.13 scrub starts 2024-03-24T11:54:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:50 smithi193 ceph-mon[33593]: 6.13 scrub ok 2024-03-24T11:54:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:50 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T11:54:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:50 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T11:54:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:50 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T11:54:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:50 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T11:54:51.185 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:51.214 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:51.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:51.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:51.354 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:51.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:51.455 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:51.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:51.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:51.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:51.623 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:51.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:51.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 16 16 34915 34899 136.303 110.375 0.00361092 0.00733122 2024-03-24T11:54:51.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:51.755 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:51.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:51.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:51.898 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:51.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:52.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:52.042 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:52.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:52.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:51 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T11:54:52.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:51 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T11:54:52.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:51 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T11:54:52.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:51 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T11:54:52.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:51 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T11:54:52.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:51 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T11:54:52.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:52.183 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:52.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:52.197 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T11:54:52.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:52.317 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:52.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:52.419 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:52.452 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:52.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:52.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:52.609 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:52.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:52.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 17 16 36863 36847 135.446 121.75 0.00546978 0.00737589 2024-03-24T11:54:52.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:52.790 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:52.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:52.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:52.924 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:52.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: pgmap v88: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: 6.11 scrub starts 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: 6.11 scrub ok 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T11:54:52.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: pgmap v88: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: 6.11 scrub starts 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: 6.11 scrub ok 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T11:54:52.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:52 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T11:54:53.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:53.060 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:53.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: pgmap v88: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: 6.11 scrub starts 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: 6.11 scrub ok 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T11:54:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:52 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T11:54:53.162 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:53.196 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:53.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:53.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:53.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:53.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:53.469 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:53.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:53.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:53.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:53.611 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:53.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:53.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 18 16 38690 38674 134.265 114.188 0.0114478 0.0074427 2024-03-24T11:54:53.713 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:53.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:53.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:53.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:53.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:53.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:53.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T11:54:53.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T11:54:53.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T11:54:53.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T11:54:53.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T11:54:53.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T11:54:53.942 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T11:54:53.942 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T11:54:53.943 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T11:54:53.943 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T11:54:53.943 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T11:54:53.943 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:53 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T11:54:53.981 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:53.998 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:54.023 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:39170 7.0 e108/97 7:0bae0beb:::benchmark_data_smithi064_87013_object39169:head v 108'2541, mlcod=108'2535)", 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:54.020949+0000", 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0020631820000000002, 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020990610000000002, 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:54:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:54.020949+0000", 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:54.020949+0000", 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:54.020951+0000", 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4580000000000001e-06 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:54.021040+0000", 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9404e-05 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:54.021057+0000", 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6874000000000001e-05 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:54.021066+0000", 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5180000000000004e-06 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:54.021263+0000", 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019685699999999999 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:54.021283+0000", 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0506e-05 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-03-24T11:54:54.027 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:54:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:54.029 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:54.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:54.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:53 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T11:54:54.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:53 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T11:54:54.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:53 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T11:54:54.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:53 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T11:54:54.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:53 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T11:54:54.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:53 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T11:54:54.130 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:54.163 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:54.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:54.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:54.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:54.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:54.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:54.451 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:54.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:54.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:54.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:54.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:54.702 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 19 16 40391 40375 132.793 106.312 0.00424571 0.00752439 2024-03-24T11:54:54.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:54.745 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:54.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:54.832 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:54 smithi064 ceph-mon[30223]: 2.2 scrub starts 2024-03-24T11:54:54.833 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:54 smithi064 ceph-mon[30223]: 2.2 scrub ok 2024-03-24T11:54:54.833 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:54 smithi064 ceph-mon[30223]: pgmap v89: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-24T11:54:54.833 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:54 smithi064 ceph-mon[42243]: 2.2 scrub starts 2024-03-24T11:54:54.833 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:54 smithi064 ceph-mon[42243]: 2.2 scrub ok 2024-03-24T11:54:54.833 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:54 smithi064 ceph-mon[42243]: pgmap v89: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-24T11:54:54.847 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:54.878 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:54.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:54.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:55.022 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:55.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:55.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:54 smithi193 ceph-mon[33593]: 2.2 scrub starts 2024-03-24T11:54:55.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:54 smithi193 ceph-mon[33593]: 2.2 scrub ok 2024-03-24T11:54:55.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:54 smithi193 ceph-mon[33593]: pgmap v89: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-24T11:54:55.124 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:55.156 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:55.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:55.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:55.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:55.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:55.455 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:55.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:55.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:55.601 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:55.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:55.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:54:55.703 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T11:54:55.704403+0000 min lat: 0.00141856 max lat: 0.171312 avg lat: 0.00759497 2024-03-24T11:54:55.703 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:54:55.703 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 20 16 42109 42093 131.519 107.375 0.0168541 0.00759497 2024-03-24T11:54:55.703 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:55.750 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:55.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:55.837 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:54:55.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:55.878 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:54:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:55.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:55.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:55.934 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:54:55.976 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:54:55.976 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:55 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T11:54:55.976 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:55 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T11:54:55.976 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:55.979 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:55 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T11:54:55.979 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:55 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T11:54:55.979 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:55.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:56.020 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:54:56.030 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:56.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:56.056 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:54:56.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:55 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T11:54:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:55 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T11:54:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:56.100 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:54:56.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:56.145 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:54:56.154 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:54:56.154 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:54:56.155 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:54:56.155 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:54:56.155 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:54:56.155 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:54:56.155 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:54:56.155 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:54:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:56.186 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:56.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:56.287 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:56.318 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:56.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:56.420 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:56.452 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:56.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:56.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:56.582 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:56.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:56.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:56.703 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 21 16 43704 43688 130.003 99.6875 0.0205399 0.00768431 2024-03-24T11:54:56.712 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:56.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:56.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T11:54:56.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:56.862 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:56.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:56.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:57.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:57.032 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:57.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[30223]: 6.19 scrub starts 2024-03-24T11:54:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[30223]: 6.19 scrub ok 2024-03-24T11:54:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[30223]: pgmap v90: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1136539256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[42243]: 6.19 scrub starts 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[42243]: 6.19 scrub ok 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[42243]: pgmap v90: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1136539256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T11:54:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:56 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T11:54:57.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:54:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:54:57.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:56 smithi193 ceph-mon[33593]: 6.19 scrub starts 2024-03-24T11:54:57.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:56 smithi193 ceph-mon[33593]: 6.19 scrub ok 2024-03-24T11:54:57.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:56 smithi193 ceph-mon[33593]: pgmap v90: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-03-24T11:54:57.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:56 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1136539256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:54:57.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:56 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T11:54:57.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:56 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T11:54:57.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:57.188 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:57.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:57.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:57.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:57.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:57.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:57.584 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:57.615 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:57.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:57.703 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 22 16 45282 45266 128.577 98.625 0.00868867 0.00777206 2024-03-24T11:54:57.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:54:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:57.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:54:57.781 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:54:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:57.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:57.895 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:57.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:57.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:54:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:54:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[42243]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:54:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[42243]: 4.1b scrub starts 2024-03-24T11:54:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[42243]: 4.1b scrub ok 2024-03-24T11:54:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[30223]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[30223]: 4.1b scrub starts 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[30223]: 4.1b scrub ok 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T11:54:58.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:57 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T11:54:58.057 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:58.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:58.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:57 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:54:58.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:57 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:54:58.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:57 smithi193 ceph-mon[33593]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:54:58.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:57 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:58.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:57 smithi193 ceph-mon[33593]: 4.1b scrub starts 2024-03-24T11:54:58.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:57 smithi193 ceph-mon[33593]: 4.1b scrub ok 2024-03-24T11:54:58.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:57 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T11:54:58.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:57 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T11:54:58.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:58.190 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:58.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:58.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:58.323 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:58.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:58.424 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:58.453 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:58.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:58.555 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:54:58.566 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:58.573 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:54:58.582 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:58.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:54:58.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:54:58.707 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 23 16 46819 46803 127.141 96.0625 0.0115079 0.00785892 2024-03-24T11:54:58.734 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:58.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:54:58.836 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:58.866 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:58.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:58.967 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[30223]: pgmap v92: 148 pgs: 148 active+clean; 2.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[30223]: 4.0 scrub starts 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[30223]: 4.0 scrub ok 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[42243]: pgmap v92: 148 pgs: 148 active+clean; 2.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[42243]: 4.0 scrub starts 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[42243]: 4.0 scrub ok 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T11:54:58.983 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:58 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T11:54:58.996 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:58.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:59.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:58 smithi193 ceph-mon[33593]: pgmap v92: 148 pgs: 148 active+clean; 2.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-24T11:54:59.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:58 smithi193 ceph-mon[33593]: 4.0 scrub starts 2024-03-24T11:54:59.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:58 smithi193 ceph-mon[33593]: 4.0 scrub ok 2024-03-24T11:54:59.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:58 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T11:54:59.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:58 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T11:54:59.098 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:54:59.129 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:59.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:54:59.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:59.260 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:59.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:59.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:54:59.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:59.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:54:59.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:59.558 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:59.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:54:59.659 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:54:59.692 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:59.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:54:59.707 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 24 15 48361 48346 125.861 96.4375 0.00573672 0.00794032 2024-03-24T11:54:59.794 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:54:59.800 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-24T11:54:59.820 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: 6.1f scrub ok 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2133669854' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:54:59.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[30223]: from='client.24769 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: 6.1f scrub ok 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2133669854' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:54:59.822 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:54:59 smithi064 ceph-mon[42243]: from='client.24769 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:54:59.824 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:54:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:59.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:54:59.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:54:59.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:54:59.962 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:54:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:54:59.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:00.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T11:55:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T11:55:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T11:55:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: 6.1f scrub ok 2024-03-24T11:55:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T11:55:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T11:55:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T11:55:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T11:55:00.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2133669854' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:55:00.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:54:59 smithi193 ceph-mon[33593]: from='client.24769 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:55:00.108 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:00.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:00.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:00.282 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:00.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:00.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:00.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:00.534 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:00.565 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:00.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:00.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:00.697 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:00.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:00.707 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 25 16 49893 49877 124.654 95.6875 0.0108883 0.00801734 2024-03-24T11:55:00.799 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:00.831 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:00.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:00.933 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:00.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: pgmap v93: 148 pgs: 148 active+clean; 2.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-24T11:55:00.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T11:55:00.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T11:55:00.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T11:55:00.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T11:55:00.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: from='client.24769 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:55:00.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: 2.18 scrub starts 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[30223]: 2.18 scrub ok 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: pgmap v93: 148 pgs: 148 active+clean; 2.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: from='client.24769 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: 2.18 scrub starts 2024-03-24T11:55:00.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:00 smithi064 ceph-mon[42243]: 2.18 scrub ok 2024-03-24T11:55:00.965 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:00.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:01.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: pgmap v93: 148 pgs: 148 active+clean; 2.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-24T11:55:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T11:55:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T11:55:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T11:55:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T11:55:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: from='client.24769 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:55:01.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:55:01.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T11:55:01.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T11:55:01.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: 2.18 scrub starts 2024-03-24T11:55:01.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:00 smithi193 ceph-mon[33593]: 2.18 scrub ok 2024-03-24T11:55:01.098 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:01.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:01.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:01.244 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:01.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:01.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:01.389 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:01.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:01.490 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:01.522 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:01.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:01.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:01.625 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:01.655 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:01.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:01.708 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 26 16 51389 51373 123.454 93.5 0.00627693 0.00809481 2024-03-24T11:55:01.757 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:01.787 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:01.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:01.889 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:01.918 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:01.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:02.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:02.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-24T11:55:02.048 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:02.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:02.077 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T11:55:02.077 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T11:55:02.077 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T11:55:02.077 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T11:55:02.077 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[30223]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:55:02.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T11:55:02.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T11:55:02.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T11:55:02.078 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T11:55:02.078 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:01 smithi064 ceph-mon[42243]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:55:02.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:01 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T11:55:02.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:01 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T11:55:02.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:01 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T11:55:02.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:01 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T11:55:02.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:01 smithi193 ceph-mon[33593]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:55:02.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:02.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:02.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:02.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:02.376 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:02.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:02.477 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:02.507 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:02.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:02.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:02.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:02.659 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:02.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:02.708 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 27 16 52550 52534 121.569 72.5625 0.0236146 0.00822019 2024-03-24T11:55:02.761 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:02.790 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:02.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:02.864 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T11:55:02.892 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:02.918 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:02.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:03.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:03.047 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:03.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: pgmap v95: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T11:55:03.108 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2170481280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: pgmap v95: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T11:55:03.109 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T11:55:03.110 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T11:55:03.110 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T11:55:03.110 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:02 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2170481280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:03.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:03.197 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:03.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:03.235 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:55:03.235 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:55:03.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: pgmap v95: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T11:55:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T11:55:03.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T11:55:03.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T11:55:03.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:02 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2170481280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:03.340 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:03.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:03.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:03.471 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:03.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:03.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:03.612 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:03.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:03.709 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 28 16 53876 53860 120.187 82.875 0.0206658 0.0083151 2024-03-24T11:55:03.714 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:03.754 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:03.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:03.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:03.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:03.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:03.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:04.028 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:04.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2170481280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T11:55:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2170481280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T11:55:04.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:03 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T11:55:04.130 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:04.158 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:04.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:04.259 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:04.291 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:04.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2170481280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T11:55:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:03 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T11:55:04.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:04.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:04.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:04.613 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:04.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:04.711 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 29 16 55375 55359 119.27 93.6875 0.00825735 0.00837913 2024-03-24T11:55:04.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:04.743 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:04.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:04.847 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:04.879 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:04.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:04.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:05.013 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:05.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[42243]: pgmap v98: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[30223]: pgmap v98: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T11:55:05.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:04 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T11:55:05.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:05.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T11:55:05.163 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:05.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:05.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:05.326 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:05.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:04 smithi193 ceph-mon[33593]: pgmap v98: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:04 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T11:55:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:04 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T11:55:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:04 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T11:55:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:04 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T11:55:05.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:05.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:05.639 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:05.672 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:05.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:05.709 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 30 16 56769 56753 118.198 87.125 0.00554002 0.00845529 2024-03-24T11:55:05.773 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:05.805 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:05.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:05.907 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:05.938 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:05.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:06.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:05 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T11:55:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:05 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T11:55:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:05 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:55:06.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:05 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:55:06.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:05 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T11:55:06.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:05 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T11:55:06.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:05 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:55:06.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:05 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:55:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:06.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:06.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:06.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:55:06.210 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:06.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:06.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:05 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T11:55:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:05 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T11:55:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:05 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:55:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:05 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:55:06.344 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:06.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:06.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:06.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:06.503 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:06.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:06.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:06.680 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:06.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:06.710 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 31 16 58130 58114 117.128 85.0625 0.00922344 0.0085323 2024-03-24T11:55:06.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:06.813 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:06.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:06.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: pgmap v99: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:06.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:55:06.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: 4.1 scrub starts 2024-03-24T11:55:06.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: 4.1 scrub ok 2024-03-24T11:55:06.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T11:55:06.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T11:55:06.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: 4.14 scrub starts 2024-03-24T11:55:06.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: 4.14 scrub ok 2024-03-24T11:55:06.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[30223]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:55:06.914 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:06.945 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:06.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:07.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:07.088 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:07.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:07.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:07.229 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:07.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: pgmap v99: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:55:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: 4.1 scrub starts 2024-03-24T11:55:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: 4.1 scrub ok 2024-03-24T11:55:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T11:55:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T11:55:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: 4.14 scrub starts 2024-03-24T11:55:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: 4.14 scrub ok 2024-03-24T11:55:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:06 smithi064 ceph-mon[42243]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:55:07.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:07.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: pgmap v99: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: 4.1 scrub starts 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: 4.1 scrub ok 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: 4.14 scrub starts 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: 4.14 scrub ok 2024-03-24T11:55:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:06 smithi193 ceph-mon[33593]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:55:07.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:07.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:07.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:07.518 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:07.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:07.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:07.648 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:07.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:07.710 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 32 16 59608 59592 116.354 92.375 0.00718441 0.00858891 2024-03-24T11:55:07.750 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:07.781 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:07.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:07.802 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:55:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:07.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:07.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:07.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:08.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:08.077 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:08.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:08.179 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:08.209 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:08.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[30223]: 5.1e scrub starts 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[30223]: 5.1e scrub ok 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[42243]: 5.1e scrub starts 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[42243]: 5.1e scrub ok 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T11:55:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T11:55:08.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T11:55:08.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:07 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T11:55:08.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:07 smithi193 ceph-mon[33593]: 5.1e scrub starts 2024-03-24T11:55:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:07 smithi193 ceph-mon[33593]: 5.1e scrub ok 2024-03-24T11:55:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:07 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:07 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T11:55:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:07 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T11:55:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:07 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T11:55:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:07 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T11:55:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:08.363 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:08.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:08.464 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:08.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:08.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:08.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:08.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:08.644 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:08.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:08.710 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 33 16 60870 60854 115.218 78.875 0.0568403 0.00867222 2024-03-24T11:55:08.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:08.801 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:08.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:08.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:08.950 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:08.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:09.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:09.090 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:09.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:09.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:09.230 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:09.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[30223]: pgmap v102: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 927 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-24T11:55:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T11:55:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T11:55:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[42243]: pgmap v102: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 927 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-24T11:55:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T11:55:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T11:55:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:08 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:09.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:08 smithi193 ceph-mon[33593]: pgmap v102: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 927 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-24T11:55:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:08 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T11:55:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:08 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T11:55:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:08 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:08 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:09.372 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:09.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:09.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:09.502 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:09.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:09.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:09.644 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:09.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:09.710 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 34 16 62247 62231 114.361 86.0625 0.0043649 0.00873795 2024-03-24T11:55:09.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:09.789 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:09.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:09.891 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:09.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:10.058 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:10.089 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:10.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:10.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:10.235 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:10.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:10.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:10.365 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:10.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:10.467 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:10.475 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:55:10.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:10.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:10.525 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:55:10.589 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:55:10.598 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:10.627 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:10.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:10.634 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:55:10.663 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:10.681 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:55:10.718 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 35 16 63423 63407 113.176 73.5 0.0303133 0.00882851 2024-03-24T11:55:10.729 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:10.734 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:55:10.756 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:10.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:10.775 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:55:10.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:55:10.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:55:10.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:55:10.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:55:10.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:55:10.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:55:10.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:55:10.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:55:10.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:55:10.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:10.916 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:10.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:11.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:11.046 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:11.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:11.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:11.178 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:11.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:11.280 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: pgmap v103: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 811 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2581672808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: 6.7 scrub starts 2024-03-24T11:55:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[30223]: 6.7 scrub ok 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: pgmap v103: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 811 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2581672808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: 6.7 scrub starts 2024-03-24T11:55:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:10 smithi064 ceph-mon[42243]: 6.7 scrub ok 2024-03-24T11:55:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:11.304 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:11.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: pgmap v103: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 811 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2581672808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: 6.7 scrub starts 2024-03-24T11:55:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:10 smithi193 ceph-mon[33593]: 6.7 scrub ok 2024-03-24T11:55:11.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:11.442 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:11.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:11.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:11.544 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:11.573 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:11.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:11.650 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-03-24T11:55:11.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:11.717 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 36 16 64522 64506 111.937 68.6875 0.00511064 0.00892739 2024-03-24T11:55:11.717 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:11.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:11.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:11.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:11.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:11.989 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:11.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:12.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/488131015' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/488131015' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[30223]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[30223]: 6.d scrub starts 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[30223]: 6.d scrub ok 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/488131015' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/488131015' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[42243]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[42243]: 6.d scrub starts 2024-03-24T11:55:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:11 smithi064 ceph-mon[42243]: 6.d scrub ok 2024-03-24T11:55:12.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:12.117 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:12.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:12.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:12.264 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:12.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/488131015' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:55:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/488131015' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:55:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:11 smithi193 ceph-mon[33593]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:55:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:11 smithi193 ceph-mon[33593]: 6.d scrub starts 2024-03-24T11:55:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:11 smithi193 ceph-mon[33593]: 6.d scrub ok 2024-03-24T11:55:12.365 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:12.394 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:12.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:12.496 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:12.519 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:12.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:12.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:12.650 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:12.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:12.717 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 37 16 65832 65816 111.124 81.875 0.00423146 0.0089933 2024-03-24T11:55:12.752 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:12.775 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:12.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:12.877 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:12.907 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:12.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:13.007 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:13.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:13.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:13.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:13.164 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:13.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:13.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[30223]: pgmap v104: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[30223]: 6.17 scrub starts 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[30223]: 6.17 scrub ok 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[42243]: pgmap v104: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T11:55:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T11:55:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[42243]: 6.17 scrub starts 2024-03-24T11:55:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[42243]: 6.17 scrub ok 2024-03-24T11:55:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T11:55:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:12 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T11:55:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:13.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:12 smithi193 ceph-mon[33593]: pgmap v104: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:55:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:12 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T11:55:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:12 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T11:55:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:12 smithi193 ceph-mon[33593]: 6.17 scrub starts 2024-03-24T11:55:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:12 smithi193 ceph-mon[33593]: 6.17 scrub ok 2024-03-24T11:55:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:12 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T11:55:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:12 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T11:55:13.405 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:13.435 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:13.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:13.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:13.565 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:13.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:13.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:13.695 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:13.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:13.717 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 38 15 66841 66826 109.862 63.125 0.0122822 0.00909718 2024-03-24T11:55:13.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:13.836 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:13.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:13.938 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:13.961 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:13.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:14.063 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:14.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:14.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:14.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:14.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:14.235 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:14.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:14.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:13 smithi064 ceph-mon[30223]: 4.1c deep-scrub starts 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 4.1c deep-scrub ok 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:13 smithi064 ceph-mon[42243]: 4.1c deep-scrub starts 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 4.1c deep-scrub ok 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T11:55:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T11:55:14.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T11:55:14.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:13 smithi193 ceph-mon[33593]: 4.1c deep-scrub starts 2024-03-24T11:55:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:13 smithi193 ceph-mon[33593]: 4.1c deep-scrub ok 2024-03-24T11:55:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:13 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T11:55:14.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:13 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T11:55:14.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:13 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T11:55:14.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:13 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T11:55:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:14.360 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:14.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:14.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:14.500 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:14.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:14.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:14.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:14.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:14.717 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 39 16 67678 67662 108.385 52.25 0.00388547 0.00921386 2024-03-24T11:55:14.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:14.760 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:14.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:14.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:14.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:14.892 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:14.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:14.994 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:15.017 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:15.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:15.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:15.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:15.159 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:15.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:15.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:15.290 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:15.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: pgmap v106: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-24T11:55:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 5.6 deep-scrub starts 2024-03-24T11:55:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 5.6 deep-scrub ok 2024-03-24T11:55:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T11:55:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T11:55:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: pgmap v106: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-24T11:55:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 5.6 deep-scrub starts 2024-03-24T11:55:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 5.6 deep-scrub ok 2024-03-24T11:55:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T11:55:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:14 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T11:55:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:14 smithi193 ceph-mon[33593]: pgmap v106: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-24T11:55:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:14 smithi193 ceph-mon[33593]: 5.6 deep-scrub starts 2024-03-24T11:55:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:14 smithi193 ceph-mon[33593]: 5.6 deep-scrub ok 2024-03-24T11:55:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:14 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T11:55:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:14 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T11:55:15.391 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:15.420 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:15.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:15.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:15.551 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:15.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:15.652 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:15.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:15.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:15.717 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T11:55:15.718541+0000 min lat: 0.00141856 max lat: 0.234923 avg lat: 0.00932082 2024-03-24T11:55:15.717 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:55:15.717 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 40 16 68632 68616 107.166 59.625 0.0427488 0.00932082 2024-03-24T11:55:15.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:15.812 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:15.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:15.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:15.939 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:15.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:16.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:16.081 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:16.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:16.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:16.213 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:16.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T11:55:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T11:55:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T11:55:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T11:55:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:15 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T11:55:16.315 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:15 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T11:55:16.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:15 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T11:55:16.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:15 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T11:55:16.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:15 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T11:55:16.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:15 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T11:55:16.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:15 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T11:55:16.347 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:16.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:16.449 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:16.482 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:16.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:16.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:16.622 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:16.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:16.717 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 41 16 69841 69825 106.395 75.5625 0.0180608 0.0093929 2024-03-24T11:55:16.725 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:16.755 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:16.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:16.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:16.893 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:16.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:16.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:16.995 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:17.026 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:17.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:17.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:17.171 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:17.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:17.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:17.301 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:16 smithi064 ceph-mon[30223]: pgmap v107: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 600 B/s rd, 48 MiB/s wr, 768 op/s 2024-03-24T11:55:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:16 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T11:55:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:16 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T11:55:17.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:16 smithi064 ceph-mon[42243]: pgmap v107: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 600 B/s rd, 48 MiB/s wr, 768 op/s 2024-03-24T11:55:17.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:16 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T11:55:17.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:16 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T11:55:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:17.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:17.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:55:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:55:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:16 smithi193 ceph-mon[33593]: pgmap v107: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 600 B/s rd, 48 MiB/s wr, 768 op/s 2024-03-24T11:55:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:16 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T11:55:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:16 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T11:55:17.404 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:17.435 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:17.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:17.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:17.601 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:17.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:17.662 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:55:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:17.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:17.718 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 42 16 71136 71120 105.787 80.9375 0.00636107 0.00944738 2024-03-24T11:55:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:17.743 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:17.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:17.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:17.887 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:17.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:17.989 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:18.027 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:18.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:18.124 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:18.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:18.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-24T11:55:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:18.173 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:18.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:18.225 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[30223]: 4.16 deep-scrub starts 2024-03-24T11:55:18.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[30223]: 4.16 deep-scrub ok 2024-03-24T11:55:18.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:18.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[30223]: 6.a scrub starts 2024-03-24T11:55:18.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[30223]: 6.a scrub ok 2024-03-24T11:55:18.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[42243]: 4.16 deep-scrub starts 2024-03-24T11:55:18.227 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[42243]: 4.16 deep-scrub ok 2024-03-24T11:55:18.227 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:18.227 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[42243]: 6.a scrub starts 2024-03-24T11:55:18.227 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:17 smithi064 ceph-mon[42243]: 6.a scrub ok 2024-03-24T11:55:18.274 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:18.303 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:18.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:17 smithi193 ceph-mon[33593]: 4.16 deep-scrub starts 2024-03-24T11:55:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:17 smithi193 ceph-mon[33593]: 4.16 deep-scrub ok 2024-03-24T11:55:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:17 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:17 smithi193 ceph-mon[33593]: 6.a scrub starts 2024-03-24T11:55:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:17 smithi193 ceph-mon[33593]: 6.a scrub ok 2024-03-24T11:55:18.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:18.450 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:18.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:18.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:18.598 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:18.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:18.699 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:18.718 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 43 16 72340 72324 105.076 75.25 0.0102296 0.00951136 2024-03-24T11:55:18.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:18.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:18.828 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:18.862 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:18.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:18.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:19.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[30223]: pgmap v108: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:55:19.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[30223]: 6.8 deep-scrub starts 2024-03-24T11:55:19.004 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[30223]: 6.8 deep-scrub ok 2024-03-24T11:55:19.004 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[30223]: 2.16 scrub starts 2024-03-24T11:55:19.004 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[30223]: 2.16 scrub ok 2024-03-24T11:55:19.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[42243]: pgmap v108: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:55:19.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[42243]: 6.8 deep-scrub starts 2024-03-24T11:55:19.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[42243]: 6.8 deep-scrub ok 2024-03-24T11:55:19.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[42243]: 2.16 scrub starts 2024-03-24T11:55:19.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:18 smithi064 ceph-mon[42243]: 2.16 scrub ok 2024-03-24T11:55:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:19.014 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:19.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:19.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:19.156 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:19.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:19.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:18 smithi193 ceph-mon[33593]: pgmap v108: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:55:19.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:18 smithi193 ceph-mon[33593]: 6.8 deep-scrub starts 2024-03-24T11:55:19.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:18 smithi193 ceph-mon[33593]: 6.8 deep-scrub ok 2024-03-24T11:55:19.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:18 smithi193 ceph-mon[33593]: 2.16 scrub starts 2024-03-24T11:55:19.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:18 smithi193 ceph-mon[33593]: 2.16 scrub ok 2024-03-24T11:55:19.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:19.289 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:19.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:19.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:19.433 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:19.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:19.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:19.569 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:19.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:19.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:55:19.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:19.719 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 44 16 73543 73527 104.397 75.1875 0.0134312 0.00957221 2024-03-24T11:55:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:19.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:19.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:19.821 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:19.856 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:19.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:19.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:19.966 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 4.c scrub starts 2024-03-24T11:55:19.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 4.c scrub ok 2024-03-24T11:55:19.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:55:19.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:55:19.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 6.1d scrub starts 2024-03-24T11:55:19.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 6.1d scrub ok 2024-03-24T11:55:19.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:55:19.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T11:55:19.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T11:55:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:20.020 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:20.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:20.121 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:20.155 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:20.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:20.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 4.1d scrub starts 2024-03-24T11:55:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 4.1d scrub ok 2024-03-24T11:55:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:55:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:55:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 4.8 scrub starts 2024-03-24T11:55:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[42243]: 4.8 scrub ok 2024-03-24T11:55:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 4.c scrub starts 2024-03-24T11:55:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 4.c scrub ok 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 6.1d scrub starts 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 6.1d scrub ok 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 4.1d scrub starts 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 4.1d scrub ok 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 4.8 scrub starts 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:19 smithi064 ceph-mon[30223]: 4.8 scrub ok 2024-03-24T11:55:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:20.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 4.c scrub starts 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 4.c scrub ok 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 6.1d scrub starts 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 6.1d scrub ok 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T11:55:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 4.1d scrub starts 2024-03-24T11:55:20.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 4.1d scrub ok 2024-03-24T11:55:20.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:55:20.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:55:20.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 4.8 scrub starts 2024-03-24T11:55:20.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:19 smithi193 ceph-mon[33593]: 4.8 scrub ok 2024-03-24T11:55:20.405 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:20.443 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:20.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:20.544 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:20.599 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:20.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:20.658 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:20.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:20.719 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 45 16 74727 74711 103.72 74 0.0237642 0.00963634 2024-03-24T11:55:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:20.755 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:20.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:20.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:20.888 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:20.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:20.929 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:20.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:21.039 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:21.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:21.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:21.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:21.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 6.10 deep-scrub starts 2024-03-24T11:55:21.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 6.10 deep-scrub ok 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: pgmap v110: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 4.d deep-scrub starts 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 4.d deep-scrub ok 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 2.1e scrub starts 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 2.1e scrub ok 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 6.10 deep-scrub starts 2024-03-24T11:55:21.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 6.10 deep-scrub ok 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: pgmap v110: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 4.d deep-scrub starts 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 4.d deep-scrub ok 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 2.1e scrub starts 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 2.1e scrub ok 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T11:55:21.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:20 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T11:55:21.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:21.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 6.10 deep-scrub starts 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 6.10 deep-scrub ok 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: pgmap v110: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 4.d deep-scrub starts 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 4.d deep-scrub ok 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 2.1e scrub starts 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 2.1e scrub ok 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T11:55:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:20 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T11:55:21.402 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:21.434 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:21.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:21.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:21.582 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:21.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:21.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:21.719 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 46 16 75923 75907 103.09 74.75 0.0199131 0.00969507 2024-03-24T11:55:21.730 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:21.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:21.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:21.861 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:21.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:21.962 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:21.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:22.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:22.142 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:22.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:22.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:22.274 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:22.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:22.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T11:55:22.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T11:55:22.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T11:55:22.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T11:55:22.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[30223]: 4.19 deep-scrub starts 2024-03-24T11:55:22.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[30223]: 4.19 deep-scrub ok 2024-03-24T11:55:22.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[30223]: 2.12 scrub starts 2024-03-24T11:55:22.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[30223]: 2.12 scrub ok 2024-03-24T11:55:22.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T11:55:22.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T11:55:22.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T11:55:22.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T11:55:22.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[42243]: 4.19 deep-scrub starts 2024-03-24T11:55:22.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[42243]: 4.19 deep-scrub ok 2024-03-24T11:55:22.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[42243]: 2.12 scrub starts 2024-03-24T11:55:22.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:22 smithi064 ceph-mon[42243]: 2.12 scrub ok 2024-03-24T11:55:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:22 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T11:55:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:22 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T11:55:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:22 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T11:55:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:22 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T11:55:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:22 smithi193 ceph-mon[33593]: 4.19 deep-scrub starts 2024-03-24T11:55:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:22 smithi193 ceph-mon[33593]: 4.19 deep-scrub ok 2024-03-24T11:55:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:22 smithi193 ceph-mon[33593]: 2.12 scrub starts 2024-03-24T11:55:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:22 smithi193 ceph-mon[33593]: 2.12 scrub ok 2024-03-24T11:55:22.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:22.404 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:22.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:22.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:22.537 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:22.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:22.639 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:22.670 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:22.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:22.720 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 47 16 77052 77036 102.398 70.5625 0.00874572 0.00976029 2024-03-24T11:55:22.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:22.803 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:22.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:22.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:22.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:23.013 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T11:55:23.047 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:23.074 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:23.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:23.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: pgmap v112: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:55:23.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:55:23.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:55:23.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:55:23.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T11:55:23.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T11:55:23.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T11:55:23.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T11:55:23.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T11:55:23.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T11:55:23.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1921240512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:55:23.155 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: pgmap v112: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T11:55:23.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:23 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1921240512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:55:23.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:23.206 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:23.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:23.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: pgmap v112: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T11:55:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:23 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1921240512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:55:23.354 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:23.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:23.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:23.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:23.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:23.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:23.620 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:23.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:23.720 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 48 16 78226 78210 101.793 73.375 0.00955266 0.0098182 2024-03-24T11:55:23.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:23.766 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:23.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:23.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:23.898 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:23.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:24.000 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:24.028 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:24.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:24.130 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:24.171 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:24.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:24.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T11:55:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T11:55:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1921240512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:55:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[30223]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:55:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[30223]: pgmap v115: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-24T11:55:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T11:55:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T11:55:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1921240512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:55:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[42243]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:55:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:24 smithi064 ceph-mon[42243]: pgmap v115: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-24T11:55:24.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:24.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:24.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:24 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T11:55:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:24 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T11:55:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:24 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:24 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1921240512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:55:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:24 smithi193 ceph-mon[33593]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:55:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:24 smithi193 ceph-mon[33593]: pgmap v115: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-24T11:55:24.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:24.466 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:24.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:24.567 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:24.597 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:24.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:24.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:24.721 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 49 15 79300 79285 101.084 67.1875 0.0133982 0.009887 2024-03-24T11:55:24.743 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:24.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:24.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:24.872 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:24.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:24.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:25.014 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:25.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:25.063 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.5 to 1 (10000) 2024-03-24T11:55:25.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:25.154 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:25.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:25.255 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:25.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:25.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:25.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:25 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/953411237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:55:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:25 smithi064 ceph-mon[30223]: from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:55:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:25 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/953411237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:55:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:25 smithi064 ceph-mon[42243]: from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:55:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:25 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/953411237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:55:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:25 smithi193 ceph-mon[33593]: from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:55:25.402 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:25.430 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:25.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:25.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:25.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:25.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:25.707 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:25.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:25.721 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 50 16 80319 80303 100.335 63.625 0.0119 0.00996111 2024-03-24T11:55:25.809 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:25.841 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:25.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:25.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:25.974 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:25.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:26.003 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:49:57.905681+0000", 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.06909938000001, 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.270907695, 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.14158", 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3351953651", 2024-03-24T11:55:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905681+0000", 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905681+0000", 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905683+0000", 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6060000000000001e-06 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905692+0000", 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3979999999999999e-06 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905704+0000", 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2123000000000001e-05 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905715+0000", 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1523e-05 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905796+0000", 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0366999999999998e-05 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905809+0000", 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2879e-05 2024-03-24T11:55:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.175618+0000", 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.269809356 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.175755+0000", 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013682000000000001 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.175884+0000", 2024-03-24T11:55:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012961900000000001 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176237+0000", 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0003522 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176480+0000", 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243238 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176513+0000", 2024-03-24T11:55:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870999999999997e-05 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176546+0000", 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3628999999999999e-05 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176588+0000", 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2066e-05 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:7146 7.b 7:d672a46e:::benchmark_data_smithi064_87013_object7145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:38.009511+0000", 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.965269259000003, 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065759936000000005, 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7146 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009511+0000", 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009511+0000", 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009513+0000", 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0140000000000001e-06 2024-03-24T11:55:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009536+0000", 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3357999999999999e-05 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009548+0000", 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1867e-05 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009556+0000", 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2269999999999993e-06 2024-03-24T11:55:26.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009617+0000", 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0744e-05 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009758+0000", 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141366 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009906+0000", 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147517 2024-03-24T11:55:26.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.010525+0000", 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061947 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.011838+0000", 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013121949999999999 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.075169+0000", 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063331081999999997 2024-03-24T11:55:26.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.075218+0000", 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9199000000000001e-05 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.075271+0000", 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2896999999999999e-05 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:7152 7.b 7:d3bea763:::benchmark_data_smithi064_87013_object7151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:38.012113+0000", 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.962667216, 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064533555000000006, 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7152 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012113+0000", 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012113+0000", 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012115+0000", 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9300000000000002e-06 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012139+0000", 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4196e-05 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012150+0000", 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0695e-05 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012159+0000", 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8780000000000004e-06 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012205+0000", 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5980999999999999e-05 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012286+0000", 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0970000000000006e-05 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.012535+0000", 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024978199999999999 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.013691+0000", 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011551510000000001 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.075718+0000", 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062027787000000001 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.076478+0000", 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075946300000000004 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.076586+0000", 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108459 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.076646+0000", 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0263000000000002e-05 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:20527 7.b 7:d6348140:::benchmark_data_smithi064_87013_object20526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:43.458464+0000", 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.516315663999997, 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046157779000000003, 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 20527 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458464+0000", 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458464+0000", 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458467+0000", 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.182e-06 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458489+0000", 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2458000000000001e-05 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458507+0000", 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7459e-05 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458515+0000", 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7280000000000001e-06 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458617+0000", 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102142 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458797+0000", 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017982600000000001 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.458942+0000", 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014449600000000001 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.460432+0000", 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014902120000000001 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.460975+0000", 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054349699999999999 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.504477+0000", 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043501907999999999 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.504576+0000", 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8355000000000003e-05 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.504622+0000", 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6516e-05 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:20847 7.b 7:da3eb44d:::benchmark_data_smithi064_87013_object20846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:43.635626+0000", 2024-03-24T11:55:26.028 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.33915391, 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034890233, 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 20847 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.635626+0000", 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.635626+0000", 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.635629+0000", 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3010000000000001e-06 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.635654+0000", 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5803e-05 2024-03-24T11:55:26.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.635673+0000", 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8349999999999999e-05 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.635682+0000", 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2769999999999994e-06 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.635751+0000", 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8959000000000002e-05 2024-03-24T11:55:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.635884+0000", 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013304699999999999 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.636014+0000", 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013008099999999999 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.637308+0000", 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001293584 2024-03-24T11:55:26.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.639440+0000", 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021321930000000001 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.670359+0000", 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030918999999999999 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.670471+0000", 2024-03-24T11:55:26.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011254799999999999 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:43.670516+0000", 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5089999999999997e-05 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:29171 7.2 e108/97 7:4440de80:::benchmark_data_smithi064_87013_object29170:head v 108'1739, mlcod=108'1737)", 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.116243+0000", 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.858536849000004, 2024-03-24T11:55:26.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082093063999999993, 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116243+0000", 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116243+0000", 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116245+0000", 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.674e-06 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116272+0000", 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6738999999999999e-05 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116282+0000", 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0219e-05 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116290+0000", 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2919999999999996e-06 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116378+0000", 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7276999999999998e-05 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116401+0000", 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3910999999999999e-05 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.117630+0000", 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001228951 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198336+0000", 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080706000999999999 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:29183 7.0 e108/97 7:0db6911e:::benchmark_data_smithi064_87013_object29182:head v 108'1867, mlcod=108'1865)", 2024-03-24T11:55:26.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.118079+0000", 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.856701213999997, 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084733374, 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.118079+0000", 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.118079+0000", 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.118081+0000", 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7260000000000001e-06 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.118133+0000", 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2744000000000002e-05 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198323+0000", 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080189773000000006 2024-03-24T11:55:26.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198332+0000", 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5129999999999996e-06 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198415+0000", 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3007e-05 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198438+0000", 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3723e-05 2024-03-24T11:55:26.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.202601+0000", 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041624940000000001 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.202812+0000", 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211394 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.042 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:29191 7.d e108/97 7:bfe44579:::benchmark_data_smithi064_87013_object29190:head v 108'1849, mlcod=108'1847)", 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.120732+0000", 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.854048145999997, 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079646350000000005, 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.120732+0000", 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.120732+0000", 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.120734+0000", 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1339999999999998e-06 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.120902+0000", 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016809600000000001 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198312+0000", 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077409338999999994 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198324+0000", 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2622e-05 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198389+0000", 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4331999999999993e-05 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198414+0000", 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5607000000000001e-05 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.200266+0000", 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00185216 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.200378+0000", 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011205999999999999 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:29182 7.4 e108/97 7:2bc31f0c:::benchmark_data_smithi064_87013_object29181:head v 108'1855, mlcod=108'1853)", 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.121083+0000", 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.853696728999999, 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080973751999999996, 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.121083+0000", 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.121083+0000", 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.121085+0000", 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.121122+0000", 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7608999999999999e-05 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198316+0000", 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077193500999999998 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198326+0000", 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0363999999999999e-05 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198389+0000", 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-05 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198414+0000", 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4922e-05 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.202012+0000", 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035973120000000001 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.202057+0000", 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5639000000000001e-05 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30169 7.b 7:d7bf31c7:::benchmark_data_smithi064_87013_object30168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.766037+0000", 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.208743411, 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092283463999999996, 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30169 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766037+0000", 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766037+0000", 2024-03-24T11:55:26.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766039+0000", 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9319999999999999e-06 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766063+0000", 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4379000000000002e-05 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766074+0000", 2024-03-24T11:55:26.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0791000000000001e-05 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766081+0000", 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6120000000000001e-06 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766127+0000", 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5946999999999999e-05 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767175+0000", 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010475689999999999 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767459+0000", 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000284505 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.770793+0000", 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033337010000000001 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.770836+0000", 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.286e-05 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858166+0000", 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087330388999999994 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858273+0000", 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106161 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858320+0000", 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7617999999999997e-05 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30176 7.b 7:de597230:::benchmark_data_smithi064_87013_object30175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.767071+0000", 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.207709170000001, 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091388602999999999, 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30176 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767071+0000", 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767071+0000", 2024-03-24T11:55:26.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767072+0000", 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.508e-06 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767095+0000", 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2299000000000001e-05 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767106+0000", 2024-03-24T11:55:26.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1277000000000001e-05 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767116+0000", 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7259999999999997e-06 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.769209+0000", 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020930699999999998 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.769317+0000", 2024-03-24T11:55:26.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108312 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.769459+0000", 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014156499999999999 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.770991+0000", 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00153232 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.771002+0000", 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0818999999999999e-05 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858366+0000", 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087364081999999996 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858419+0000", 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3114e-05 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858460+0000", 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0510999999999999e-05 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30186 7.b 7:d1250a05:::benchmark_data_smithi064_87013_object30185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.773372+0000", 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.201408336999997, 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086805445999999994, 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30186 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773372+0000", 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773372+0000", 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773374+0000", 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0889999999999998e-06 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773397+0000", 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3589999999999999e-05 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773410+0000", 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2602000000000001e-05 2024-03-24T11:55:26.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773419+0000", 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.67e-06 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773472+0000", 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3619000000000003e-05 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773578+0000", 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105398 2024-03-24T11:55:26.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773724+0000", 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014664800000000001 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.774987+0000", 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001262195 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.775084+0000", 2024-03-24T11:55:26.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7656000000000001e-05 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860075+0000", 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084991026999999997 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860134+0000", 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9039999999999997e-05 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860177+0000", 2024-03-24T11:55:26.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2911999999999998e-05 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30194 7.b 7:df8f0b15:::benchmark_data_smithi064_87013_object30193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.779309+0000", 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.195470966000002, 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081026438000000006, 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30194 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779309+0000", 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779309+0000", 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779311+0000", 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6840000000000001e-06 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779330+0000", 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9006999999999999e-05 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779340+0000", 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0271999999999999e-05 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779346+0000", 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8119999999999996e-06 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779398+0000", 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1557999999999997e-05 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779490+0000", 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2237000000000005e-05 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779579+0000", 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9030000000000006e-05 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.780593+0000", 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001013746 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.780742+0000", 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149357 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860246+0000", 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079504107000000004 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860296+0000", 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0071999999999998e-05 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860336+0000", 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9555999999999999e-05 2024-03-24T11:55:26.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30196 7.b 7:d819836d:::benchmark_data_smithi064_87013_object30195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.783097+0000", 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.191683515000001, 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079021147, 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30196 2024-03-24T11:55:26.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783097+0000", 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783097+0000", 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783098+0000", 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.55e-06 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783136+0000", 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8141e-05 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783146+0000", 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783152+0000", 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.677e-06 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783194+0000", 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2116000000000001e-05 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783284+0000", 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9914999999999995e-05 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783375+0000", 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0919000000000001e-05 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.784220+0000", 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084454199999999999 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.784411+0000", 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019179000000000001 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.861959+0000", 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077547952000000003 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.862068+0000", 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109158 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.862118+0000", 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9378000000000001e-05 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:64343 7.b 7:d40b9cb5:::benchmark_data_smithi064_87013_object64342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:11.552900+0000", 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.421880509999999, 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034845326000000003, 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 64343 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.552900+0000", 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.552900+0000", 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.552906+0000", 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4350000000000002e-06 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.552940+0000", 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3621000000000002e-05 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.552952+0000", 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2782999999999999e-05 2024-03-24T11:55:26.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.552962+0000", 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.628e-06 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.553015+0000", 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.553138+0000", 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012209499999999999 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.553276+0000", 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138851 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.555940+0000", 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026633690000000001 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.587572+0000", 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031632300000000002 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.587643+0000", 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1149000000000006e-05 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.587700+0000", 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6478999999999999e-05 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.587745+0000", 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5225999999999997e-05 2024-03-24T11:55:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:64352 7.b 7:d14a634c:::benchmark_data_smithi064_87013_object64351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:11.556493+0000", 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.418286974000001, 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036901043000000001, 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 64352 2024-03-24T11:55:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.556493+0000", 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.556493+0000", 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.556495+0000", 2024-03-24T11:55:26.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.764e-06 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.556519+0000", 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4474e-05 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.587492+0000", 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030972757 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.587507+0000", 2024-03-24T11:55:26.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5101e-05 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.587775+0000", 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026779199999999999 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.587925+0000", 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015011000000000001 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.588121+0000", 2024-03-24T11:55:26.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019540700000000001 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.592997+0000", 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004876231 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.593010+0000", 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2864000000000001e-05 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.593046+0000", 2024-03-24T11:55:26.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6093000000000002e-05 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.593344+0000", 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000298356 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:11.593394+0000", 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0093999999999998e-05 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:66970 7.b 7:d0236d52:::benchmark_data_smithi064_87013_object66969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:13.810805+0000", 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.163975247, 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038733835000000001, 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 66970 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810805+0000", 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810805+0000", 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810808+0000", 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8449999999999999e-06 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810837+0000", 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8762e-05 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810850+0000", 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3687e-05 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810860+0000", 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4499999999999993e-06 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.812856+0000", 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001996294 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.813018+0000", 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016247500000000001 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.813151+0000", 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132967 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.823156+0000", 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010004993 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.823199+0000", 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2673999999999998e-05 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.849394+0000", 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026195317999999999 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.849492+0000", 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7758999999999997e-05 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.849539+0000", 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6610999999999998e-05 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:66974 7.b 7:d1a65ba5:::benchmark_data_smithi064_87013_object66973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:13.810892+0000", 2024-03-24T11:55:26.098 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.163888023, 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038944291999999998, 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 66974 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810892+0000", 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810892+0000", 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810893+0000", 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7800000000000001e-07 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810916+0000", 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3023e-05 2024-03-24T11:55:26.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810925+0000", 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8419999999999994e-06 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.810931+0000", 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.815738+0000", 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048070099999999996 2024-03-24T11:55:26.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.815966+0000", 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000227531 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.818485+0000", 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025188290000000002 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.823232+0000", 2024-03-24T11:55:26.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047471750000000002 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.823269+0000", 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6677999999999997e-05 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.849556+0000", 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026287517 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.849794+0000", 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023769899999999999 2024-03-24T11:55:26.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.849836+0000", 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2644e-05 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:66977 7.b 7:d47b4047:::benchmark_data_smithi064_87013_object66976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:13.811755+0000", 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.163024857, 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041135563, 2024-03-24T11:55:26.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 66977 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.811755+0000", 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.811755+0000", 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.811757+0000", 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.79e-06 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.811779+0000", 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1718000000000001e-05 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.811790+0000", 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1258e-05 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.811798+0000", 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0360000000000006e-06 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.823301+0000", 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011502662 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.823412+0000", 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011163300000000001 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.823543+0000", 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130185 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.828649+0000", 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051060180000000004 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.828684+0000", 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5175000000000001e-05 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.852783+0000", 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024098998999999999 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.852838+0000", 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5071000000000002e-05 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.852891+0000", 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3018e-05 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:66983 7.b 7:d3aad4d8:::benchmark_data_smithi064_87013_object66982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:13.817478+0000", 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.157302034000001, 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035510414999999997, 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:26.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 66983 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.817478+0000", 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.817478+0000", 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.817481+0000", 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8109999999999999e-06 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.817508+0000", 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7107e-05 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.817521+0000", 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2869e-05 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.817529+0000", 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3909999999999998e-06 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.828711+0000", 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011181735999999999 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.828819+0000", 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.829076+0000", 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025737500000000001 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.840518+0000", 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01144131 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.842378+0000", 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018606650000000001 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.852900+0000", 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010521943000000001 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.852951+0000", 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0606999999999998e-05 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.852989+0000", 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7601000000000001e-05 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.115 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:26.116 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:26.116 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:55:26.116 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:26.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:26.147 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:26.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:26.248 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:26.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:26.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:26.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:55:26.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:55:26.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[30223]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:55:26.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[30223]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:55:26.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[30223]: pgmap v117: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 986 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:55:26.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:26.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:26.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:55:26.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:55:26.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[42243]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:55:26.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[42243]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:55:26.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[42243]: pgmap v117: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 986 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:55:26.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:26.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:26 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:55:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:26 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:55:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:26 smithi193 ceph-mon[33593]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:55:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:26 smithi193 ceph-mon[33593]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:55:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:26 smithi193 ceph-mon[33593]: pgmap v117: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 986 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:55:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:55:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:26.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:26.420 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:26.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:26.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:26.568 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:26.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:26.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:26.722 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 51 16 81477 81461 99.7854 72.375 0.0224073 0.0100154 2024-03-24T11:55:26.726 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:26.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:26.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:26.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:26.873 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:26.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:26.873 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:55:26.921 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:55:26.963 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:55:26.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:27.004 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:55:27.014 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:27.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:27.042 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:55:27.080 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:55:27.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:27.119 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:55:27.158 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:55:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:27.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:55:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:55:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:55:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:55:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:55:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:55:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:55:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:55:27.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T11:55:27.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:27.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:27.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:55:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:55:27.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:27.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:27.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:27.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:55:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:55:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:27.579 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:27.611 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:27.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:27.713 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:27.722 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 52 16 82535 82519 99.138 66.125 0.0102815 0.0100817 2024-03-24T11:55:27.760 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:27.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:27.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:27.893 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:27.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:27.995 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:28.029 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:28.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:27 smithi064 ceph-mon[30223]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:55:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2580146161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:27 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:27 smithi064 ceph-mon[42243]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:55:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:27 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2580146161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:27 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:27 smithi193 ceph-mon[33593]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:55:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:27 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2580146161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:27 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:28.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-24T11:55:28.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:28.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:28.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:28.329 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:28.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:28.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:28.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:28.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:28.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:28.606 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:28.676 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:28.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:28.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:55:28.723 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 53 16 83629 83613 98.5562 68.375 0.0240358 0.0101409 2024-03-24T11:55:28.777 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:28.806 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:28.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:28.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:28.914 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:28 smithi064 ceph-mon[30223]: pgmap v119: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 991 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:55:28.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:28 smithi064 ceph-mon[42243]: pgmap v119: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 991 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:55:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:28.978 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:28.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:29.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:28 smithi193 ceph-mon[33593]: pgmap v119: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 991 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:55:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:29.134 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:29.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:29.235 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:29.268 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:29.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:29.370 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:29.398 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:29.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:29.501 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:29.505 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:29.527 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:29.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:29.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:29.682 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:29.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:29.723 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 54 15 84753 84738 98.0332 70.3125 0.0106858 0.0101961 2024-03-24T11:55:29.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:29.826 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:29.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:29.927 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:29.960 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:29.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:30.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:30.093 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:30.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:30.194 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:30.229 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:30.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:30.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:30.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:30.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:30.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:30.501 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:30.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:30.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:30.641 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:30.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:30.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:55:30.726 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 55 16 85879 85863 97.5276 70.3125 0.0167933 0.0102475 2024-03-24T11:55:30.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:30.791 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:30.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:30.893 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:30.922 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:30.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:30.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:31.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:31.037 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:30 smithi193 ceph-mon[33593]: pgmap v120: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 830 B/s rd, 57 MiB/s wr, 910 op/s 2024-03-24T11:55:31.045 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:30 smithi064 ceph-mon[30223]: pgmap v120: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 830 B/s rd, 57 MiB/s wr, 910 op/s 2024-03-24T11:55:31.047 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:30 smithi064 ceph-mon[42243]: pgmap v120: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 830 B/s rd, 57 MiB/s wr, 910 op/s 2024-03-24T11:55:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:31.105 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:31.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:31.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:31.263 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:31.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:31.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:31.415 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:31.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:31.425 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:31.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:31.548 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:31.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:31.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:31.695 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:31.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:31.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 56 16 86820 86804 96.8364 58.8125 0.0215129 0.0103199 2024-03-24T11:55:31.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:31.802 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:55:31.802 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:55:31.837 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:31.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:31.939 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:31.979 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:31.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:32.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:32.110 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:32.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:32.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:32.242 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:32.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:32.343 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:32.374 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:32.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:32.476 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:32.504 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:32.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:32.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T11:55:32.605 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:32.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:32.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 57 16 87791 87775 96.2023 60.6875 0.00477672 0.0103883 2024-03-24T11:55:32.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:32.774 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:32.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:32 smithi064 ceph-mon[30223]: pgmap v121: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:55:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:32 smithi064 ceph-mon[42243]: pgmap v121: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:55:32.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:32.923 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:32.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:33.025 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:33.054 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:33.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:33.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:33.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:32 smithi193 ceph-mon[33593]: pgmap v121: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:55:33.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:33.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:33.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:33.286 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:33.316 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:33.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:33.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:33.462 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:33.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:33.564 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:33.592 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:33.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:33.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:33.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 58 16 88835 88819 95.6687 65.25 0.0323698 0.0104472 2024-03-24T11:55:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:33.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:33.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:33.827 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:33.857 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:33.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:33.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:34.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:34.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:34.117 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:34.152 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:34.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:34.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:34.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:34.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:34.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:34.431 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:34.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:34.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:34.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:34.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:34.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 59 16 89900 89884 95.1754 66.5625 0.0189606 0.0105012 2024-03-24T11:55:34.726 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:34.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:34.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:34.852 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:34.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:34.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:34.881 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:34.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:35.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:34 smithi064 ceph-mon[30223]: pgmap v122: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T11:55:35.020 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:34 smithi064 ceph-mon[42243]: pgmap v122: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T11:55:35.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:35.043 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:35.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:35.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:34 smithi193 ceph-mon[33593]: pgmap v122: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T11:55:35.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:35.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:35.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:35.316 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:35.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:35.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:35.459 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:35.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:35.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:35.613 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:35.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:35.712 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-03-24T11:55:35.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:35.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T11:55:35.726689+0000 min lat: 0.00141856 max lat: 0.234923 avg lat: 0.0105616 2024-03-24T11:55:35.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:55:35.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 60 16 90898 90882 94.6285 62.375 0.0131127 0.0105616 2024-03-24T11:55:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:35.759 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:35.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:35.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1518086333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:55:35.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:35 smithi064 ceph-mon[30223]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:55:35.837 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1518086333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:55:35.837 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:35 smithi064 ceph-mon[42243]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:55:35.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:35.892 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:35.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:35.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:36.031 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:36.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1518086333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:55:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:35 smithi193 ceph-mon[33593]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:55:36.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:36.157 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:36.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:36.259 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:36.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:36.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:36.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:36.421 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:36.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:36.523 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:36.554 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:36.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:36.655 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:36.684 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:36.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:36.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 61 16 91997 91981 94.2033 68.6875 0.0152012 0.0106102 2024-03-24T11:55:36.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:36.824 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:36.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:36.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:36.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:37.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:36 smithi064 ceph-mon[30223]: pgmap v123: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T11:55:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:36 smithi064 ceph-mon[30223]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:55:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:36 smithi064 ceph-mon[30223]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:55:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:36 smithi064 ceph-mon[42243]: pgmap v123: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T11:55:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:36 smithi064 ceph-mon[42243]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:55:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:36 smithi064 ceph-mon[42243]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:55:37.069 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:37.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:55:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:55:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:36 smithi193 ceph-mon[33593]: pgmap v123: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T11:55:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:36 smithi193 ceph-mon[33593]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:55:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:36 smithi193 ceph-mon[33593]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:55:37.098 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:37.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:37.200 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:37.230 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:37.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:37.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:37.376 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:37.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:37.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:37.525 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:37.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:37.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:37.670 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:37.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:37.722 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:55:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:55:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:37.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 62 16 93071 93055 93.7666 67.125 0.0118137 0.0106592 2024-03-24T11:55:37.772 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:37.803 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:37.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:37.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:37.942 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:37.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:38.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:37 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:38.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:37 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:38.085 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:38.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:37 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:38.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:38.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:38.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:38.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:38.229 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:38.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:38.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:38.367 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:38.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:38.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:38.508 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:38.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:38.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:38.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:38.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:38.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 63 15 94162 94147 93.3616 68.25 0.0105764 0.0107058 2024-03-24T11:55:38.754 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:38.783 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:38.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:38.884 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:38.917 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:38.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:39.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:39.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:38 smithi064 ceph-mon[30223]: pgmap v125: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:55:39.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:38 smithi064 ceph-mon[42243]: pgmap v125: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:55:39.059 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:39.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:38 smithi193 ceph-mon[33593]: pgmap v125: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:55:39.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:39.203 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:39.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:39.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:39.352 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:39.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:39.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:39.496 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:39.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:39.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:39.649 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:39.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:39.725 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 64 16 95208 95192 92.9234 65.3125 0.0107999 0.0107566 2024-03-24T11:55:39.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:39.791 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:39.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:39.893 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:39.925 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:39.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:39.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:40.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:40.077 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:40.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:40.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:40.211 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:40.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:40.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:40.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-24T11:55:40.409 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:40.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:40.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:40.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:40.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:40.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:40.724 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:40.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:40.726 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 65 16 96263 96247 92.5083 65.9375 0.0205808 0.0108029 2024-03-24T11:55:40.827 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:40.857 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:40.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:40.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:40.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:40.988 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:40.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:41.047 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:40 smithi064 ceph-mon[30223]: pgmap v126: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:55:41.047 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:41.047 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:40 smithi064 ceph-mon[42243]: pgmap v126: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:55:41.047 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:41.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:40 smithi193 ceph-mon[33593]: pgmap v126: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:55:41.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:41.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:41.130 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:41.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:41.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:41.286 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:41.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:41.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:41.416 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:41.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:41.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:41.547 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:41.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:41.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:41.691 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:41.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:41.726 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 66 16 97336 97320 92.1228 67.0625 0.0100234 0.0108496 2024-03-24T11:55:41.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:41.845 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:41.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:41.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:41.975 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:41.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:42.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:42.106 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:42.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:42.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:42.251 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:42.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:42.353 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:42.380 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:42.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:42.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:42.519 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:42.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:42.539 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:55:42.582 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:55:42.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:42.642 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:55:42.649 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:42.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:42.656 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:55:42.657 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:55:42.693 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:55:42.726 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 67 16 98275 98259 91.6238 58.6875 0.0137949 0.010909 2024-03-24T11:55:42.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:55:42.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:42.780 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T11:55:42.786 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:55:42.795 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:42.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:42.833 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:55:42.874 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:55:42.885 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:55:42.885 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:55:42.885 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:55:42.885 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:55:42.885 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:55:42.886 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:55:42.886 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:55:42.886 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:55:42.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:42.925 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:42.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:43.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:42 smithi064 ceph-mon[30223]: pgmap v127: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:55:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/292830279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:42 smithi064 ceph-mon[30223]: from='client.34799 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:43.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3927830560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:42 smithi064 ceph-mon[42243]: pgmap v127: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:55:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/292830279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:42 smithi064 ceph-mon[42243]: from='client.34799 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:43.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3927830560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:43.069 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:43.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:42 smithi193 ceph-mon[33593]: pgmap v127: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:55:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/292830279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:42 smithi193 ceph-mon[33593]: from='client.34799 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:55:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3927830560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:43.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:43.212 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:43.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:43.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:43.353 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:43.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:43.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:43.497 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:43.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:43.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:43.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:43.726 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 68 16 99345 99329 91.2599 66.875 0.0166614 0.0109528 2024-03-24T11:55:43.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:43.787 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:43.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:43.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:43.929 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:43.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:44.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:43 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:43 smithi064 ceph-mon[30223]: from='client.34799 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:55:44.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:43 smithi064 ceph-mon[30223]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:55:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:43 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:43 smithi064 ceph-mon[42243]: from='client.34799 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:55:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:43 smithi064 ceph-mon[42243]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:55:44.063 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:44.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:44.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:43 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:43 smithi193 ceph-mon[33593]: from='client.34799 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:55:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:43 smithi193 ceph-mon[33593]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:55:44.165 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:44.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:44.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T11:55:44.295 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:44.327 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:44.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:44.428 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:44.458 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:44.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:44.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:44.609 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:44.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:44.705 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:44.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:44.726 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 69 16 100384 100368 90.8779 64.9375 0.00879283 0.0109984 2024-03-24T11:55:44.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:44.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:44.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:44.931 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:44.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:45.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:45.034 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:44 smithi064 ceph-mon[30223]: pgmap v129: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:55:45.034 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:44 smithi064 ceph-mon[42243]: pgmap v129: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:55:45.079 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:45.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:45.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:44 smithi193 ceph-mon[33593]: pgmap v129: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:55:45.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:45.214 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:45.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:45.315 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:45.345 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:45.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:45.447 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:45.487 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:45.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:45.549 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:45.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:45.619 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:45.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:45.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:45.727 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 70 16 101458 101442 90.5381 67.125 0.0241214 0.0110399 2024-03-24T11:55:45.765 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:45.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:45.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:55:45.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:45.917 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:45.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:46.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:46.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:45 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:55:46.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:45 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:55:46.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:45 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:55:46.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:45 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:55:46.076 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:46.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:46.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:45 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:55:46.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:45 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:55:46.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:46.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:46.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:46.358 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:46.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:46.460 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:46.489 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:46.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:46.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:46.621 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:46.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:46.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:55:46.723 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:46.727 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 71 15 102475 102460 90.1585 63.625 0.0255465 0.0110864 2024-03-24T11:55:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:46.753 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:46.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:46.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:46.908 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:46.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:47.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:47.038 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:47.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:47.050 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:46 smithi064 ceph-mon[42243]: pgmap v130: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:55:47.050 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:46 smithi064 ceph-mon[42243]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:55:47.050 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:46 smithi064 ceph-mon[30223]: pgmap v130: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:55:47.050 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:46 smithi064 ceph-mon[30223]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:55:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:46 smithi193 ceph-mon[33593]: pgmap v130: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:55:47.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:46 smithi193 ceph-mon[33593]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:55:47.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:55:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:55:47.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:47.164 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:47.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:47.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:47.306 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:47.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:47.407 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:47.436 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:47.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:47.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:47.569 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:47.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:47.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:47.711 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:47.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:47.727 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 72 16 103470 103454 89.7692 62.125 0.0112075 0.0111347 2024-03-24T11:55:47.785 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:55:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:55:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:47.812 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:47.841 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:47.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:47.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:47.971 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:47.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:48.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:47 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:48.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:47 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:48.072 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:48.089 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:47 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:48.106 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:48.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:48.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:48.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:48.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:48.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:48.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:48.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:48.477 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:48.509 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:48.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:48.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:48.643 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:48.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:48.730 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 73 16 104454 104438 89.382 61.5 0.00910573 0.0111827 2024-03-24T11:55:48.745 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:48.776 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:48.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:48.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:48.921 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:48.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:49.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:49.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:48 smithi064 ceph-mon[30223]: pgmap v132: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-24T11:55:49.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:48 smithi064 ceph-mon[42243]: pgmap v132: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-24T11:55:49.065 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:49.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:48 smithi193 ceph-mon[33593]: pgmap v132: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-24T11:55:49.167 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:49.196 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:49.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:49.235 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:49.298 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:49.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:49.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:49.431 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:49.462 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:49.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:49.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:49.608 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:49.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:49.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:49.730 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 74 16 105387 105371 88.9602 58.3125 0.0115841 0.0112346 2024-03-24T11:55:49.737 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:49.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:49.742 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:49.817 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.7 to 1 (10000) 2024-03-24T11:55:49.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:49.895 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:49.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:49.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:49 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1822977771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:55:49.968 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:49 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1822977771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:55:49.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:50.026 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:50.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:50.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:49 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1822977771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:55:50.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:50.171 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:50.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:50.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:50.302 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:50.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:50.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:50.439 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:50.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:50.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:50.579 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:50.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:50.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:50.718 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:50.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:50.729 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 75 16 106407 106391 88.6241 63.75 0.0103124 0.0112787 2024-03-24T11:55:50.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:50.865 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:50.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:50.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:51.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:51.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:50 smithi193 ceph-mon[33593]: pgmap v133: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:55:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:50 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1822977771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:55:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:50 smithi193 ceph-mon[33593]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:55:51.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:51.141 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:51.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:51.145 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:50 smithi064 ceph-mon[30223]: pgmap v133: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:55:51.145 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:50 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1822977771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:55:51.145 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:50 smithi064 ceph-mon[30223]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:55:51.145 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:50 smithi064 ceph-mon[42243]: pgmap v133: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:55:51.145 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:50 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1822977771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:55:51.145 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:50 smithi064 ceph-mon[42243]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:55:51.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:51.276 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:51.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:51.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:51.410 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:51.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:51.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:51.542 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:51.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:51.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:51.685 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:51.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:51.729 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 76 16 107523 107507 88.3758 69.75 0.0258504 0.0113103 2024-03-24T11:55:51.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:51.819 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:51.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:51.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:51.950 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:51.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:52.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:52.080 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:52.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:52.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:52.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:52.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:52.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:52.343 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:52.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:52.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:52.487 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:52.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:52.588 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:52.618 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:52.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:52.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:52.729 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 77 16 108620 108604 88.1184 68.5625 0.0119389 0.0113432 2024-03-24T11:55:52.765 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:52.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:52.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:52.896 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:52.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:52.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:53.041 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:53.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:53.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:53.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:52 smithi064 ceph-mon[30223]: pgmap v135: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-24T11:55:53.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:52 smithi064 ceph-mon[42243]: pgmap v135: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-24T11:55:53.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:52 smithi193 ceph-mon[33593]: pgmap v135: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-24T11:55:53.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:53.186 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:53.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:53.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:53.295 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-24T11:55:53.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:55:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:53.338 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:53.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:53.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:53.470 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:53.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:53.572 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:53.598 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:53.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:53.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:53.732 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 78 16 109609 109593 87.7811 61.8125 0.00849778 0.011387 2024-03-24T11:55:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:53.733 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:53.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:53.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:53.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:53.899 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:53.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:54.000 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:55:54.000 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T11:55:54.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T11:55:54.000 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T11:55:54.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.000 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:29115 7.e 7:75ba391f:::benchmark_data_smithi064_87013_object29114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:47.952218+0000", 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.171811586999993, 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14419949700000001, 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 29115 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952218+0000", 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952218+0000", 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952220+0000", 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.576e-06 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952253+0000", 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3386000000000002e-05 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952264+0000", 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1188e-05 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952272+0000", 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7719999999999994e-06 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952948+0000", 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067639000000000002 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.953070+0000", 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122144 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.953200+0000", 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012967799999999999 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.956194+0000", 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002993664 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.956206+0000", 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2223e-05 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.096311+0000", 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14010472600000001 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.096372+0000", 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1455e-05 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.096417+0000", 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5294999999999999e-05 2024-03-24T11:55:54.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:29116 7.e 7:7b8491f4:::benchmark_data_smithi064_87013_object29115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:47.952290+0000", 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.171739622999993, 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15235985599999999, 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 29116 2024-03-24T11:55:54.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952290+0000", 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952290+0000", 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952291+0000", 2024-03-24T11:55:54.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.85e-07 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952664+0000", 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037378200000000002 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952676+0000", 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1073999999999999e-05 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952683+0000", 2024-03-24T11:55:54.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6729999999999992e-06 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.956235+0000", 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035517980000000001 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.956343+0000", 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010835799999999999 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.956632+0000", 2024-03-24T11:55:54.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028822899999999998 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.011979+0000", 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055347483000000003 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.012019+0000", 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0407000000000001e-05 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.104540+0000", 2024-03-24T11:55:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092520187000000004 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.104602+0000", 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2179000000000002e-05 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.104650+0000", 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7901000000000001e-05 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.012 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:29118 7.e 7:74e5787e:::benchmark_data_smithi064_87013_object29117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:47.952695+0000", 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.171334294999994, 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.152220256, 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 29118 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952695+0000", 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952695+0000", 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952696+0000", 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5300000000000002e-07 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952720+0000", 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3461000000000001e-05 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952728+0000", 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0450000000000001e-06 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.952734+0000", 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3600000000000001e-06 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.958013+0000", 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052790720000000001 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.958136+0000", 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000123041 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:47.958252+0000", 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115889 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.012055+0000", 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053803149000000002 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.012122+0000", 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6769999999999999e-05 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.104765+0000", 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092643340000000005 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.104870+0000", 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105023 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.104915+0000", 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5153e-05 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:32539 7.e 7:75f04f1b:::benchmark_data_smithi064_87013_object32538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:54.018 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:50.244275+0000", 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 62.879754548999998, 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114702147, 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32539 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244275+0000", 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244275+0000", 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244277+0000", 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.658e-06 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244302+0000", 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4941000000000001e-05 2024-03-24T11:55:54.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244311+0000", 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4299999999999995e-06 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244318+0000", 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.097e-06 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244380+0000", 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2286999999999995e-05 2024-03-24T11:55:54.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244551+0000", 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017014800000000001 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.244644+0000", 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3585999999999995e-05 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.246322+0000", 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001677624 2024-03-24T11:55:54.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.246640+0000", 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031824300000000001 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.358880+0000", 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112239623 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.358940+0000", 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0288999999999998e-05 2024-03-24T11:55:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.358977+0000", 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7221000000000002e-05 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:32548 7.8 7:1debf2bd:::benchmark_data_smithi064_87013_object32547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:50.247073+0000", 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 62.876956354000001, 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111926092, 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32548 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247073+0000", 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247073+0000", 2024-03-24T11:55:54.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247075+0000", 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1730000000000002e-06 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247129+0000", 2024-03-24T11:55:54.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3990000000000003e-05 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247218+0000", 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9025999999999998e-05 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247227+0000", 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2859999999999999e-06 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247505+0000", 2024-03-24T11:55:54.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027867499999999998 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247676+0000", 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170812 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,7", 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.247813+0000", 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638600000000001 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249856+0000", 2024-03-24T11:55:54.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020438679999999999 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250076+0000", 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022008600000000001 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250112+0000", 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5964999999999999e-05 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.358939+0000", 2024-03-24T11:55:54.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.108827008 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.358999+0000", 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9817e-05 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.34742.0:32539 7.e e108/97 ondisk, result = 0)", 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:50.249920+0000", 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 62.874109904999997, 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10906757, 2024-03-24T11:55:54.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249920+0000", 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249920+0000", 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249921+0000", 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5880000000000001e-06 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249927+0000", 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8050000000000003e-06 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249937+0000", 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0027000000000001e-05 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249944+0000", 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2790000000000001e-06 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250042+0000", 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7379000000000004e-05 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250050+0000", 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9889999999999992e-06 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.358987+0000", 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.108937503 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.34742.0:32548 7.8 e108/97 ondisk, result = 0)", 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:50.249992+0000", 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 62.874037758999997, 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109015205, 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249992+0000", 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249992+0000", 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249993+0000", 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6000000000000002e-07 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.249996+0000", 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6689999999999999e-06 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250003+0000", 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3849999999999999e-06 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250008+0000", 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1189999999999996e-06 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250102+0000", 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4707e-05 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250108+0000", 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3029999999999999e-06 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.359007+0000", 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10889916199999999 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:32545 7.2 e108/97 7:4d95bfa0:::benchmark_data_smithi064_87013_object32544:head v 108'1947, mlcod=108'1943)", 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:50.250129+0000", 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 62.873900018999997, 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112362103, 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250129+0000", 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250129+0000", 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250131+0000", 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.497e-06 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.250596+0000", 2024-03-24T11:55:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046533700000000002 2024-03-24T11:55:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.359027+0000", 2024-03-24T11:55:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.108430533 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.359037+0000", 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0314e-05 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.359098+0000", 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0433999999999998e-05 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.359115+0000", 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7110000000000001e-05 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.362446+0000", 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033314989999999999 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:50.362492+0000", 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5379e-05 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:65828 7.e 7:71fb73c4:::benchmark_data_smithi064_87013_object65827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:12.716303+0000", 2024-03-24T11:55:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.407726359999998, 2024-03-24T11:55:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123258122, 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65828 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716303+0000", 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716303+0000", 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716305+0000", 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0820000000000001e-06 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716330+0000", 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4341e-05 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716341+0000", 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1447000000000001e-05 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716350+0000", 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8789999999999992e-06 2024-03-24T11:55:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716412+0000", 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2479999999999998e-05 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716540+0000", 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127417 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.716677+0000", 2024-03-24T11:55:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013691699999999999 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839428+0000", 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12275092 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839475+0000", 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7142999999999999e-05 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839483+0000", 2024-03-24T11:55:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3359999999999995e-06 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839513+0000", 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9553e-05 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839561+0000", 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8606999999999997e-05 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:65832 7.e 7:71c0e793:::benchmark_data_smithi064_87013_object65831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:12.718274+0000", 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.405755835999997, 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123323129, 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65832 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718274+0000", 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718274+0000", 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718276+0000", 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7989999999999998e-06 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718300+0000", 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3543000000000001e-05 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718313+0000", 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.257e-05 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718322+0000", 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0100000000000001e-06 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718528+0000", 2024-03-24T11:55:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206646 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718613+0000", 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4611999999999996e-05 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.837630+0000", 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119017287 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839575+0000", 2024-03-24T11:55:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019448689999999999 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841470+0000", 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001894581 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841521+0000", 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1493999999999999e-05 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841564+0000", 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3204000000000002e-05 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841597+0000", 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2514000000000001e-05 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:65833 7.e 7:7c4eadda:::benchmark_data_smithi064_87013_object65832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:12.718804+0000", 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.405225352000002, 2024-03-24T11:55:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.122913276, 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65833 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718804+0000", 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718804+0000", 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718806+0000", 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.342e-06 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718826+0000", 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0985999999999999e-05 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.837251+0000", 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118424932 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.837264+0000", 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2308e-05 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839615+0000", 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023507760000000002 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839746+0000", 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131734 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839859+0000", 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112404 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841635+0000", 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017765560000000001 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841645+0000", 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0215e-05 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841667+0000", 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1963e-05 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841697+0000", 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9203000000000001e-05 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.841717+0000", 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0857e-05 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.34742.0:65828 7.e e115/97 ondisk, result = 0)", 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:12.718893+0000", 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.405136122000002, 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.120544, 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718893+0000", 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718893+0000", 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718895+0000", 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2079999999999999e-06 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.718900+0000", 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0499999999999999e-06 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.837244+0000", 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11834478700000001 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.837253+0000", 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7649999999999999e-06 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839412+0000", 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021590889999999999 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839421+0000", 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5720000000000003e-06 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:12.839437+0000", 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6529e-05 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:66144 7.e 7:71f54f0c:::benchmark_data_smithi064_87013_object66143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:13.073368+0000", 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.050661486000003, 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11071389399999999, 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 66144 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.073368+0000", 2024-03-24T11:55:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.073368+0000", 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.073369+0000", 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7900000000000007e-07 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.073389+0000", 2024-03-24T11:55:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9922999999999999e-05 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.073397+0000", 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0369999999999995e-06 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.073403+0000", 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.353e-06 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.076502+0000", 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030986719999999998 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.076625+0000", 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122977 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.076764+0000", 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013929099999999999 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.083133+0000", 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063692030000000004 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.083377+0000", 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024341099999999999 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.183840+0000", 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100463292 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.184017+0000", 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017681899999999999 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.184082+0000", 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4937000000000005e-05 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:67065 7.e 7:7f47008d:::benchmark_data_smithi064_87013_object67064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:13.879086+0000", 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.244943436, 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14802199499999999, 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 67065 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879086+0000", 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879086+0000", 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879088+0000", 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9520000000000001e-06 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879119+0000", 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0947e-05 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879131+0000", 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2368000000000001e-05 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879200+0000", 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8350999999999997e-05 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879337+0000", 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136876 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879463+0000", 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012678 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.879591+0000", 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012784 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.881840+0000", 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022483590000000001 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.882309+0000", 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046956 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.026997+0000", 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14468806100000001 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.027060+0000", 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2974999999999999e-05 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.027108+0000", 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7926000000000002e-05 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:67362 7.e 7:745c9bc1:::benchmark_data_smithi064_87013_object67361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:14.375073+0000", 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.748956294000003, 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.09569569, 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 67362 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.375073+0000", 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.375073+0000", 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.375075+0000", 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.854e-06 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.375103+0000", 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7878999999999999e-05 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.375115+0000", 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1664e-05 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.375122+0000", 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8739999999999995e-06 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.375182+0000", 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9126999999999998e-05 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.376069+0000", 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088738800000000002 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.376214+0000", 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145289 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379306+0000", 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003091758 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.381950+0000", 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026443339999999999 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.470652+0000", 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088701187000000001 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.470717+0000", 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5316999999999998e-05 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.470769+0000", 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2018999999999998e-05 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:67365 7.e 7:7f1f4611:::benchmark_data_smithi064_87013_object67364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:14.379283+0000", 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.744746827999997, 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096219165999999995, 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 67365 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379283+0000", 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379283+0000", 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379285+0000", 2024-03-24T11:55:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9420000000000002e-06 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379308+0000", 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2884999999999999e-05 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379318+0000", 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0689999999999999e-05 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379326+0000", 2024-03-24T11:55:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9710000000000004e-06 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379399+0000", 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2954000000000007e-05 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379584+0000", 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185045 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.379752+0000", 2024-03-24T11:55:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016824099999999999 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.382153+0000", 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002400606 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.382414+0000", 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026127099999999999 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.475289+0000", 2024-03-24T11:55:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092874337000000001 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.475450+0000", 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000161235 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.475502+0000", 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989e-05 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:68191 7.e 7:731fa3f8:::benchmark_data_smithi064_87013_object68190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:15.168850+0000", 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.955179448000003, 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.115307176, 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 68191 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.168850+0000", 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.168850+0000", 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.168852+0000", 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.956e-06 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.169020+0000", 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167706 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.169032+0000", 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2103000000000001e-05 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.169040+0000", 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7049999999999999e-06 2024-03-24T11:55:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.169095+0000", 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5843e-05 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.169296+0000", 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020111400000000001 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.169858+0000", 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056125200000000004 2024-03-24T11:55:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.176144+0000", 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062863870000000001 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.176186+0000", 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1498999999999997e-05 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.283304+0000", 2024-03-24T11:55:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10711865199999999 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.283856+0000", 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055151500000000003 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.284157+0000", 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000301444 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:68196 7.e 7:7858b68a:::benchmark_data_smithi064_87013_object68195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:15.171924+0000", 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.952105799999998, 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121405423, 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 68196 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.171924+0000", 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.171924+0000", 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.171926+0000", 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0150000000000002e-06 2024-03-24T11:55:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.171949+0000", 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.298e-05 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.171962+0000", 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2907e-05 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.171970+0000", 2024-03-24T11:55:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8279999999999992e-06 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.173471+0000", 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001500412 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.173583+0000", 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112371 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.173710+0000", 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126408 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.178113+0000", 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044037470000000004 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.178149+0000", 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5766999999999998e-05 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.291819+0000", 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.113669991 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.293147+0000", 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013279050000000001 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.293329+0000", 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018209199999999999 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:68199 7.e 7:718e6a52:::benchmark_data_smithi064_87013_object68198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:15.175183+0000", 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.948846474, 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12087659000000001, 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 68199 2024-03-24T11:55:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.175183+0000", 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.175183+0000", 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.175185+0000", 2024-03-24T11:55:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.175205+0000", 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0707999999999999e-05 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.175213+0000", 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0809999999999994e-06 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.175220+0000", 2024-03-24T11:55:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1639999999999998e-06 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.176217+0000", 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099772599999999999 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.176336+0000", 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118683 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.176490+0000", 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000154473 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.178175+0000", 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001684379 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.178268+0000", 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2655e-05 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.295948+0000", 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.117680505 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.296008+0000", 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9904000000000001e-05 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:15.296060+0000", 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1712000000000002e-05 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:104050 7.e 7:7996d2b2:::benchmark_data_smithi064_87013_object104049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:48.286288+0000", 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.837741673, 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085133692999999996, 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104050 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.286288+0000", 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.286288+0000", 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.286290+0000", 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9209999999999999e-06 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.286377+0000", 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6976000000000002e-05 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.286389+0000", 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2182e-05 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.286397+0000", 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8089999999999993e-06 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.288663+0000", 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022658270000000002 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.288978+0000", 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000315588 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.290982+0000", 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020035650000000001 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.295713+0000", 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004731402 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.296833+0000", 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011198060000000001 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.371277+0000", 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074444352000000005 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.371381+0000", 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103967 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:48.371422+0000", 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0297999999999999e-05 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:55:54.113 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:55:54.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:54.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:54.259 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:54.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:54.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:54.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:54.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:54.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:54.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:54.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:54.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:54.711 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:54.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:54.730 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 79 16 110583 110567 87.4405 60.875 0.0249464 0.0114306 2024-03-24T11:55:54.771 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:55:54.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:54.815 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:55:54.846 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T11:55:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:54.859 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:54.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:54.873 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:55:54.921 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:55:54.959 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:55:54.960 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:54.988 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:54.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:55.000 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:55:55.040 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:55:55.079 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:55:55.079 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:54 smithi064 ceph-mon[30223]: pgmap v136: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:55:55.080 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/286815578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:55.080 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1850229772' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:55:55.080 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:54 smithi064 ceph-mon[30223]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:55:55.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:54 smithi064 ceph-mon[42243]: pgmap v136: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:55:55.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/286815578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:55.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1850229772' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:55:55.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:54 smithi064 ceph-mon[42243]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:55:55.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:54 smithi193 ceph-mon[33593]: pgmap v136: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:55:55.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/286815578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:55:55.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1850229772' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:55:55.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:54 smithi193 ceph-mon[33593]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:55:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:55:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:55:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:55:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:55:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:55:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:55:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:55:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:55:55.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:55.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:55.118 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:55.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:55.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:55.250 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:55.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:55.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:55.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:55.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:55.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:55.540 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:55.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:55.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:55.692 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:55.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:55.730 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T11:55:55.731489+0000 min lat: 0.00141856 max lat: 0.234923 avg lat: 0.0114686 2024-03-24T11:55:55.730 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:55:55.730 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 80 16 111616 111600 87.1545 64.5625 0.0097535 0.0114686 2024-03-24T11:55:55.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:55:55.835 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:55.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:55:55.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:55.975 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:55.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:56.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:55 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:56.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:55 smithi064 ceph-mon[30223]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:55:56.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:55 smithi064 ceph-mon[30223]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:55:56.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:56.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:55 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:56.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:55 smithi064 ceph-mon[42243]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:55:56.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:55 smithi064 ceph-mon[42243]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:55:56.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:56.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:55 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:55:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:55 smithi193 ceph-mon[33593]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:55:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:55 smithi193 ceph-mon[33593]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:55:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:56.111 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:56.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:56.212 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:56.241 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:56.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:56.343 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:56.373 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:56.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:56.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:56.521 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:56.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:56.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T11:55:56.599 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:56.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:56.679 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:56.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:56.731 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 81 16 112672 112656 86.8921 66 0.0190624 0.0115035 2024-03-24T11:55:56.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:56.809 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:56.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:56.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:56.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:57.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:57.097 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:57.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:57.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[30223]: pgmap v138: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 629 op/s 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[30223]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[42243]: pgmap v138: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 629 op/s 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T11:55:57.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:56 smithi064 ceph-mon[42243]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:55:57.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:57.259 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:57.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:57.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:55:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:55:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:56 smithi193 ceph-mon[33593]: pgmap v138: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 629 op/s 2024-03-24T11:55:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:56 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:55:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:56 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:55:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:56 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T11:55:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:56 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T11:55:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:56 smithi193 ceph-mon[33593]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:55:57.360 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:57.392 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:57.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:57.494 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:57.527 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:57.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:57.628 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:57.646 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:55:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:57.658 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:57.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:57.731 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 82 16 113713 113697 86.6258 65.0625 0.0213377 0.0115386 2024-03-24T11:55:57.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:55:57.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:57.798 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:57.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:55:57.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:57.929 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:57.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:58.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[30223]: 5.12 scrub starts 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[30223]: 5.12 scrub ok 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[30223]: 3.2 scrub starts 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[30223]: 3.2 scrub ok 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[42243]: 5.12 scrub starts 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[42243]: 5.12 scrub ok 2024-03-24T11:55:58.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[42243]: 3.2 scrub starts 2024-03-24T11:55:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[42243]: 3.2 scrub ok 2024-03-24T11:55:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T11:55:58.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:57 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T11:55:58.063 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:58.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:58.099 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:55:58.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:58.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:58.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:58.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:57 smithi193 ceph-mon[33593]: 5.12 scrub starts 2024-03-24T11:55:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:57 smithi193 ceph-mon[33593]: 5.12 scrub ok 2024-03-24T11:55:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:57 smithi193 ceph-mon[33593]: 3.2 scrub starts 2024-03-24T11:55:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:57 smithi193 ceph-mon[33593]: 3.2 scrub ok 2024-03-24T11:55:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:57 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:57 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T11:55:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:57 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T11:55:58.353 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:58.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:58.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:55:58.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:58.507 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:58.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:58.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:58.658 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:58.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:58.733 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 83 16 114764 114748 86.3716 65.6875 0.00446067 0.0115722 2024-03-24T11:55:58.760 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:58.791 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:58.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:58.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:55:58.922 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:58.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:55:59.024 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:55:59.058 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:59.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:55:59.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:59.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:55:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:59.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:59.232 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:58 smithi064 ceph-mon[30223]: pgmap v140: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T11:55:59.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:58 smithi064 ceph-mon[42243]: pgmap v140: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T11:55:59.305 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:55:59.332 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:59.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:55:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:58 smithi193 ceph-mon[33593]: pgmap v140: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T11:55:59.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:59.479 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:59.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:59.580 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:55:59.610 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:55:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:59.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:55:59.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:55:59.733 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 84 16 115735 115719 86.0658 60.6875 0.00673512 0.0116144 2024-03-24T11:55:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:59.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:55:59.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:55:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:55:59.888 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:55:59.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:55:59.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:00.015 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:00.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:00.118 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:00.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:59 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T11:56:00.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:55:59 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T11:56:00.136 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:59 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T11:56:00.136 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:55:59 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T11:56:00.145 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:00.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:00.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:00.277 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:00.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:59 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T11:56:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:55:59 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T11:56:00.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:00.421 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:00.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:00.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:00.552 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:00.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:00.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:00.698 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:00.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:00.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:00.733 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 85 16 116651 116635 85.7269 57.25 0.0222385 0.0116594 2024-03-24T11:56:00.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:00.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:00.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:00.975 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:00.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[30223]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[42243]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T11:56:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T11:56:01.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:00 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T11:56:01.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:01.121 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:01.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:01.222 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:01.251 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:01.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:56:01.319 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:56:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:00 smithi193 ceph-mon[33593]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-24T11:56:01.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:00 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T11:56:01.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:00 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T11:56:01.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:00 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T11:56:01.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:00 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T11:56:01.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:01.401 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:01.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:01.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:01.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:01.643 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:01.670 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:01.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:01.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 86 16 117451 117435 85.3115 50 0.0305883 0.0117154 2024-03-24T11:56:01.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:01.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:01.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:01.946 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:01.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:02.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:02.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:02.080 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:02.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:02.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:02.213 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:02.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:01 smithi064 ceph-mon[30223]: 2.a scrub starts 2024-03-24T11:56:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:01 smithi064 ceph-mon[30223]: 2.a scrub ok 2024-03-24T11:56:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:01 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T11:56:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:01 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T11:56:02.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:01 smithi064 ceph-mon[42243]: 2.a scrub starts 2024-03-24T11:56:02.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:01 smithi064 ceph-mon[42243]: 2.a scrub ok 2024-03-24T11:56:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:01 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T11:56:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:01 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T11:56:02.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:02.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:01 smithi193 ceph-mon[33593]: 2.a scrub starts 2024-03-24T11:56:02.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:01 smithi193 ceph-mon[33593]: 2.a scrub ok 2024-03-24T11:56:02.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:01 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T11:56:02.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:01 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T11:56:02.363 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:02.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:02.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:02.497 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:02.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:02.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:02.628 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:02.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:02.730 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:02.733 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 87 16 118510 118494 85.0917 66.1875 0.00671634 0.0117468 2024-03-24T11:56:02.762 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:02.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:02.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:02.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:02.972 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[30223]: pgmap v142: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 880 op/s 2024-03-24T11:56:02.972 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[30223]: from='client.34811 v1:172.21.15.64:0/2623257743' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[42243]: pgmap v142: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 880 op/s 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[42243]: from='client.34811 v1:172.21.15.64:0/2623257743' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T11:56:02.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:02 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T11:56:03.008 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:03.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:03.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:03.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:03.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:03.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:03.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:03.316 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:03.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:02 smithi193 ceph-mon[33593]: pgmap v142: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 880 op/s 2024-03-24T11:56:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:02 smithi193 ceph-mon[33593]: from='client.34811 v1:172.21.15.64:0/2623257743' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:02 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T11:56:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:02 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T11:56:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:02 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T11:56:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:02 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T11:56:03.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:03.447 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:03.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:03.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:03.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:03.692 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:03.692 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:03.698 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:03.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:03.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:03.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 88 16 119395 119379 84.7534 55.3125 0.0417964 0.0117924 2024-03-24T11:56:03.789 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:56:03.789 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:56:03.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:03.877 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:03.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:03.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:03.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:04.032 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:04.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:04.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:04.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T11:56:04.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T11:56:04.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[30223]: 7.8 scrub starts 2024-03-24T11:56:04.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[30223]: 6.b scrub starts 2024-03-24T11:56:04.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[30223]: 6.b scrub ok 2024-03-24T11:56:04.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[30223]: 5.1b scrub starts 2024-03-24T11:56:04.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[30223]: 5.1b scrub ok 2024-03-24T11:56:04.140 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T11:56:04.140 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T11:56:04.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[42243]: 7.8 scrub starts 2024-03-24T11:56:04.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[42243]: 6.b scrub starts 2024-03-24T11:56:04.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[42243]: 6.b scrub ok 2024-03-24T11:56:04.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[42243]: 5.1b scrub starts 2024-03-24T11:56:04.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:03 smithi064 ceph-mon[42243]: 5.1b scrub ok 2024-03-24T11:56:04.153 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-03-24T11:56:01.185343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7470848350,"num_objects":114185,"num_object_clones":0,"num_object_copies":342555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114185,"num_whiteouts":0,"num_read":1142,"num_read_kb":988,"num_write":114602,"num_write_kb":7297629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89640,"ondisk_log_size":89640,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22452984,"kb_used_data":21852324,"kb_used_omap":12,"kb_used_meta":600563,"kb_avail":727508232,"statfs":{"total":767960285184,"available":744968429568,"internally_reserved":0,"allocated":22376779776,"data_stored":22373654396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":614977072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":461635584,"num_objects":7044,"num_object_clones":0,"num_object_copies":21132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7044,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7044,"num_write_kb":450816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002698"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":128,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.855607+0000","last_change":"2024-03-24T11:53:56.612999+0000","last_active":"2024-03-24T11:55:55.855607+0000","last_peered":"2024-03-24T11:55:55.855607+0000","last_clean":"2024-03-24T11:55:55.855607+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:55:55.855607+0000","last_undegraded":"2024-03-24T11:55:55.855607+0000","last_fullsized":"2024-03-24T11:55:55.855607+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:56.612915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:56.612915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":138,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852578+0000","last_change":"2024-03-24T11:54:50.876045+0000","last_active":"2024-03-24T11:55:55.852578+0000","last_peered":"2024-03-24T11:55:55.852578+0000","last_clean":"2024-03-24T11:55:55.852578+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:55:55.852578+0000","last_undegraded":"2024-03-24T11:55:55.852578+0000","last_fullsized":"2024-03-24T11:55:55.852578+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:50.875918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:54:50.875918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:19.746580+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.846590+0000","last_change":"2024-03-24T11:54:13.817576+0000","last_active":"2024-03-24T11:55:55.846590+0000","last_peered":"2024-03-24T11:55:55.846590+0000","last_clean":"2024-03-24T11:55:55.846590+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:55:55.846590+0000","last_undegraded":"2024-03-24T11:55:55.846590+0000","last_fullsized":"2024-03-24T11:55:55.846590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T11:54:13.817437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"117'22","reported_seq":176,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.857939+0000","last_change":"2024-03-24T11:55:21.286888+0000","last_active":"2024-03-24T11:55:55.857939+0000","last_peered":"2024-03-24T11:55:55.857939+0000","last_clean":"2024-03-24T11:55:55.857939+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:55:55.857939+0000","last_undegraded":"2024-03-24T11:55:55.857939+0000","last_fullsized":"2024-03-24T11:55:55.857939+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'22","last_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T11:55:21.286727+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:35.123566+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":134,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852739+0000","last_change":"2024-03-24T11:55:12.810702+0000","last_active":"2024-03-24T11:55:55.852739+0000","last_peered":"2024-03-24T11:55:55.852739+0000","last_clean":"2024-03-24T11:55:55.852739+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:55:55.852739+0000","last_undegraded":"2024-03-24T11:55:55.852739+0000","last_fullsized":"2024-03-24T11:55:55.852739+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:12.810626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.810626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:30.053632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":131,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856011+0000","last_change":"2024-03-24T11:53:55.873297+0000","last_active":"2024-03-24T11:55:55.856011+0000","last_peered":"2024-03-24T11:55:55.856011+0000","last_clean":"2024-03-24T11:55:55.856011+0000","last_became_active":"2024-03-24T11:52:19.378282+0000","last_became_peered":"2024-03-24T11:52:19.378282+0000","last_unstale":"2024-03-24T11:55:55.856011+0000","last_undegraded":"2024-03-24T11:55:55.856011+0000","last_fullsized":"2024-03-24T11:55:55.856011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:55.873215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:55.873215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.859109+0000","last_change":"2024-03-24T11:55:20.239169+0000","last_active":"2024-03-24T11:55:55.859109+0000","last_peered":"2024-03-24T11:55:55.859109+0000","last_clean":"2024-03-24T11:55:55.859109+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:55:55.859109+0000","last_undegraded":"2024-03-24T11:55:55.859109+0000","last_fullsized":"2024-03-24T11:55:55.859109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.239058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.239058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.694647+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":150,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852760+0000","last_change":"2024-03-24T11:54:07.689916+0000","last_active":"2024-03-24T11:55:55.852760+0000","last_peered":"2024-03-24T11:55:55.852760+0000","last_clean":"2024-03-24T11:55:55.852760+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:55:55.852760+0000","last_undegraded":"2024-03-24T11:55:55.852760+0000","last_fullsized":"2024-03-24T11:55:55.852760+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:54:07.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:07.689767+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":134,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.844244+0000","last_change":"2024-03-24T11:54:54.848417+0000","last_active":"2024-03-24T11:55:55.844244+0000","last_peered":"2024-03-24T11:55:55.844244+0000","last_clean":"2024-03-24T11:55:55.844244+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:55:55.844244+0000","last_undegraded":"2024-03-24T11:55:55.844244+0000","last_fullsized":"2024-03-24T11:55:55.844244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:54.848274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:54:54.848274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:11.927198+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:56:00.086458+0000","last_change":"2024-03-24T11:56:00.086492+0000","last_active":"2024-03-24T11:56:00.086458+0000","last_peered":"2024-03-24T11:56:00.086458+0000","last_clean":"2024-03-24T11:56:00.086458+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:56:00.086458+0000","last_undegraded":"2024-03-24T11:56:00.086458+0000","last_fullsized":"2024-03-24T11:56:00.086458+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:00.086396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:00.086396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:01.178550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.848715+0000","last_change":"2024-03-24T11:55:09.427153+0000","last_active":"2024-03-24T11:55:55.848715+0000","last_peered":"2024-03-24T11:55:55.848715+0000","last_clean":"2024-03-24T11:55:55.848715+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:55:55.848715+0000","last_undegraded":"2024-03-24T11:55:55.848715+0000","last_fullsized":"2024-03-24T11:55:55.848715+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:09.427033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:09.427033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:11.944518+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":154,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853117+0000","last_change":"2024-03-24T11:54:56.987528+0000","last_active":"2024-03-24T11:55:55.853117+0000","last_peered":"2024-03-24T11:55:55.853117+0000","last_clean":"2024-03-24T11:55:55.853117+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:55:55.853117+0000","last_undegraded":"2024-03-24T11:55:55.853117+0000","last_fullsized":"2024-03-24T11:55:55.853117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:54:56.987400+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_clean_scrub_stamp":"2024-03-24T11:54:56.987400+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:08.251477+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":127,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845753+0000","last_change":"2024-03-24T11:54:58.045843+0000","last_active":"2024-03-24T11:55:55.845753+0000","last_peered":"2024-03-24T11:55:55.845753+0000","last_clean":"2024-03-24T11:55:55.845753+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:55:55.845753+0000","last_undegraded":"2024-03-24T11:55:55.845753+0000","last_fullsized":"2024-03-24T11:55:55.845753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.045730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.045730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:26.511106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":130,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.844780+0000","last_change":"2024-03-24T11:53:51.845805+0000","last_active":"2024-03-24T11:55:55.844780+0000","last_peered":"2024-03-24T11:55:55.844780+0000","last_clean":"2024-03-24T11:55:55.844780+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:55:55.844780+0000","last_undegraded":"2024-03-24T11:55:55.844780+0000","last_fullsized":"2024-03-24T11:55:55.844780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:51.845701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:51.845701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:56.641186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.849892+0000","last_change":"2024-03-24T11:55:07.333642+0000","last_active":"2024-03-24T11:55:55.849892+0000","last_peered":"2024-03-24T11:55:55.849892+0000","last_clean":"2024-03-24T11:55:55.849892+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:55:55.849892+0000","last_undegraded":"2024-03-24T11:55:55.849892+0000","last_fullsized":"2024-03-24T11:55:55.849892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:07.333529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:07.333529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:15.496508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":158,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852840+0000","last_change":"2024-03-24T11:54:47.842331+0000","last_active":"2024-03-24T11:55:55.852840+0000","last_peered":"2024-03-24T11:55:55.852840+0000","last_clean":"2024-03-24T11:55:55.852840+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:55:55.852840+0000","last_undegraded":"2024-03-24T11:55:55.852840+0000","last_fullsized":"2024-03-24T11:55:55.852840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:54:47.842199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:47.842199+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:00.497022+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.857993+0000","last_change":"2024-03-24T11:54:58.383503+0000","last_active":"2024-03-24T11:55:55.857993+0000","last_peered":"2024-03-24T11:55:55.857993+0000","last_clean":"2024-03-24T11:55:55.857993+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:55:55.857993+0000","last_undegraded":"2024-03-24T11:55:55.857993+0000","last_fullsized":"2024-03-24T11:55:55.857993+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.383392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.383392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:11.089085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":130,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853214+0000","last_change":"2024-03-24T11:54:15.707976+0000","last_active":"2024-03-24T11:55:55.853214+0000","last_peered":"2024-03-24T11:55:55.853214+0000","last_clean":"2024-03-24T11:55:55.853214+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:55:55.853214+0000","last_undegraded":"2024-03-24T11:55:55.853214+0000","last_fullsized":"2024-03-24T11:55:55.853214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.707872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.707872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":154,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858502+0000","last_change":"2024-03-24T11:54:59.374964+0000","last_active":"2024-03-24T11:55:55.858502+0000","last_peered":"2024-03-24T11:55:55.858502+0000","last_clean":"2024-03-24T11:55:55.858502+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:55:55.858502+0000","last_undegraded":"2024-03-24T11:55:55.858502+0000","last_fullsized":"2024-03-24T11:55:55.858502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:54:59.374820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:59.374820+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:20.757347+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":190,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856953+0000","last_change":"2024-03-24T11:55:19.242716+0000","last_active":"2024-03-24T11:55:55.856953+0000","last_peered":"2024-03-24T11:55:55.856953+0000","last_clean":"2024-03-24T11:55:55.856953+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:55:55.856953+0000","last_undegraded":"2024-03-24T11:55:55.856953+0000","last_fullsized":"2024-03-24T11:55:55.856953+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:55:19.242577+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.242577+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:40.357687+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":134,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853640+0000","last_change":"2024-03-24T11:54:52.942734+0000","last_active":"2024-03-24T11:55:55.853640+0000","last_peered":"2024-03-24T11:55:55.853640+0000","last_clean":"2024-03-24T11:55:55.853640+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:55:55.853640+0000","last_undegraded":"2024-03-24T11:55:55.853640+0000","last_fullsized":"2024-03-24T11:55:55.853640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:52.942614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.942614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:14.652492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.857160+0000","last_change":"2024-03-24T11:55:14.026391+0000","last_active":"2024-03-24T11:55:55.857160+0000","last_peered":"2024-03-24T11:55:55.857160+0000","last_clean":"2024-03-24T11:55:55.857160+0000","last_became_active":"2024-03-24T11:52:19.384249+0000","last_became_peered":"2024-03-24T11:52:19.384249+0000","last_unstale":"2024-03-24T11:55:55.857160+0000","last_undegraded":"2024-03-24T11:55:55.857160+0000","last_fullsized":"2024-03-24T11:55:55.857160+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:14.026252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.026252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.097029+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851302+0000","last_change":"2024-03-24T11:54:57.016465+0000","last_active":"2024-03-24T11:55:55.851302+0000","last_peered":"2024-03-24T11:55:55.851302+0000","last_clean":"2024-03-24T11:55:55.851302+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:55:55.851302+0000","last_undegraded":"2024-03-24T11:55:55.851302+0000","last_fullsized":"2024-03-24T11:55:55.851302+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:57.016388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:57.016388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:57.237683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"115'22","reported_seq":176,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853396+0000","last_change":"2024-03-24T11:55:12.618998+0000","last_active":"2024-03-24T11:55:55.853396+0000","last_peered":"2024-03-24T11:55:55.853396+0000","last_clean":"2024-03-24T11:55:55.853396+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:55:55.853396+0000","last_undegraded":"2024-03-24T11:55:55.853396+0000","last_fullsized":"2024-03-24T11:55:55.853396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'22","last_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.618646+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:26.578040+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856746+0000","last_change":"2024-03-24T11:55:18.959374+0000","last_active":"2024-03-24T11:55:55.856746+0000","last_peered":"2024-03-24T11:55:55.856746+0000","last_clean":"2024-03-24T11:55:55.856746+0000","last_became_active":"2024-03-24T11:52:21.390764+0000","last_became_peered":"2024-03-24T11:52:21.390764+0000","last_unstale":"2024-03-24T11:55:55.856746+0000","last_undegraded":"2024-03-24T11:55:55.856746+0000","last_fullsized":"2024-03-24T11:55:55.856746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.958919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.958919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:20.739685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.846213+0000","last_change":"2024-03-24T11:55:06.054268+0000","last_active":"2024-03-24T11:55:55.846213+0000","last_peered":"2024-03-24T11:55:55.846213+0000","last_clean":"2024-03-24T11:55:55.846213+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:55:55.846213+0000","last_undegraded":"2024-03-24T11:55:55.846213+0000","last_fullsized":"2024-03-24T11:55:55.846213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.054160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.054160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.176617+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":141,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856291+0000","last_change":"2024-03-24T11:54:06.564061+0000","last_active":"2024-03-24T11:55:55.856291+0000","last_peered":"2024-03-24T11:55:55.856291+0000","last_clean":"2024-03-24T11:55:55.856291+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:55:55.856291+0000","last_undegraded":"2024-03-24T11:55:55.856291+0000","last_fullsized":"2024-03-24T11:55:55.856291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:54:06.563966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:06.563966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":161,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851995+0000","last_change":"2024-03-24T11:54:50.090222+0000","last_active":"2024-03-24T11:55:55.851995+0000","last_peered":"2024-03-24T11:55:55.851995+0000","last_clean":"2024-03-24T11:55:55.851995+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:55:55.851995+0000","last_undegraded":"2024-03-24T11:55:55.851995+0000","last_fullsized":"2024-03-24T11:55:55.851995+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:54:50.090089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:50.090089+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.925144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.850050+0000","last_change":"2024-03-24T11:54:54.491445+0000","last_active":"2024-03-24T11:55:55.850050+0000","last_peered":"2024-03-24T11:55:55.850050+0000","last_clean":"2024-03-24T11:55:55.850050+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:55:55.850050+0000","last_undegraded":"2024-03-24T11:55:55.850050+0000","last_fullsized":"2024-03-24T11:55:55.850050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:54:54.491303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:54.491303+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:02.713090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":187,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851992+0000","last_change":"2024-03-24T11:53:47.334525+0000","last_active":"2024-03-24T11:55:55.851992+0000","last_peered":"2024-03-24T11:55:55.851992+0000","last_clean":"2024-03-24T11:55:55.851992+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:55:55.851992+0000","last_undegraded":"2024-03-24T11:55:55.851992+0000","last_fullsized":"2024-03-24T11:55:55.851992+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:53:47.334412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:47.334412+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.844386+0000","last_change":"2024-03-24T11:55:00.754485+0000","last_active":"2024-03-24T11:55:55.844386+0000","last_peered":"2024-03-24T11:55:55.844386+0000","last_clean":"2024-03-24T11:55:55.844386+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:55:55.844386+0000","last_undegraded":"2024-03-24T11:55:55.844386+0000","last_fullsized":"2024-03-24T11:55:55.844386+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:00.754339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:00.754339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:14.930276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":155,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.846119+0000","last_change":"2024-03-24T11:53:46.014153+0000","last_active":"2024-03-24T11:55:55.846119+0000","last_peered":"2024-03-24T11:55:55.846119+0000","last_clean":"2024-03-24T11:55:55.846119+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:55:55.846119+0000","last_undegraded":"2024-03-24T11:55:55.846119+0000","last_fullsized":"2024-03-24T11:55:55.846119+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_clean_scrub_stamp":"2024-03-24T11:53:46.014065+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082702+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":128,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858826+0000","last_change":"2024-03-24T11:54:49.424771+0000","last_active":"2024-03-24T11:55:55.858826+0000","last_peered":"2024-03-24T11:55:55.858826+0000","last_clean":"2024-03-24T11:55:55.858826+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:55:55.858826+0000","last_undegraded":"2024-03-24T11:55:55.858826+0000","last_fullsized":"2024-03-24T11:55:55.858826+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:49.424403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:49.424403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:56:01.069368+0000","last_change":"2024-03-24T11:56:01.069407+0000","last_active":"2024-03-24T11:56:01.069368+0000","last_peered":"2024-03-24T11:56:01.069368+0000","last_clean":"2024-03-24T11:56:01.069368+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:56:01.069368+0000","last_undegraded":"2024-03-24T11:56:01.069368+0000","last_fullsized":"2024-03-24T11:56:01.069368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:01.069280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.632287+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.069280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:11.863245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":138,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851845+0000","last_change":"2024-03-24T11:53:35.343688+0000","last_active":"2024-03-24T11:55:55.851845+0000","last_peered":"2024-03-24T11:55:55.851845+0000","last_clean":"2024-03-24T11:55:55.851845+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:55:55.851845+0000","last_undegraded":"2024-03-24T11:55:55.851845+0000","last_fullsized":"2024-03-24T11:55:55.851845+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:35.343594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:35.343594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.924957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":155,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.857905+0000","last_change":"2024-03-24T11:54:14.552219+0000","last_active":"2024-03-24T11:55:55.857905+0000","last_peered":"2024-03-24T11:55:55.857905+0000","last_clean":"2024-03-24T11:55:55.857905+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:55:55.857905+0000","last_undegraded":"2024-03-24T11:55:55.857905+0000","last_fullsized":"2024-03-24T11:55:55.857905+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:54:14.552123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:14.552123+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":127,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.847804+0000","last_change":"2024-03-24T11:53:51.547818+0000","last_active":"2024-03-24T11:55:55.847804+0000","last_peered":"2024-03-24T11:55:55.847804+0000","last_clean":"2024-03-24T11:55:55.847804+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:55:55.847804+0000","last_undegraded":"2024-03-24T11:55:55.847804+0000","last_fullsized":"2024-03-24T11:55:55.847804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:51.547680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:51.547680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:56.187751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":138,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851215+0000","last_change":"2024-03-24T11:54:58.059296+0000","last_active":"2024-03-24T11:55:55.851215+0000","last_peered":"2024-03-24T11:55:55.851215+0000","last_clean":"2024-03-24T11:55:55.851215+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:55:55.851215+0000","last_undegraded":"2024-03-24T11:55:55.851215+0000","last_fullsized":"2024-03-24T11:55:55.851215+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.059216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.059216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:21.338010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845433+0000","last_change":"2024-03-24T11:55:18.619637+0000","last_active":"2024-03-24T11:55:55.845433+0000","last_peered":"2024-03-24T11:55:55.845433+0000","last_clean":"2024-03-24T11:55:55.845433+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:55:55.845433+0000","last_undegraded":"2024-03-24T11:55:55.845433+0000","last_fullsized":"2024-03-24T11:55:55.845433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.619533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.619533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:43.172807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":150,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858299+0000","last_change":"2024-03-24T11:55:00.363697+0000","last_active":"2024-03-24T11:55:55.858299+0000","last_peered":"2024-03-24T11:55:55.858299+0000","last_clean":"2024-03-24T11:55:55.858299+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:55:55.858299+0000","last_undegraded":"2024-03-24T11:55:55.858299+0000","last_fullsized":"2024-03-24T11:55:55.858299+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T11:55:00.363559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:00.363559+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.459697+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":128,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856110+0000","last_change":"2024-03-24T11:54:51.408148+0000","last_active":"2024-03-24T11:55:55.856110+0000","last_peered":"2024-03-24T11:55:55.856110+0000","last_clean":"2024-03-24T11:55:55.856110+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:55:55.856110+0000","last_undegraded":"2024-03-24T11:55:55.856110+0000","last_fullsized":"2024-03-24T11:55:55.856110+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:51.408018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:51.408018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.125170+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.132276+0000","last_change":"2024-03-24T11:55:56.132313+0000","last_active":"2024-03-24T11:55:56.132276+0000","last_peered":"2024-03-24T11:55:56.132276+0000","last_clean":"2024-03-24T11:55:56.132276+0000","last_became_active":"2024-03-24T11:52:19.377484+0000","last_became_peered":"2024-03-24T11:52:19.377484+0000","last_unstale":"2024-03-24T11:55:56.132276+0000","last_undegraded":"2024-03-24T11:55:56.132276+0000","last_fullsized":"2024-03-24T11:55:56.132276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:56.132189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:55:56.132189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:01.345699+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856916+0000","last_change":"2024-03-24T11:55:15.993168+0000","last_active":"2024-03-24T11:55:55.856916+0000","last_peered":"2024-03-24T11:55:55.856916+0000","last_clean":"2024-03-24T11:55:55.856916+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:55:55.856916+0000","last_undegraded":"2024-03-24T11:55:55.856916+0000","last_fullsized":"2024-03-24T11:55:55.856916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:15.992998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:15.992998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:24.619769+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":153,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853281+0000","last_change":"2024-03-24T11:53:58.582983+0000","last_active":"2024-03-24T11:55:55.853281+0000","last_peered":"2024-03-24T11:55:55.853281+0000","last_clean":"2024-03-24T11:55:55.853281+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:55:55.853281+0000","last_undegraded":"2024-03-24T11:55:55.853281+0000","last_fullsized":"2024-03-24T11:55:55.853281+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:53:58.582825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:58.582825+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.843604+0000","last_change":"2024-03-24T11:55:19.108172+0000","last_active":"2024-03-24T11:55:55.843604+0000","last_peered":"2024-03-24T11:55:55.843604+0000","last_clean":"2024-03-24T11:55:55.843604+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:55:55.843604+0000","last_undegraded":"2024-03-24T11:55:55.843604+0000","last_fullsized":"2024-03-24T11:55:55.843604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.108015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.299012+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858588+0000","last_change":"2024-03-24T11:55:22.292901+0000","last_active":"2024-03-24T11:55:55.858588+0000","last_peered":"2024-03-24T11:55:55.858588+0000","last_clean":"2024-03-24T11:55:55.858588+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:55:55.858588+0000","last_undegraded":"2024-03-24T11:55:55.858588+0000","last_fullsized":"2024-03-24T11:55:55.858588+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:22.292785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.292785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:22.310604+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851459+0000","last_change":"2024-03-24T11:55:01.977733+0000","last_active":"2024-03-24T11:55:55.851459+0000","last_peered":"2024-03-24T11:55:55.851459+0000","last_clean":"2024-03-24T11:55:55.851459+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:55:55.851459+0000","last_undegraded":"2024-03-24T11:55:55.851459+0000","last_fullsized":"2024-03-24T11:55:55.851459+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.977610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.977610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:17.519759+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856568+0000","last_change":"2024-03-24T11:54:50.854728+0000","last_active":"2024-03-24T11:55:55.856568+0000","last_peered":"2024-03-24T11:55:55.856568+0000","last_clean":"2024-03-24T11:55:55.856568+0000","last_became_active":"2024-03-24T11:52:17.355956+0000","last_became_peered":"2024-03-24T11:52:17.355956+0000","last_unstale":"2024-03-24T11:55:55.856568+0000","last_undegraded":"2024-03-24T11:55:55.856568+0000","last_fullsized":"2024-03-24T11:55:55.856568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:54:50.854256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:50.854256+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":134,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853996+0000","last_change":"2024-03-24T11:55:11.806686+0000","last_active":"2024-03-24T11:55:55.853996+0000","last_peered":"2024-03-24T11:55:55.853996+0000","last_clean":"2024-03-24T11:55:55.853996+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:55:55.853996+0000","last_undegraded":"2024-03-24T11:55:55.853996+0000","last_fullsized":"2024-03-24T11:55:55.853996+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.806582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.806582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:34.108587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":193,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.863074+0000","last_change":"2024-03-24T11:55:01.393587+0000","last_active":"2024-03-24T11:55:55.863074+0000","last_peered":"2024-03-24T11:55:55.863074+0000","last_clean":"2024-03-24T11:55:55.863074+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:55:55.863074+0000","last_undegraded":"2024-03-24T11:55:55.863074+0000","last_fullsized":"2024-03-24T11:55:55.863074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:01.393318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.393318+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:12.523269+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845144+0000","last_change":"2024-03-24T11:54:05.851953+0000","last_active":"2024-03-24T11:55:55.845144+0000","last_peered":"2024-03-24T11:55:55.845144+0000","last_clean":"2024-03-24T11:55:55.845144+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:55:55.845144+0000","last_undegraded":"2024-03-24T11:55:55.845144+0000","last_fullsized":"2024-03-24T11:55:55.845144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:05.851860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:05.851860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":158,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845495+0000","last_change":"2024-03-24T11:55:13.643173+0000","last_active":"2024-03-24T11:55:55.845495+0000","last_peered":"2024-03-24T11:55:55.845495+0000","last_clean":"2024-03-24T11:55:55.845495+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:55:55.845495+0000","last_undegraded":"2024-03-24T11:55:55.845495+0000","last_fullsized":"2024-03-24T11:55:55.845495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:55:13.643063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:13.643063+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:20.737682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":127,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845544+0000","last_change":"2024-03-24T11:53:55.915626+0000","last_active":"2024-03-24T11:55:55.845544+0000","last_peered":"2024-03-24T11:55:55.845544+0000","last_clean":"2024-03-24T11:55:55.845544+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:55:55.845544+0000","last_undegraded":"2024-03-24T11:55:55.845544+0000","last_fullsized":"2024-03-24T11:55:55.845544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:55.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:55.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":192,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845370+0000","last_change":"2024-03-24T11:54:58.781360+0000","last_active":"2024-03-24T11:55:55.845370+0000","last_peered":"2024-03-24T11:55:55.845370+0000","last_clean":"2024-03-24T11:55:55.845370+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:55:55.845370+0000","last_undegraded":"2024-03-24T11:55:55.845370+0000","last_fullsized":"2024-03-24T11:55:55.845370+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:54:58.781231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.781231+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.844678+0000","last_change":"2024-03-24T11:55:21.647704+0000","last_active":"2024-03-24T11:55:55.844678+0000","last_peered":"2024-03-24T11:55:55.844678+0000","last_clean":"2024-03-24T11:55:55.844678+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:55:55.844678+0000","last_undegraded":"2024-03-24T11:55:55.844678+0000","last_fullsized":"2024-03-24T11:55:55.844678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:21.647570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T11:55:21.647570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:30.306492+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":164,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.862934+0000","last_change":"2024-03-24T11:55:06.420550+0000","last_active":"2024-03-24T11:55:55.862934+0000","last_peered":"2024-03-24T11:55:55.862934+0000","last_clean":"2024-03-24T11:55:55.862934+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:55:55.862934+0000","last_undegraded":"2024-03-24T11:55:55.862934+0000","last_fullsized":"2024-03-24T11:55:55.862934+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:55:06.420406+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.420406+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.347160+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.848670+0000","last_change":"2024-03-24T11:55:11.462967+0000","last_active":"2024-03-24T11:55:55.848670+0000","last_peered":"2024-03-24T11:55:55.848670+0000","last_clean":"2024-03-24T11:55:55.848670+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:55:55.848670+0000","last_undegraded":"2024-03-24T11:55:55.848670+0000","last_fullsized":"2024-03-24T11:55:55.848670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.462846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.462846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:18.783674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":130,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845220+0000","last_change":"2024-03-24T11:53:52.802831+0000","last_active":"2024-03-24T11:55:55.845220+0000","last_peered":"2024-03-24T11:55:55.845220+0000","last_clean":"2024-03-24T11:55:55.845220+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:55:55.845220+0000","last_undegraded":"2024-03-24T11:55:55.845220+0000","last_fullsized":"2024-03-24T11:55:55.845220+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:52.802702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:52.802702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:56.641216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851174+0000","last_change":"2024-03-24T11:54:48.200814+0000","last_active":"2024-03-24T11:55:55.851174+0000","last_peered":"2024-03-24T11:55:55.851174+0000","last_clean":"2024-03-24T11:55:55.851174+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:55:55.851174+0000","last_undegraded":"2024-03-24T11:55:55.851174+0000","last_fullsized":"2024-03-24T11:55:55.851174+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:48.200693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:48.200693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:59.819727+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":153,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858084+0000","last_change":"2024-03-24T11:54:52.361385+0000","last_active":"2024-03-24T11:55:55.858084+0000","last_peered":"2024-03-24T11:55:55.858084+0000","last_clean":"2024-03-24T11:55:55.858084+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:55:55.858084+0000","last_undegraded":"2024-03-24T11:55:55.858084+0000","last_fullsized":"2024-03-24T11:55:55.858084+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:54:52.361260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.361260+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:12.035583+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":128,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852987+0000","last_change":"2024-03-24T11:53:58.355251+0000","last_active":"2024-03-24T11:55:55.852987+0000","last_peered":"2024-03-24T11:55:55.852987+0000","last_clean":"2024-03-24T11:55:55.852987+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:55:55.852987+0000","last_undegraded":"2024-03-24T11:55:55.852987+0000","last_fullsized":"2024-03-24T11:55:55.852987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:58.355159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:58.355159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.583092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":132,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858176+0000","last_change":"2024-03-24T11:54:15.542707+0000","last_active":"2024-03-24T11:55:55.858176+0000","last_peered":"2024-03-24T11:55:55.858176+0000","last_clean":"2024-03-24T11:55:55.858176+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:55:55.858176+0000","last_undegraded":"2024-03-24T11:55:55.858176+0000","last_fullsized":"2024-03-24T11:55:55.858176+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.542625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.542625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852941+0000","last_change":"2024-03-24T11:55:01.390204+0000","last_active":"2024-03-24T11:55:55.852941+0000","last_peered":"2024-03-24T11:55:55.852941+0000","last_clean":"2024-03-24T11:55:55.852941+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:55:55.852941+0000","last_undegraded":"2024-03-24T11:55:55.852941+0000","last_fullsized":"2024-03-24T11:55:55.852941+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.390019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.390019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:30.098357+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"115'13","reported_seq":163,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845856+0000","last_change":"2024-03-24T11:55:16.094065+0000","last_active":"2024-03-24T11:55:55.845856+0000","last_peered":"2024-03-24T11:55:55.845856+0000","last_clean":"2024-03-24T11:55:55.845856+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:55:55.845856+0000","last_undegraded":"2024-03-24T11:55:55.845856+0000","last_fullsized":"2024-03-24T11:55:55.845856+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'13","last_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T11:55:16.093668+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:25.577045+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"127'7152","reported_seq":7222,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.240374+0000","last_change":"2024-03-24T11:54:16.232420+0000","last_active":"2024-03-24T11:55:56.240374+0000","last_peered":"2024-03-24T11:55:56.240374+0000","last_clean":"2024-03-24T11:55:56.240374+0000","last_became_active":"2024-03-24T11:54:16.232098+0000","last_became_peered":"2024-03-24T11:54:16.232098+0000","last_unstale":"2024-03-24T11:55:56.240374+0000","last_undegraded":"2024-03-24T11:55:56.240374+0000","last_fullsized":"2024-03-24T11:55:56.240374+0000","mapping_epoch":97,"log_start":"108'2400","ondisk_log_start":"108'2400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468713472,"num_objects":7152,"num_object_clones":0,"num_object_copies":21456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7152,"num_write_kb":457728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":128,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858662+0000","last_change":"2024-03-24T11:53:58.599178+0000","last_active":"2024-03-24T11:55:55.858662+0000","last_peered":"2024-03-24T11:55:55.858662+0000","last_clean":"2024-03-24T11:55:55.858662+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:55:55.858662+0000","last_undegraded":"2024-03-24T11:55:55.858662+0000","last_fullsized":"2024-03-24T11:55:55.858662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:58.599068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:58.599068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":158,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853880+0000","last_change":"2024-03-24T11:55:20.861818+0000","last_active":"2024-03-24T11:55:55.853880+0000","last_peered":"2024-03-24T11:55:55.853880+0000","last_clean":"2024-03-24T11:55:55.853880+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:55:55.853880+0000","last_undegraded":"2024-03-24T11:55:55.853880+0000","last_fullsized":"2024-03-24T11:55:55.853880+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:55:20.861648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.861648+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:38.570780+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":183,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.854690+0000","last_change":"2024-03-24T11:54:48.888639+0000","last_active":"2024-03-24T11:55:55.854690+0000","last_peered":"2024-03-24T11:55:55.854690+0000","last_clean":"2024-03-24T11:55:55.854690+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:55:55.854690+0000","last_undegraded":"2024-03-24T11:55:55.854690+0000","last_fullsized":"2024-03-24T11:55:55.854690+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:54:48.888525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:48.888525+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:00.110375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852829+0000","last_change":"2024-03-24T11:55:19.652898+0000","last_active":"2024-03-24T11:55:55.852829+0000","last_peered":"2024-03-24T11:55:55.852829+0000","last_clean":"2024-03-24T11:55:55.852829+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:55:55.852829+0000","last_undegraded":"2024-03-24T11:55:55.852829+0000","last_fullsized":"2024-03-24T11:55:55.852829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:19.652641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.652641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:28.881745+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"127'7324","reported_seq":7394,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:56:01.110191+0000","last_change":"2024-03-24T11:54:16.238387+0000","last_active":"2024-03-24T11:56:01.110191+0000","last_peered":"2024-03-24T11:56:01.110191+0000","last_clean":"2024-03-24T11:56:01.110191+0000","last_became_active":"2024-03-24T11:54:16.238055+0000","last_became_peered":"2024-03-24T11:54:16.238055+0000","last_unstale":"2024-03-24T11:56:01.110191+0000","last_undegraded":"2024-03-24T11:56:01.110191+0000","last_fullsized":"2024-03-24T11:56:01.110191+0000","mapping_epoch":97,"log_start":"108'2000","ondisk_log_start":"108'2000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479985664,"num_objects":7324,"num_object_clones":0,"num_object_copies":21972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":468736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":127,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845418+0000","last_change":"2024-03-24T11:54:04.892146+0000","last_active":"2024-03-24T11:55:55.845418+0000","last_peered":"2024-03-24T11:55:55.845418+0000","last_clean":"2024-03-24T11:55:55.845418+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:55:55.845418+0000","last_undegraded":"2024-03-24T11:55:55.845418+0000","last_fullsized":"2024-03-24T11:55:55.845418+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:04.892038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:04.892038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.857476+0000","last_change":"2024-03-24T11:55:04.005537+0000","last_active":"2024-03-24T11:55:55.857476+0000","last_peered":"2024-03-24T11:55:55.857476+0000","last_clean":"2024-03-24T11:55:55.857476+0000","last_became_active":"2024-03-24T11:52:15.138792+0000","last_became_peered":"2024-03-24T11:52:15.138792+0000","last_unstale":"2024-03-24T11:55:55.857476+0000","last_undegraded":"2024-03-24T11:55:55.857476+0000","last_fullsized":"2024-03-24T11:55:55.857476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:04.005410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:55:04.005410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:19.205017+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":158,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856677+0000","last_change":"2024-03-24T11:55:01.917263+0000","last_active":"2024-03-24T11:55:55.856677+0000","last_peered":"2024-03-24T11:55:55.856677+0000","last_clean":"2024-03-24T11:55:55.856677+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:55:55.856677+0000","last_undegraded":"2024-03-24T11:55:55.856677+0000","last_fullsized":"2024-03-24T11:55:55.856677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:55:01.917120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.917120+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.443473+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":184,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845314+0000","last_change":"2024-03-24T11:54:00.896739+0000","last_active":"2024-03-24T11:55:55.845314+0000","last_peered":"2024-03-24T11:55:55.845314+0000","last_clean":"2024-03-24T11:55:55.845314+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:55:55.845314+0000","last_undegraded":"2024-03-24T11:55:55.845314+0000","last_fullsized":"2024-03-24T11:55:55.845314+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:54:00.896635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:00.896635+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"127'7444","reported_seq":7514,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:56:01.127947+0000","last_change":"2024-03-24T11:54:16.240916+0000","last_active":"2024-03-24T11:56:01.127947+0000","last_peered":"2024-03-24T11:56:01.127947+0000","last_clean":"2024-03-24T11:56:01.127947+0000","last_became_active":"2024-03-24T11:54:16.240514+0000","last_became_peered":"2024-03-24T11:54:16.240514+0000","last_unstale":"2024-03-24T11:56:01.127947+0000","last_undegraded":"2024-03-24T11:56:01.127947+0000","last_fullsized":"2024-03-24T11:56:01.127947+0000","mapping_epoch":97,"log_start":"106'1300","ondisk_log_start":"106'1300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":6144,"log_dups_size":0,"ondisk_log_size":6144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487849984,"num_objects":7444,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7444,"num_write_kb":476416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.859185+0000","last_change":"2024-03-24T11:55:12.389096+0000","last_active":"2024-03-24T11:55:55.859185+0000","last_peered":"2024-03-24T11:55:55.859185+0000","last_clean":"2024-03-24T11:55:55.859185+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:55:55.859185+0000","last_undegraded":"2024-03-24T11:55:55.859185+0000","last_fullsized":"2024-03-24T11:55:55.859185+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:12.388835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.388835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:32.363194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.844710+0000","last_change":"2024-03-24T11:54:52.822234+0000","last_active":"2024-03-24T11:55:55.844710+0000","last_peered":"2024-03-24T11:55:55.844710+0000","last_clean":"2024-03-24T11:55:55.844710+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:55:55.844710+0000","last_undegraded":"2024-03-24T11:55:55.844710+0000","last_fullsized":"2024-03-24T11:55:55.844710+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:52.822135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.822135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:12.399116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":180,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845137+0000","last_change":"2024-03-24T11:54:07.842648+0000","last_active":"2024-03-24T11:55:55.845137+0000","last_peered":"2024-03-24T11:55:55.845137+0000","last_clean":"2024-03-24T11:55:55.845137+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:55:55.845137+0000","last_undegraded":"2024-03-24T11:55:55.845137+0000","last_fullsized":"2024-03-24T11:55:55.845137+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-03-24T11:54:07.842555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:07.842555+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":131,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845191+0000","last_change":"2024-03-24T11:53:45.014753+0000","last_active":"2024-03-24T11:55:55.845191+0000","last_peered":"2024-03-24T11:55:55.845191+0000","last_clean":"2024-03-24T11:55:55.845191+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:55:55.845191+0000","last_undegraded":"2024-03-24T11:55:55.845191+0000","last_fullsized":"2024-03-24T11:55:55.845191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.014670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.014670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"127'7020","reported_seq":7090,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.318433+0000","last_change":"2024-03-24T11:54:16.238411+0000","last_active":"2024-03-24T11:55:56.318433+0000","last_peered":"2024-03-24T11:55:56.318433+0000","last_clean":"2024-03-24T11:55:56.318433+0000","last_became_active":"2024-03-24T11:54:16.238056+0000","last_became_peered":"2024-03-24T11:54:16.238056+0000","last_unstale":"2024-03-24T11:55:56.318433+0000","last_undegraded":"2024-03-24T11:55:56.318433+0000","last_fullsized":"2024-03-24T11:55:56.318433+0000","mapping_epoch":97,"log_start":"106'1500","ondisk_log_start":"106'1500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.086599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460062720,"num_objects":7020,"num_object_clones":0,"num_object_copies":21060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:57.171221+0000","last_change":"2024-03-24T11:55:57.171256+0000","last_active":"2024-03-24T11:55:57.171221+0000","last_peered":"2024-03-24T11:55:57.171221+0000","last_clean":"2024-03-24T11:55:57.171221+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:55:57.171221+0000","last_undegraded":"2024-03-24T11:55:57.171221+0000","last_fullsized":"2024-03-24T11:55:57.171221+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:57.171146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:57.171146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:09.649463+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":372,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.854949+0000","last_change":"2024-03-24T11:55:22.048179+0000","last_active":"2024-03-24T11:55:55.854949+0000","last_peered":"2024-03-24T11:55:55.854949+0000","last_clean":"2024-03-24T11:55:55.854949+0000","last_became_active":"2024-03-24T11:52:15.144954+0000","last_became_peered":"2024-03-24T11:52:15.144954+0000","last_unstale":"2024-03-24T11:55:55.854949+0000","last_undegraded":"2024-03-24T11:55:55.854949+0000","last_fullsized":"2024-03-24T11:55:55.854949+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:55:22.048036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.048036+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:46.092889+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":215,"num_read_kb":210,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853833+0000","last_change":"2024-03-24T11:54:48.278466+0000","last_active":"2024-03-24T11:55:55.853833+0000","last_peered":"2024-03-24T11:55:55.853833+0000","last_clean":"2024-03-24T11:55:55.853833+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:55:55.853833+0000","last_undegraded":"2024-03-24T11:55:55.853833+0000","last_fullsized":"2024-03-24T11:55:55.853833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:48.278341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:48.278341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:01.815810+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":162,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856287+0000","last_change":"2024-03-24T11:53:57.903905+0000","last_active":"2024-03-24T11:55:55.856287+0000","last_peered":"2024-03-24T11:55:55.856287+0000","last_clean":"2024-03-24T11:55:55.856287+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:55:55.856287+0000","last_undegraded":"2024-03-24T11:55:55.856287+0000","last_fullsized":"2024-03-24T11:55:55.856287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-03-24T11:53:57.903789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:57.903789+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.855456+0000","last_change":"2024-03-24T11:55:13.600385+0000","last_active":"2024-03-24T11:55:55.855456+0000","last_peered":"2024-03-24T11:55:55.855456+0000","last_clean":"2024-03-24T11:55:55.855456+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:55:55.855456+0000","last_undegraded":"2024-03-24T11:55:55.855456+0000","last_fullsized":"2024-03-24T11:55:55.855456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T11:55:13.600133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:19.274997+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"127'6817","reported_seq":6887,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.238044+0000","last_change":"2024-03-24T11:54:16.238815+0000","last_active":"2024-03-24T11:55:56.238044+0000","last_peered":"2024-03-24T11:55:56.238044+0000","last_clean":"2024-03-24T11:55:56.238044+0000","last_became_active":"2024-03-24T11:54:16.238515+0000","last_became_peered":"2024-03-24T11:54:16.238515+0000","last_unstale":"2024-03-24T11:55:56.238044+0000","last_undegraded":"2024-03-24T11:55:56.238044+0000","last_fullsized":"2024-03-24T11:55:56.238044+0000","mapping_epoch":97,"log_start":"106'1300","ondisk_log_start":"106'1300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5517,"log_dups_size":0,"ondisk_log_size":5517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446758912,"num_objects":6817,"num_object_clones":0,"num_object_copies":20451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6817,"num_write_kb":436288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":134,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845714+0000","last_change":"2024-03-24T11:55:10.692203+0000","last_active":"2024-03-24T11:55:55.845714+0000","last_peered":"2024-03-24T11:55:55.845714+0000","last_clean":"2024-03-24T11:55:55.845714+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:55:55.845714+0000","last_undegraded":"2024-03-24T11:55:55.845714+0000","last_fullsized":"2024-03-24T11:55:55.845714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:10.692130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:55:10.692130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:22.802403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":149,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856678+0000","last_change":"2024-03-24T11:53:55.571857+0000","last_active":"2024-03-24T11:55:55.856678+0000","last_peered":"2024-03-24T11:55:55.856678+0000","last_clean":"2024-03-24T11:55:55.856678+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:55:55.856678+0000","last_undegraded":"2024-03-24T11:55:55.856678+0000","last_fullsized":"2024-03-24T11:55:55.856678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T11:53:55.571746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:55.571746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.844886+0000","last_change":"2024-03-24T11:55:20.647830+0000","last_active":"2024-03-24T11:55:55.844886+0000","last_peered":"2024-03-24T11:55:55.844886+0000","last_clean":"2024-03-24T11:55:55.844886+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:55:55.844886+0000","last_undegraded":"2024-03-24T11:55:55.844886+0000","last_fullsized":"2024-03-24T11:55:55.844886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.647693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.647693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:37.648168+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":394,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.847909+0000","last_change":"2024-03-24T11:53:59.598556+0000","last_active":"2024-03-24T11:55:55.847909+0000","last_peered":"2024-03-24T11:55:55.847909+0000","last_clean":"2024-03-24T11:55:55.847909+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:55:55.847909+0000","last_undegraded":"2024-03-24T11:55:55.847909+0000","last_fullsized":"2024-03-24T11:55:55.847909+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:53:59.598408+0000","last_deep_scrub":"71'119","last_deep_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_clean_scrub_stamp":"2024-03-24T11:53:59.598408+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:56.187698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":210,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851641+0000","last_change":"2024-03-24T11:55:22.095371+0000","last_active":"2024-03-24T11:55:55.851641+0000","last_peered":"2024-03-24T11:55:55.851641+0000","last_clean":"2024-03-24T11:55:55.851641+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:55:55.851641+0000","last_undegraded":"2024-03-24T11:55:55.851641+0000","last_fullsized":"2024-03-24T11:55:55.851641+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:55:22.095232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.095232+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:31.457374+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.849998+0000","last_change":"2024-03-24T11:55:04.414736+0000","last_active":"2024-03-24T11:55:55.849998+0000","last_peered":"2024-03-24T11:55:55.849998+0000","last_clean":"2024-03-24T11:55:55.849998+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:55:55.849998+0000","last_undegraded":"2024-03-24T11:55:55.849998+0000","last_fullsized":"2024-03-24T11:55:55.849998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:04.414651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:04.414651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:31.408133+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"127'7353","reported_seq":7423,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:56:01.141814+0000","last_change":"2024-03-24T11:54:16.237252+0000","last_active":"2024-03-24T11:56:01.141814+0000","last_peered":"2024-03-24T11:56:01.141814+0000","last_clean":"2024-03-24T11:56:01.141814+0000","last_became_active":"2024-03-24T11:54:16.236927+0000","last_became_peered":"2024-03-24T11:54:16.236927+0000","last_unstale":"2024-03-24T11:56:01.141814+0000","last_undegraded":"2024-03-24T11:56:01.141814+0000","last_fullsized":"2024-03-24T11:56:01.141814+0000","mapping_epoch":97,"log_start":"108'2400","ondisk_log_start":"108'2400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":4953,"log_dups_size":0,"ondisk_log_size":4953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481886208,"num_objects":7353,"num_object_clones":0,"num_object_copies":22059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":470592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.857562+0000","last_change":"2024-03-24T11:55:14.967693+0000","last_active":"2024-03-24T11:55:55.857562+0000","last_peered":"2024-03-24T11:55:55.857562+0000","last_clean":"2024-03-24T11:55:55.857562+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:55:55.857562+0000","last_undegraded":"2024-03-24T11:55:55.857562+0000","last_fullsized":"2024-03-24T11:55:55.857562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:14.967137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.967137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:24.422604+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852735+0000","last_change":"2024-03-24T11:53:52.992618+0000","last_active":"2024-03-24T11:55:55.852735+0000","last_peered":"2024-03-24T11:55:55.852735+0000","last_clean":"2024-03-24T11:55:55.852735+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:55:55.852735+0000","last_undegraded":"2024-03-24T11:55:55.852735+0000","last_fullsized":"2024-03-24T11:55:55.852735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:52.992540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:52.992540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.847799+0000","last_change":"2024-03-24T11:54:49.540232+0000","last_active":"2024-03-24T11:55:55.847799+0000","last_peered":"2024-03-24T11:55:55.847799+0000","last_clean":"2024-03-24T11:55:55.847799+0000","last_became_active":"2024-03-24T11:52:15.146310+0000","last_became_peered":"2024-03-24T11:52:15.146310+0000","last_unstale":"2024-03-24T11:55:55.847799+0000","last_undegraded":"2024-03-24T11:55:55.847799+0000","last_fullsized":"2024-03-24T11:55:55.847799+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:49.540127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:49.540127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.600167+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":161,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845653+0000","last_change":"2024-03-24T11:55:14.029361+0000","last_active":"2024-03-24T11:55:55.845653+0000","last_peered":"2024-03-24T11:55:55.845653+0000","last_clean":"2024-03-24T11:55:55.845653+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:55:55.845653+0000","last_undegraded":"2024-03-24T11:55:55.845653+0000","last_fullsized":"2024-03-24T11:55:55.845653+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:55:14.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.029279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:26.539716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":130,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845643+0000","last_change":"2024-03-24T11:54:02.846430+0000","last_active":"2024-03-24T11:55:55.845643+0000","last_peered":"2024-03-24T11:55:55.845643+0000","last_clean":"2024-03-24T11:55:55.845643+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:55:55.845643+0000","last_undegraded":"2024-03-24T11:55:55.845643+0000","last_fullsized":"2024-03-24T11:55:55.845643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:02.846284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:02.846284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:56.641242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"127'7458","reported_seq":7528,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:56:01.121993+0000","last_change":"2024-03-24T11:54:16.237179+0000","last_active":"2024-03-24T11:56:01.121993+0000","last_peered":"2024-03-24T11:56:01.121993+0000","last_clean":"2024-03-24T11:56:01.121993+0000","last_became_active":"2024-03-24T11:54:16.236857+0000","last_became_peered":"2024-03-24T11:54:16.236857+0000","last_unstale":"2024-03-24T11:56:01.121993+0000","last_undegraded":"2024-03-24T11:56:01.121993+0000","last_fullsized":"2024-03-24T11:56:01.121993+0000","mapping_epoch":97,"log_start":"107'1700","ondisk_log_start":"107'1700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488767488,"num_objects":7458,"num_object_clones":0,"num_object_copies":22374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7458,"num_write_kb":477312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.855116+0000","last_change":"2024-03-24T11:54:48.768095+0000","last_active":"2024-03-24T11:55:55.855116+0000","last_peered":"2024-03-24T11:55:55.855116+0000","last_clean":"2024-03-24T11:55:55.855116+0000","last_became_active":"2024-03-24T11:52:21.851069+0000","last_became_peered":"2024-03-24T11:52:21.851069+0000","last_unstale":"2024-03-24T11:55:55.855116+0000","last_undegraded":"2024-03-24T11:55:55.855116+0000","last_fullsized":"2024-03-24T11:55:55.855116+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:48.767968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:48.767968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.848835+0000","last_change":"2024-03-24T11:54:51.499930+0000","last_active":"2024-03-24T11:55:55.848835+0000","last_peered":"2024-03-24T11:55:55.848835+0000","last_clean":"2024-03-24T11:55:55.848835+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:55:55.848835+0000","last_undegraded":"2024-03-24T11:55:55.848835+0000","last_fullsized":"2024-03-24T11:55:55.848835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:51.499817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:51.499817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:17.253146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":171,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845048+0000","last_change":"2024-03-24T11:53:39.923840+0000","last_active":"2024-03-24T11:55:55.845048+0000","last_peered":"2024-03-24T11:55:55.845048+0000","last_clean":"2024-03-24T11:55:55.845048+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:55:55.845048+0000","last_undegraded":"2024-03-24T11:55:55.845048+0000","last_fullsized":"2024-03-24T11:55:55.845048+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:53:39.923699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:39.923699+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":138,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851359+0000","last_change":"2024-03-24T11:55:15.058768+0000","last_active":"2024-03-24T11:55:55.851359+0000","last_peered":"2024-03-24T11:55:55.851359+0000","last_clean":"2024-03-24T11:55:55.851359+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:55:55.851359+0000","last_undegraded":"2024-03-24T11:55:55.851359+0000","last_fullsized":"2024-03-24T11:55:55.851359+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:15.058688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T11:55:15.058688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:33.054204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"127'6915","reported_seq":6984,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.223161+0000","last_change":"2024-03-24T11:54:16.238466+0000","last_active":"2024-03-24T11:55:56.223161+0000","last_peered":"2024-03-24T11:55:56.223161+0000","last_clean":"2024-03-24T11:55:56.223161+0000","last_became_active":"2024-03-24T11:54:16.238176+0000","last_became_peered":"2024-03-24T11:54:16.238176+0000","last_unstale":"2024-03-24T11:55:56.223161+0000","last_undegraded":"2024-03-24T11:55:56.223161+0000","last_fullsized":"2024-03-24T11:55:56.223161+0000","mapping_epoch":97,"log_start":"108'2200","ondisk_log_start":"108'2200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":4715,"log_dups_size":0,"ondisk_log_size":4715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453181440,"num_objects":6915,"num_object_clones":0,"num_object_copies":20745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6915,"num_write_kb":442560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":134,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853074+0000","last_change":"2024-03-24T11:54:55.993898+0000","last_active":"2024-03-24T11:55:55.853074+0000","last_peered":"2024-03-24T11:55:55.853074+0000","last_clean":"2024-03-24T11:55:55.853074+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:55:55.853074+0000","last_undegraded":"2024-03-24T11:55:55.853074+0000","last_fullsized":"2024-03-24T11:55:55.853074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:55.993770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:55.993770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:03.654855+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.860108+0000","last_change":"2024-03-24T11:55:02.950666+0000","last_active":"2024-03-24T11:55:55.860108+0000","last_peered":"2024-03-24T11:55:55.860108+0000","last_clean":"2024-03-24T11:55:55.860108+0000","last_became_active":"2024-03-24T11:52:15.142974+0000","last_became_peered":"2024-03-24T11:52:15.142974+0000","last_unstale":"2024-03-24T11:55:55.860108+0000","last_undegraded":"2024-03-24T11:55:55.860108+0000","last_fullsized":"2024-03-24T11:55:55.860108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:55:02.950528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:02.950528+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:22.293270+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":176,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858202+0000","last_change":"2024-03-24T11:54:57.356969+0000","last_active":"2024-03-24T11:55:55.858202+0000","last_peered":"2024-03-24T11:55:55.858202+0000","last_clean":"2024-03-24T11:55:55.858202+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:55:55.858202+0000","last_undegraded":"2024-03-24T11:55:55.858202+0000","last_fullsized":"2024-03-24T11:55:55.858202+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:54:57.356797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:57.356797+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:27.206630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":130,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.854763+0000","last_change":"2024-03-24T11:54:14.658407+0000","last_active":"2024-03-24T11:55:55.854763+0000","last_peered":"2024-03-24T11:55:55.854763+0000","last_clean":"2024-03-24T11:55:55.854763+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:55:55.854763+0000","last_undegraded":"2024-03-24T11:55:55.854763+0000","last_fullsized":"2024-03-24T11:55:55.854763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:14.658297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:14.658297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"127'6903","reported_seq":6972,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.251903+0000","last_change":"2024-03-24T11:54:16.236291+0000","last_active":"2024-03-24T11:55:56.251903+0000","last_peered":"2024-03-24T11:55:56.251903+0000","last_clean":"2024-03-24T11:55:56.251903+0000","last_became_active":"2024-03-24T11:54:16.235994+0000","last_became_peered":"2024-03-24T11:54:16.235994+0000","last_unstale":"2024-03-24T11:55:56.251903+0000","last_undegraded":"2024-03-24T11:55:56.251903+0000","last_fullsized":"2024-03-24T11:55:56.251903+0000","mapping_epoch":97,"log_start":"108'2200","ondisk_log_start":"108'2200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452395008,"num_objects":6903,"num_object_clones":0,"num_object_copies":20709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":441792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.848336+0000","last_change":"2024-03-24T11:55:06.364864+0000","last_active":"2024-03-24T11:55:55.848336+0000","last_peered":"2024-03-24T11:55:55.848336+0000","last_clean":"2024-03-24T11:55:55.848336+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:55:55.848336+0000","last_undegraded":"2024-03-24T11:55:55.848336+0000","last_fullsized":"2024-03-24T11:55:55.848336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.364777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.364777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:20.281583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":146,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851905+0000","last_change":"2024-03-24T11:54:49.113470+0000","last_active":"2024-03-24T11:55:55.851905+0000","last_peered":"2024-03-24T11:55:55.851905+0000","last_clean":"2024-03-24T11:55:55.851905+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:55:55.851905+0000","last_undegraded":"2024-03-24T11:55:55.851905+0000","last_fullsized":"2024-03-24T11:55:55.851905+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:49.113375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:54:49.113375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:06.207220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":170,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.854669+0000","last_change":"2024-03-24T11:55:05.889871+0000","last_active":"2024-03-24T11:55:55.854669+0000","last_peered":"2024-03-24T11:55:55.854669+0000","last_clean":"2024-03-24T11:55:55.854669+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:55:55.854669+0000","last_undegraded":"2024-03-24T11:55:55.854669+0000","last_fullsized":"2024-03-24T11:55:55.854669+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:55:05.889771+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:55:05.889771+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:29.514305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":185,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.858173+0000","last_change":"2024-03-24T11:54:13.589433+0000","last_active":"2024-03-24T11:55:55.858173+0000","last_peered":"2024-03-24T11:55:55.858173+0000","last_clean":"2024-03-24T11:55:55.858173+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:55:55.858173+0000","last_undegraded":"2024-03-24T11:55:55.858173+0000","last_fullsized":"2024-03-24T11:55:55.858173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:54:13.589346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:13.589346+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"127'7266","reported_seq":7336,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:56:01.123447+0000","last_change":"2024-03-24T11:54:16.242004+0000","last_active":"2024-03-24T11:56:01.123447+0000","last_peered":"2024-03-24T11:56:01.123447+0000","last_clean":"2024-03-24T11:56:01.123447+0000","last_became_active":"2024-03-24T11:54:16.241724+0000","last_became_peered":"2024-03-24T11:54:16.241724+0000","last_unstale":"2024-03-24T11:56:01.123447+0000","last_undegraded":"2024-03-24T11:56:01.123447+0000","last_fullsized":"2024-03-24T11:56:01.123447+0000","mapping_epoch":97,"log_start":"105'1100","ondisk_log_start":"105'1100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476184576,"num_objects":7266,"num_object_clones":0,"num_object_copies":21798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.862830+0000","last_change":"2024-03-24T11:55:55.660561+0000","last_active":"2024-03-24T11:55:55.862830+0000","last_peered":"2024-03-24T11:55:55.862830+0000","last_clean":"2024-03-24T11:55:55.862830+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:55:55.862830+0000","last_undegraded":"2024-03-24T11:55:55.862830+0000","last_fullsized":"2024-03-24T11:55:55.862830+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:55.660368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:55.660368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:14.298173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":138,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845622+0000","last_change":"2024-03-24T11:55:03.730546+0000","last_active":"2024-03-24T11:55:55.845622+0000","last_peered":"2024-03-24T11:55:55.845622+0000","last_clean":"2024-03-24T11:55:55.845622+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:55:55.845622+0000","last_undegraded":"2024-03-24T11:55:55.845622+0000","last_fullsized":"2024-03-24T11:55:55.845622+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:03.730436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:03.730436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:33.413314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.848014+0000","last_change":"2024-03-24T11:54:52.448522+0000","last_active":"2024-03-24T11:55:55.848014+0000","last_peered":"2024-03-24T11:55:55.848014+0000","last_clean":"2024-03-24T11:55:55.848014+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:55:55.848014+0000","last_undegraded":"2024-03-24T11:55:55.848014+0000","last_fullsized":"2024-03-24T11:55:55.848014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:52.448412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.448412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:59.084141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":153,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853735+0000","last_change":"2024-03-24T11:54:01.616111+0000","last_active":"2024-03-24T11:55:55.853735+0000","last_peered":"2024-03-24T11:55:55.853735+0000","last_clean":"2024-03-24T11:55:55.853735+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:55:55.853735+0000","last_undegraded":"2024-03-24T11:55:55.853735+0000","last_fullsized":"2024-03-24T11:55:55.853735+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:54:01.616018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:01.616018+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"127'7293","reported_seq":7363,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:56:01.134284+0000","last_change":"2024-03-24T11:54:16.237411+0000","last_active":"2024-03-24T11:56:01.134284+0000","last_peered":"2024-03-24T11:56:01.134284+0000","last_clean":"2024-03-24T11:56:01.134284+0000","last_became_active":"2024-03-24T11:54:16.237102+0000","last_became_peered":"2024-03-24T11:54:16.237102+0000","last_unstale":"2024-03-24T11:56:01.134284+0000","last_undegraded":"2024-03-24T11:56:01.134284+0000","last_fullsized":"2024-03-24T11:56:01.134284+0000","mapping_epoch":97,"log_start":"105'1100","ondisk_log_start":"105'1100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":6193,"log_dups_size":0,"ondisk_log_size":6193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477954048,"num_objects":7293,"num_object_clones":0,"num_object_copies":21879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7293,"num_write_kb":466752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":134,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845167+0000","last_change":"2024-03-24T11:55:11.673583+0000","last_active":"2024-03-24T11:55:55.845167+0000","last_peered":"2024-03-24T11:55:55.845167+0000","last_clean":"2024-03-24T11:55:55.845167+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:55:55.845167+0000","last_undegraded":"2024-03-24T11:55:55.845167+0000","last_fullsized":"2024-03-24T11:55:55.845167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.673492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.673492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:16.257227+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":195,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853527+0000","last_change":"2024-03-24T11:55:02.895074+0000","last_active":"2024-03-24T11:55:55.853527+0000","last_peered":"2024-03-24T11:55:55.853527+0000","last_clean":"2024-03-24T11:55:55.853527+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:55:55.853527+0000","last_undegraded":"2024-03-24T11:55:55.853527+0000","last_fullsized":"2024-03-24T11:55:55.853527+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:02.894942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:02.894942+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:25.378002+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856304+0000","last_change":"2024-03-24T11:53:45.930437+0000","last_active":"2024-03-24T11:55:55.856304+0000","last_peered":"2024-03-24T11:55:55.856304+0000","last_clean":"2024-03-24T11:55:55.856304+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:55:55.856304+0000","last_undegraded":"2024-03-24T11:55:55.856304+0000","last_fullsized":"2024-03-24T11:55:55.856304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:45.930342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:45.930342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":168,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856118+0000","last_change":"2024-03-24T11:54:59.949653+0000","last_active":"2024-03-24T11:55:55.856118+0000","last_peered":"2024-03-24T11:55:55.856118+0000","last_clean":"2024-03-24T11:55:55.856118+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:55:55.856118+0000","last_undegraded":"2024-03-24T11:55:55.856118+0000","last_fullsized":"2024-03-24T11:55:55.856118+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:54:59.949501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:59.949501+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:16.451264+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"127'6961","reported_seq":7031,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.280876+0000","last_change":"2024-03-24T11:54:16.239589+0000","last_active":"2024-03-24T11:55:56.280876+0000","last_peered":"2024-03-24T11:55:56.280876+0000","last_clean":"2024-03-24T11:55:56.280876+0000","last_became_active":"2024-03-24T11:54:16.239274+0000","last_became_peered":"2024-03-24T11:54:16.239274+0000","last_unstale":"2024-03-24T11:55:56.280876+0000","last_undegraded":"2024-03-24T11:55:56.280876+0000","last_fullsized":"2024-03-24T11:55:56.280876+0000","mapping_epoch":97,"log_start":"106'1500","ondisk_log_start":"106'1500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.089044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456196096,"num_objects":6961,"num_object_clones":0,"num_object_copies":20883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":445504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853377+0000","last_change":"2024-03-24T11:55:01.911844+0000","last_active":"2024-03-24T11:55:55.853377+0000","last_peered":"2024-03-24T11:55:55.853377+0000","last_clean":"2024-03-24T11:55:55.853377+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:55:55.853377+0000","last_undegraded":"2024-03-24T11:55:55.853377+0000","last_fullsized":"2024-03-24T11:55:55.853377+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:55:01.911663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.911663+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:03.109694+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":193,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.855558+0000","last_change":"2024-03-24T11:54:58.454553+0000","last_active":"2024-03-24T11:55:55.855558+0000","last_peered":"2024-03-24T11:55:55.855558+0000","last_clean":"2024-03-24T11:55:55.855558+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:55:55.855558+0000","last_undegraded":"2024-03-24T11:55:55.855558+0000","last_fullsized":"2024-03-24T11:55:55.855558+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:54:58.454433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.454433+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:05.743552+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.856389+0000","last_change":"2024-03-24T11:53:51.949166+0000","last_active":"2024-03-24T11:55:55.856389+0000","last_peered":"2024-03-24T11:55:55.856389+0000","last_clean":"2024-03-24T11:55:55.856389+0000","last_became_active":"2024-03-24T11:52:15.134326+0000","last_became_peered":"2024-03-24T11:52:15.134326+0000","last_unstale":"2024-03-24T11:55:55.856389+0000","last_undegraded":"2024-03-24T11:55:55.856389+0000","last_fullsized":"2024-03-24T11:55:55.856389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_clean_scrub_stamp":"2024-03-24T11:53:51.949009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":157,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853159+0000","last_change":"2024-03-24T11:55:18.793369+0000","last_active":"2024-03-24T11:55:55.853159+0000","last_peered":"2024-03-24T11:55:55.853159+0000","last_clean":"2024-03-24T11:55:55.853159+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:55:55.853159+0000","last_undegraded":"2024-03-24T11:55:55.853159+0000","last_fullsized":"2024-03-24T11:55:55.853159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:55:18.793281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.793281+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.906014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"127'7034","reported_seq":7104,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.185335+0000","last_change":"2024-03-24T11:54:16.237184+0000","last_active":"2024-03-24T11:55:56.185335+0000","last_peered":"2024-03-24T11:55:56.185335+0000","last_clean":"2024-03-24T11:55:56.185335+0000","last_became_active":"2024-03-24T11:54:16.236885+0000","last_became_peered":"2024-03-24T11:54:16.236885+0000","last_unstale":"2024-03-24T11:55:56.185335+0000","last_undegraded":"2024-03-24T11:55:56.185335+0000","last_fullsized":"2024-03-24T11:55:56.185335+0000","mapping_epoch":97,"log_start":"105'500","ondisk_log_start":"105'500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.586184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460980224,"num_objects":7034,"num_object_clones":0,"num_object_copies":21102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7034,"num_write_kb":450176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853562+0000","last_change":"2024-03-24T11:55:20.629992+0000","last_active":"2024-03-24T11:55:55.853562+0000","last_peered":"2024-03-24T11:55:55.853562+0000","last_clean":"2024-03-24T11:55:55.853562+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:55:55.853562+0000","last_undegraded":"2024-03-24T11:55:55.853562+0000","last_fullsized":"2024-03-24T11:55:55.853562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.629800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.629800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:31.421523+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":131,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.857650+0000","last_change":"2024-03-24T11:53:58.860517+0000","last_active":"2024-03-24T11:55:55.857650+0000","last_peered":"2024-03-24T11:55:55.857650+0000","last_clean":"2024-03-24T11:55:55.857650+0000","last_became_active":"2024-03-24T11:52:19.387115+0000","last_became_peered":"2024-03-24T11:52:19.387115+0000","last_unstale":"2024-03-24T11:55:55.857650+0000","last_undegraded":"2024-03-24T11:55:55.857650+0000","last_fullsized":"2024-03-24T11:55:55.857650+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:58.860428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:53:58.860428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":130,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853881+0000","last_change":"2024-03-24T11:54:59.405029+0000","last_active":"2024-03-24T11:55:55.853881+0000","last_peered":"2024-03-24T11:55:55.853881+0000","last_clean":"2024-03-24T11:55:55.853881+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:55:55.853881+0000","last_undegraded":"2024-03-24T11:55:55.853881+0000","last_fullsized":"2024-03-24T11:55:55.853881+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:54:59.404740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:59.404740+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.460597+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"116'25","reported_seq":178,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853416+0000","last_change":"2024-03-24T11:55:19.864480+0000","last_active":"2024-03-24T11:55:55.853416+0000","last_peered":"2024-03-24T11:55:55.853416+0000","last_clean":"2024-03-24T11:55:55.853416+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:55:55.853416+0000","last_undegraded":"2024-03-24T11:55:55.853416+0000","last_fullsized":"2024-03-24T11:55:55.853416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'25","last_scrub_stamp":"2024-03-24T11:55:19.864301+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T11:55:19.864301+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.864301+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:45.201869+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"127'7057","reported_seq":7126,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.326119+0000","last_change":"2024-03-24T11:54:16.237670+0000","last_active":"2024-03-24T11:55:56.326119+0000","last_peered":"2024-03-24T11:55:56.326119+0000","last_clean":"2024-03-24T11:55:56.326119+0000","last_became_active":"2024-03-24T11:54:16.237374+0000","last_became_peered":"2024-03-24T11:54:16.237374+0000","last_unstale":"2024-03-24T11:55:56.326119+0000","last_undegraded":"2024-03-24T11:55:56.326119+0000","last_fullsized":"2024-03-24T11:55:56.326119+0000","mapping_epoch":97,"log_start":"107'1600","ondisk_log_start":"107'1600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462487552,"num_objects":7057,"num_object_clones":0,"num_object_copies":21171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7057,"num_write_kb":451648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.848096+0000","last_change":"2024-03-24T11:55:18.299756+0000","last_active":"2024-03-24T11:55:55.848096+0000","last_peered":"2024-03-24T11:55:55.848096+0000","last_clean":"2024-03-24T11:55:55.848096+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:55:55.848096+0000","last_undegraded":"2024-03-24T11:55:55.848096+0000","last_fullsized":"2024-03-24T11:55:55.848096+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.299637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:40.446628+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.854108+0000","last_change":"2024-03-24T11:55:07.488338+0000","last_active":"2024-03-24T11:55:55.854108+0000","last_peered":"2024-03-24T11:55:55.854108+0000","last_clean":"2024-03-24T11:55:55.854108+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:55:55.854108+0000","last_undegraded":"2024-03-24T11:55:55.854108+0000","last_fullsized":"2024-03-24T11:55:55.854108+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:07.488042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:07.488042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:22.154511+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.853214+0000","last_change":"2024-03-24T11:54:51.317621+0000","last_active":"2024-03-24T11:55:55.853214+0000","last_peered":"2024-03-24T11:55:55.853214+0000","last_clean":"2024-03-24T11:55:55.853214+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:55:55.853214+0000","last_undegraded":"2024-03-24T11:55:55.853214+0000","last_fullsized":"2024-03-24T11:55:55.853214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:51.317368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:51.317368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.404690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":173,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.851242+0000","last_change":"2024-03-24T11:55:19.105549+0000","last_active":"2024-03-24T11:55:55.851242+0000","last_peered":"2024-03-24T11:55:55.851242+0000","last_clean":"2024-03-24T11:55:55.851242+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:55:55.851242+0000","last_undegraded":"2024-03-24T11:55:55.851242+0000","last_fullsized":"2024-03-24T11:55:55.851242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:55:19.105397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.105397+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:27.881078+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"127'7021","reported_seq":7091,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.188366+0000","last_change":"2024-03-24T11:54:16.237786+0000","last_active":"2024-03-24T11:55:56.188366+0000","last_peered":"2024-03-24T11:55:56.188366+0000","last_clean":"2024-03-24T11:55:56.188366+0000","last_became_active":"2024-03-24T11:54:16.237483+0000","last_became_peered":"2024-03-24T11:54:16.237483+0000","last_unstale":"2024-03-24T11:55:56.188366+0000","last_undegraded":"2024-03-24T11:55:56.188366+0000","last_fullsized":"2024-03-24T11:55:56.188366+0000","mapping_epoch":97,"log_start":"105'400","ondisk_log_start":"105'400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":6621,"log_dups_size":0,"ondisk_log_size":6621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.586142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460128256,"num_objects":7021,"num_object_clones":0,"num_object_copies":21063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7021,"num_write_kb":449344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":127,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.852976+0000","last_change":"2024-03-24T11:55:01.085505+0000","last_active":"2024-03-24T11:55:55.852976+0000","last_peered":"2024-03-24T11:55:55.852976+0000","last_clean":"2024-03-24T11:55:55.852976+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:55:55.852976+0000","last_undegraded":"2024-03-24T11:55:55.852976+0000","last_fullsized":"2024-03-24T11:55:55.852976+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.085396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.085396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:10.637560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.854146+0000","last_change":"2024-03-24T11:55:22.642429+0000","last_active":"2024-03-24T11:55:55.854146+0000","last_peered":"2024-03-24T11:55:55.854146+0000","last_clean":"2024-03-24T11:55:55.854146+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:55:55.854146+0000","last_undegraded":"2024-03-24T11:55:55.854146+0000","last_fullsized":"2024-03-24T11:55:55.854146+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:22.642148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.642148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:24.428410+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.857068+0000","last_change":"2024-03-24T11:55:06.933106+0000","last_active":"2024-03-24T11:55:55.857068+0000","last_peered":"2024-03-24T11:55:55.857068+0000","last_clean":"2024-03-24T11:55:55.857068+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:55:55.857068+0000","last_undegraded":"2024-03-24T11:55:55.857068+0000","last_fullsized":"2024-03-24T11:55:55.857068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.932966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.932966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:17.361036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.845225+0000","last_change":"2024-03-24T11:53:36.884179+0000","last_active":"2024-03-24T11:55:55.845225+0000","last_peered":"2024-03-24T11:55:55.845225+0000","last_clean":"2024-03-24T11:55:55.845225+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:55:55.845225+0000","last_undegraded":"2024-03-24T11:55:55.845225+0000","last_fullsized":"2024-03-24T11:55:55.845225+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:53:36.884086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:53:36.884086+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:58.082564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"127'6969","reported_seq":7039,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:56.204692+0000","last_change":"2024-03-24T11:54:16.239807+0000","last_active":"2024-03-24T11:55:56.204692+0000","last_peered":"2024-03-24T11:55:56.204692+0000","last_clean":"2024-03-24T11:55:56.204692+0000","last_became_active":"2024-03-24T11:54:16.239338+0000","last_became_peered":"2024-03-24T11:54:16.239338+0000","last_unstale":"2024-03-24T11:55:56.204692+0000","last_undegraded":"2024-03-24T11:55:56.204692+0000","last_fullsized":"2024-03-24T11:55:56.204692+0000","mapping_epoch":97,"log_start":"108'1800","ondisk_log_start":"108'1800","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456720384,"num_objects":6969,"num_object_clones":0,"num_object_copies":20907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6969,"num_write_kb":446016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":134,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.844583+0000","last_change":"2024-03-24T11:55:17.618704+0000","last_active":"2024-03-24T11:55:55.844583+0000","last_peered":"2024-03-24T11:55:55.844583+0000","last_clean":"2024-03-24T11:55:55.844583+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:55:55.844583+0000","last_undegraded":"2024-03-24T11:55:55.844583+0000","last_fullsized":"2024-03-24T11:55:55.844583+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:17.618606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T11:55:17.618606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:42.294065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.855228+0000","last_change":"2024-03-24T11:55:04.412885+0000","last_active":"2024-03-24T11:55:55.855228+0000","last_peered":"2024-03-24T11:55:55.855228+0000","last_clean":"2024-03-24T11:55:55.855228+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:55:55.855228+0000","last_undegraded":"2024-03-24T11:55:55.855228+0000","last_fullsized":"2024-03-24T11:55:55.855228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:04.412611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:04.412611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:19.594594+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":167,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.844393+0000","last_change":"2024-03-24T11:55:19.645322+0000","last_active":"2024-03-24T11:55:55.844393+0000","last_peered":"2024-03-24T11:55:55.844393+0000","last_clean":"2024-03-24T11:55:55.844393+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:55:55.844393+0000","last_undegraded":"2024-03-24T11:55:55.844393+0000","last_fullsized":"2024-03-24T11:55:55.844393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:55:19.645176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.645176+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:39.822493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":193,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-24T11:55:55.848069+0000","last_change":"2024-03-24T11:55:02.415044+0000","last_active":"2024-03-24T11:55:55.848069+0000","last_peered":"2024-03-24T11:55:55.848069+0000","last_clean":"2024-03-24T11:55:55.848069+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:55:55.848069+0000","last_undegraded":"2024-03-24T11:55:55.848069+0000","last_fullsized":"2024-03-24T11:55:55.848069+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:02.414931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:02.414931+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:30.803168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7470252032,"num_objects":113987,"num_object_clones":0,"num_object_copies":341961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113987,"num_write_kb":7295168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22433038336,"data_stored":22433038336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88987,"ondisk_log_size":88987,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":215,"num_read_kb":210,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":444,"ondisk_log_size":444,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005105,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2402812,"kb_used_data":2327220,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91342340,"statfs":{"total":95995035648,"available":93534556160,"internally_reserved":0,"allocated":2383073280,"data_stored":2382685545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332152,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2417016,"kb_used_data":2341424,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91328136,"statfs":{"total":95995035648,"available":93520011264,"internally_reserved":0,"allocated":2397618176,"data_stored":2397234355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2750856,"kb_used_data":2687736,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":90994296,"statfs":{"total":95995035648,"available":93178159104,"internally_reserved":0,"allocated":2752241664,"data_stored":2751848686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3671552,"kb_used_data":3586228,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":90073600,"statfs":{"total":95995035648,"available":92235366400,"internally_reserved":0,"allocated":3672297472,"data_stored":3671908604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2740788,"kb_used_data":2677672,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91004364,"statfs":{"total":95995035648,"available":93188468736,"internally_reserved":0,"allocated":2741936128,"data_stored":2741553051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117061,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4121160,"kb_used_data":4013628,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":89623992,"statfs":{"total":95995035648,"available":91774967808,"internally_reserved":0,"allocated":4109955072,"data_stored":4109551945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2399172,"kb_used_data":2336060,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91345980,"statfs":{"total":95995035648,"available":93538283520,"internally_reserved":0,"allocated":2392125440,"data_stored":2391728401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659196,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1949628,"kb_used_data":1882356,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91795524,"statfs":{"total":95995035648,"available":93998616576,"internally_reserved":0,"allocated":1927532544,"data_stored":1927143809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3676569600,"data_stored":3676569600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2754150400,"data_stored":2754150400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2757427200,"data_stored":2757427200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4132831232,"data_stored":4132831232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2394750976,"data_stored":2394750976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1934229504,"data_stored":1934229504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2399862784,"data_stored":2399862784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2383216640,"data_stored":2383216640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:56:04.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:56:04.185 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:04.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:04.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:04.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:04.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:03 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T11:56:04.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:03 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T11:56:04.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:03 smithi193 ceph-mon[33593]: 7.8 scrub starts 2024-03-24T11:56:04.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:03 smithi193 ceph-mon[33593]: 6.b scrub starts 2024-03-24T11:56:04.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:03 smithi193 ceph-mon[33593]: 6.b scrub ok 2024-03-24T11:56:04.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:03 smithi193 ceph-mon[33593]: 5.1b scrub starts 2024-03-24T11:56:04.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:03 smithi193 ceph-mon[33593]: 5.1b scrub ok 2024-03-24T11:56:04.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:04.499 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:04.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:04.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:04.626 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:04.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:04.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:04.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 89 16 120155 120139 84.3347 47.5 0.0346489 0.011852 2024-03-24T11:56:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:04.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:04.874 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:04.903 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:04.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:05.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:05.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[30223]: pgmap v143: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T11:56:05.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T11:56:05.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T11:56:05.032 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[30223]: 7.e scrub starts 2024-03-24T11:56:05.032 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[30223]: 7.2 scrub starts 2024-03-24T11:56:05.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[42243]: pgmap v143: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T11:56:05.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T11:56:05.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T11:56:05.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[42243]: 7.e scrub starts 2024-03-24T11:56:05.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:04 smithi064 ceph-mon[42243]: 7.2 scrub starts 2024-03-24T11:56:05.051 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:05.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:05.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T11:56:05.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:05.216 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:05.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:05.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:04 smithi193 ceph-mon[33593]: pgmap v143: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T11:56:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:04 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T11:56:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:04 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T11:56:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:04 smithi193 ceph-mon[33593]: 7.e scrub starts 2024-03-24T11:56:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:04 smithi193 ceph-mon[33593]: 7.2 scrub starts 2024-03-24T11:56:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:05.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:05.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:05.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:05.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:05.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:05.692 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:05.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:05.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total time run: 90.0152 2024-03-24T11:56:05.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total writes made: 120954 2024-03-24T11:56:05.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Write size: 65536 2024-03-24T11:56:05.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object size: 65536 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Bandwidth (MB/sec): 83.9816 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Bandwidth: 31.6909 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max bandwidth (MB/sec): 210.188 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min bandwidth (MB/sec): 47.5 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average IOPS: 1343 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev IOPS: 507.054 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max IOPS: 3363 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min IOPS: 760 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average Latency(s): 0.0119024 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Latency(s): 0.00926222 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max latency(s): 0.234923 2024-03-24T11:56:05.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min latency(s): 0.00141856 2024-03-24T11:56:05.739 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T11:56:05.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:05.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:05.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:05.970 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:05.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:06.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:06.096 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:06.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:06.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T11:56:06.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T11:56:06.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T11:56:06.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T11:56:06.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[30223]: 7.8 scrub ok 2024-03-24T11:56:06.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[30223]: 5.16 scrub starts 2024-03-24T11:56:06.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[30223]: 5.16 scrub ok 2024-03-24T11:56:06.125 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T11:56:06.125 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T11:56:06.125 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T11:56:06.125 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T11:56:06.125 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[42243]: 7.8 scrub ok 2024-03-24T11:56:06.125 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[42243]: 5.16 scrub starts 2024-03-24T11:56:06.125 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:05 smithi064 ceph-mon[42243]: 5.16 scrub ok 2024-03-24T11:56:06.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:06.233 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:06.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:06.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:05 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T11:56:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:05 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T11:56:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:05 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T11:56:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:05 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T11:56:06.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:05 smithi193 ceph-mon[33593]: 7.8 scrub ok 2024-03-24T11:56:06.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:05 smithi193 ceph-mon[33593]: 5.16 scrub starts 2024-03-24T11:56:06.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:05 smithi193 ceph-mon[33593]: 5.16 scrub ok 2024-03-24T11:56:06.363 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:06.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:06.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:06.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:06.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:06.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:06.623 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:06.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:06.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:06.754 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:06.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:06.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:06.894 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:06.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:06.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:07.042 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:07.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:07.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:07.193 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:07.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:07.271 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:07.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: pgmap v144: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: 4.3 scrub starts 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: 4.3 scrub ok 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: 4.11 scrub starts 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: 4.11 scrub ok 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: 4.e scrub starts 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[30223]: 4.e scrub ok 2024-03-24T11:56:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: pgmap v144: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:56:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: 4.3 scrub starts 2024-03-24T11:56:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: 4.3 scrub ok 2024-03-24T11:56:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: 4.11 scrub starts 2024-03-24T11:56:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: 4.11 scrub ok 2024-03-24T11:56:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T11:56:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T11:56:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: 4.e scrub starts 2024-03-24T11:56:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:06 smithi064 ceph-mon[42243]: 4.e scrub ok 2024-03-24T11:56:07.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: pgmap v144: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: 4.3 scrub starts 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: 4.3 scrub ok 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: 4.11 scrub starts 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: 4.11 scrub ok 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: 4.e scrub starts 2024-03-24T11:56:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:06 smithi193 ceph-mon[33593]: 4.e scrub ok 2024-03-24T11:56:07.344 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:07.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:07.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:07.500 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:07.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:07.602 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:07.633 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:56:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:07.635 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:07.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:07.737 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:07.768 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:07.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:07.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:07.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:08.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: 5.c scrub starts 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: 5.c scrub ok 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: 7.e scrub ok 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T11:56:08.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[30223]: 7.1 scrub starts 2024-03-24T11:56:08.055 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:08.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:08.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T11:56:08.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T11:56:08.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: 5.c scrub starts 2024-03-24T11:56:08.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: 5.c scrub ok 2024-03-24T11:56:08.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: 7.e scrub ok 2024-03-24T11:56:08.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:08.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T11:56:08.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T11:56:08.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:07 smithi064 ceph-mon[42243]: 7.1 scrub starts 2024-03-24T11:56:08.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:08.195 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:08.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:08.296 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:08.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:56:08.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:56:08.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:56:08.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:56:08.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:56:08.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:56:08.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:56:08.332 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:56:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T11:56:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T11:56:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: 5.c scrub starts 2024-03-24T11:56:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: 5.c scrub ok 2024-03-24T11:56:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: 7.e scrub ok 2024-03-24T11:56:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T11:56:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T11:56:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:07 smithi193 ceph-mon[33593]: 7.1 scrub starts 2024-03-24T11:56:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:56:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:56:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:56:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:56:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:56:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:56:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:56:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:56:08.359 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:08.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:08.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:08.504 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:08.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:08.609 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:08.641 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:08.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:08.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:08.780 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:08.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:08.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:08.911 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:08.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:08.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:09.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:09.059 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:09.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:09.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:09.197 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:09.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[30223]: 7.2 scrub ok 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[30223]: pgmap v145: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.28k op/s 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[30223]: 7.7 scrub starts 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/403982007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[30223]: 5.7 scrub starts 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[30223]: 5.7 scrub ok 2024-03-24T11:56:09.208 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[42243]: 7.2 scrub ok 2024-03-24T11:56:09.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[42243]: pgmap v145: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.28k op/s 2024-03-24T11:56:09.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[42243]: 7.7 scrub starts 2024-03-24T11:56:09.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/403982007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:09.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T11:56:09.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T11:56:09.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[42243]: 5.7 scrub starts 2024-03-24T11:56:09.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:08 smithi064 ceph-mon[42243]: 5.7 scrub ok 2024-03-24T11:56:09.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:08 smithi193 ceph-mon[33593]: 7.2 scrub ok 2024-03-24T11:56:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:08 smithi193 ceph-mon[33593]: pgmap v145: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.28k op/s 2024-03-24T11:56:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:08 smithi193 ceph-mon[33593]: 7.7 scrub starts 2024-03-24T11:56:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:08 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/403982007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:08 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T11:56:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:08 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T11:56:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:08 smithi193 ceph-mon[33593]: 5.7 scrub starts 2024-03-24T11:56:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:08 smithi193 ceph-mon[33593]: 5.7 scrub ok 2024-03-24T11:56:09.340 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:09.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:09.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:09.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:09.542 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:09.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:09.627 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:09.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:09.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:09.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:09.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:09.926 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:09.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:10.029 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:10.058 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:10.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:10.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:10.189 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:10.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:10.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:09 smithi064 ceph-mon[30223]: 7.1 scrub ok 2024-03-24T11:56:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:09 smithi064 ceph-mon[30223]: 7.b scrub starts 2024-03-24T11:56:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:09 smithi064 ceph-mon[42243]: 7.1 scrub ok 2024-03-24T11:56:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:09 smithi064 ceph-mon[42243]: 7.b scrub starts 2024-03-24T11:56:10.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:10.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:09 smithi193 ceph-mon[33593]: 7.1 scrub ok 2024-03-24T11:56:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:09 smithi193 ceph-mon[33593]: 7.b scrub starts 2024-03-24T11:56:10.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:10.471 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:10.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:10.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:10.619 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:10.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:10.720 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:10.755 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:10.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:10.857 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:10.890 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:10.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T11:56:10.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:11.023 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:11.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:11.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:11.186 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:11.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:11.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:56:11.289 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:11.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: pgmap v146: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 46 MiB/s wr, 773 op/s 2024-03-24T11:56:11.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1848076070' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: 5.1 scrub starts 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: 5.1 scrub ok 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: 7.7 scrub ok 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[30223]: 7.3 scrub starts 2024-03-24T11:56:11.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: pgmap v146: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 46 MiB/s wr, 773 op/s 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1848076070' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: 5.1 scrub starts 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: 5.1 scrub ok 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: 7.7 scrub ok 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:11.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:10 smithi064 ceph-mon[42243]: 7.3 scrub starts 2024-03-24T11:56:11.316 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:11.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: pgmap v146: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 46 MiB/s wr, 773 op/s 2024-03-24T11:56:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:56:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:56:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:56:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1848076070' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:11.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:11.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: 5.1 scrub starts 2024-03-24T11:56:11.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: 5.1 scrub ok 2024-03-24T11:56:11.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: 7.7 scrub ok 2024-03-24T11:56:11.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T11:56:11.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T11:56:11.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:11.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:11.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:10 smithi193 ceph-mon[33593]: 7.3 scrub starts 2024-03-24T11:56:11.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:11.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:11.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:11.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:11.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:11.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:11.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:11.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:11.902 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:11.941 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:11.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:12.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:12.092 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:12.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:12.192 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:12.240 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:12.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T11:56:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T11:56:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: 7.5 scrub starts 2024-03-24T11:56:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[30223]: 7.b scrub ok 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: 7.5 scrub starts 2024-03-24T11:56:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:56:12.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T11:56:12.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:56:12.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:56:12.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T11:56:12.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T11:56:12.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:56:12.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:56:12.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:11 smithi064 ceph-mon[42243]: 7.b scrub ok 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: 7.5 scrub starts 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:56:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:56:12.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:11 smithi193 ceph-mon[33593]: 7.b scrub ok 2024-03-24T11:56:12.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:12.394 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:12.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:12.495 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:12.530 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:12.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:12.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:12.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:12.667 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:12.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:12.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:12.804 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:12.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:12.906 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:12.937 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:12.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:13.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:13.067 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:13.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:13.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:13.211 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:13.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:12 smithi064 ceph-mon[30223]: pgmap v149: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 787 op/s 2024-03-24T11:56:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:12 smithi064 ceph-mon[30223]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:56:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:12 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T11:56:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:12 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T11:56:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:12 smithi064 ceph-mon[42243]: pgmap v149: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 787 op/s 2024-03-24T11:56:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:12 smithi064 ceph-mon[42243]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:56:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:12 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T11:56:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:12 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T11:56:13.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:12 smithi193 ceph-mon[33593]: pgmap v149: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 787 op/s 2024-03-24T11:56:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:12 smithi193 ceph-mon[33593]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:56:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:12 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T11:56:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:12 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T11:56:13.355 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:13.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:13.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:13.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:13.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:13.642 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:13.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:13.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:13.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:13.886 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:13.910 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:13.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:14.012 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:14.041 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:14.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:14.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[30223]: 7.5 scrub ok 2024-03-24T11:56:14.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[30223]: 7.c scrub starts 2024-03-24T11:56:14.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[30223]: 7.3 scrub ok 2024-03-24T11:56:14.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T11:56:14.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T11:56:14.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[42243]: 7.5 scrub ok 2024-03-24T11:56:14.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[42243]: 7.c scrub starts 2024-03-24T11:56:14.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[42243]: 7.3 scrub ok 2024-03-24T11:56:14.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T11:56:14.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:13 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T11:56:14.143 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:14.175 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:14.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:14.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:14.313 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:14.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:13 smithi193 ceph-mon[33593]: 7.5 scrub ok 2024-03-24T11:56:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:13 smithi193 ceph-mon[33593]: 7.c scrub starts 2024-03-24T11:56:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:13 smithi193 ceph-mon[33593]: 7.3 scrub ok 2024-03-24T11:56:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:13 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T11:56:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:13 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T11:56:14.414 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:14.444 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:14.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:14.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:14.578 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:14.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:14.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:14.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:14.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:14.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:14.822 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:14.853 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:14.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:14.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:14.988 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:14 smithi193 ceph-mon[33593]: pgmap v151: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 821 op/s 2024-03-24T11:56:14.991 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:14.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:15.074 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[30223]: pgmap v151: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 821 op/s 2024-03-24T11:56:15.074 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T11:56:15.074 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T11:56:15.075 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[30223]: 2.19 scrub starts 2024-03-24T11:56:15.075 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[30223]: 2.19 scrub ok 2024-03-24T11:56:15.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[42243]: pgmap v151: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 821 op/s 2024-03-24T11:56:15.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T11:56:15.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T11:56:15.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[42243]: 2.19 scrub starts 2024-03-24T11:56:15.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:14 smithi064 ceph-mon[42243]: 2.19 scrub ok 2024-03-24T11:56:15.093 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:15.121 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:15.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:15.209 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:15.223 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:15.254 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:15.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:15.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:15.290 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:56:15.290 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-03-24T11:56:15.290 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-03-24T11:56:15.290 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-03-24T11:56:15.290 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:56:15.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:14 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T11:56:15.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:14 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T11:56:15.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:14 smithi193 ceph-mon[33593]: 2.19 scrub starts 2024-03-24T11:56:15.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:14 smithi193 ceph-mon[33593]: 2.19 scrub ok 2024-03-24T11:56:15.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:15.405 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:15.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:15.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:15.551 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:15.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:15.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:15.653 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:15.681 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:15.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:15.783 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:15.814 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:15.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:15.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:15.943 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:15.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:16.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:16.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:16.083 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:16.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:16.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:16.240 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:16.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:16.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T11:56:16.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 2.c deep-scrub starts 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 2.c deep-scrub ok 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[30223]: 7.c scrub ok 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 2.c deep-scrub starts 2024-03-24T11:56:16.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 2.c deep-scrub ok 2024-03-24T11:56:16.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T11:56:16.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T11:56:16.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T11:56:16.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T11:56:16.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:16 smithi064 ceph-mon[42243]: 7.c scrub ok 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 2.c deep-scrub starts 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 2.c deep-scrub ok 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T11:56:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-mon[33593]: 7.c scrub ok 2024-03-24T11:56:16.343 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:16.374 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:16.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:16.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:16.516 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:16.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:16.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:56:16.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:16.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T11:56:16.681 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:56:16.696 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:56:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:16.750 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:16.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:16.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:16.904 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:16.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:17.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:17.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:56:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:56:17.054 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:17.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:17.156 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:17.189 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:17.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:17.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[30223]: pgmap v152: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 821 op/s 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[30223]: 4.1e scrub starts 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[30223]: 4.1e scrub ok 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[30223]: 7.a deep-scrub starts 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[42243]: pgmap v152: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 821 op/s 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[42243]: 4.1e scrub starts 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[42243]: 4.1e scrub ok 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T11:56:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T11:56:17.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-mon[42243]: 7.a deep-scrub starts 2024-03-24T11:56:17.334 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:17.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:17 smithi193 ceph-mon[33593]: pgmap v152: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 821 op/s 2024-03-24T11:56:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:17 smithi193 ceph-mon[33593]: 4.1e scrub starts 2024-03-24T11:56:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:17 smithi193 ceph-mon[33593]: 4.1e scrub ok 2024-03-24T11:56:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:17 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T11:56:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:17 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T11:56:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:17 smithi193 ceph-mon[33593]: 7.a deep-scrub starts 2024-03-24T11:56:17.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:17.449 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-03-24T11:56:15.189994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7573477750,"num_objects":115752,"num_object_clones":0,"num_object_copies":347256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115752,"num_whiteouts":0,"num_read":1158,"num_read_kb":1004,"num_write":126971,"num_write_kb":7743518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89709,"ondisk_log_size":89709,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22913696,"kb_used_data":22277760,"kb_used_omap":12,"kb_used_meta":635251,"kb_avail":727047520,"statfs":{"total":767960285184,"available":744496660480,"internally_reserved":0,"allocated":22812426240,"data_stored":22809223924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":650497584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,9,13],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-322764800,"num_objects":-4925,"num_object_clones":0,"num_object_copies":-14775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4925,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4925,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002157"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978423+0000","last_change":"2024-03-24T11:55:59.687133+0000","last_active":"2024-03-24T11:56:10.978423+0000","last_peered":"2024-03-24T11:56:10.978423+0000","last_clean":"2024-03-24T11:56:10.978423+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:56:10.978423+0000","last_undegraded":"2024-03-24T11:56:10.978423+0000","last_fullsized":"2024-03-24T11:56:10.978423+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:59.686983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:59.686983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.251542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":142,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990273+0000","last_change":"2024-03-24T11:54:50.876045+0000","last_active":"2024-03-24T11:56:10.990273+0000","last_peered":"2024-03-24T11:56:10.990273+0000","last_clean":"2024-03-24T11:56:10.990273+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:56:10.990273+0000","last_undegraded":"2024-03-24T11:56:10.990273+0000","last_fullsized":"2024-03-24T11:56:10.990273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:50.875918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:54:50.875918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:19.746580+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980612+0000","last_change":"2024-03-24T11:54:13.817576+0000","last_active":"2024-03-24T11:56:10.980612+0000","last_peered":"2024-03-24T11:56:10.980612+0000","last_clean":"2024-03-24T11:56:10.980612+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:56:10.980612+0000","last_undegraded":"2024-03-24T11:56:10.980612+0000","last_fullsized":"2024-03-24T11:56:10.980612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T11:54:13.817437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"117'22","reported_seq":180,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978886+0000","last_change":"2024-03-24T11:55:21.286888+0000","last_active":"2024-03-24T11:56:10.978886+0000","last_peered":"2024-03-24T11:56:10.978886+0000","last_clean":"2024-03-24T11:56:10.978886+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:56:10.978886+0000","last_undegraded":"2024-03-24T11:56:10.978886+0000","last_fullsized":"2024-03-24T11:56:10.978886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'22","last_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T11:55:21.286727+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:35.123566+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990439+0000","last_change":"2024-03-24T11:55:12.810702+0000","last_active":"2024-03-24T11:56:10.990439+0000","last_peered":"2024-03-24T11:56:10.990439+0000","last_clean":"2024-03-24T11:56:10.990439+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:56:10.990439+0000","last_undegraded":"2024-03-24T11:56:10.990439+0000","last_fullsized":"2024-03-24T11:56:10.990439+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:12.810626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.810626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:30.053632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.005515+0000","last_change":"2024-03-24T11:56:03.305225+0000","last_active":"2024-03-24T11:56:11.005515+0000","last_peered":"2024-03-24T11:56:11.005515+0000","last_clean":"2024-03-24T11:56:11.005515+0000","last_became_active":"2024-03-24T11:52:19.378282+0000","last_became_peered":"2024-03-24T11:52:19.378282+0000","last_unstale":"2024-03-24T11:56:11.005515+0000","last_undegraded":"2024-03-24T11:56:11.005515+0000","last_fullsized":"2024-03-24T11:56:11.005515+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:03.305072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:03.305072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:17.590871+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.977953+0000","last_change":"2024-03-24T11:55:20.239169+0000","last_active":"2024-03-24T11:56:10.977953+0000","last_peered":"2024-03-24T11:56:10.977953+0000","last_clean":"2024-03-24T11:56:10.977953+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:56:10.977953+0000","last_undegraded":"2024-03-24T11:56:10.977953+0000","last_fullsized":"2024-03-24T11:56:10.977953+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.239058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.239058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.694647+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990507+0000","last_change":"2024-03-24T11:56:08.033498+0000","last_active":"2024-03-24T11:56:10.990507+0000","last_peered":"2024-03-24T11:56:10.990507+0000","last_clean":"2024-03-24T11:56:10.990507+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:56:10.990507+0000","last_undegraded":"2024-03-24T11:56:10.990507+0000","last_fullsized":"2024-03-24T11:56:10.990507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:08.033370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.033370+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:20.438822+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980286+0000","last_change":"2024-03-24T11:54:54.848417+0000","last_active":"2024-03-24T11:56:10.980286+0000","last_peered":"2024-03-24T11:56:10.980286+0000","last_clean":"2024-03-24T11:56:10.980286+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:56:10.980286+0000","last_undegraded":"2024-03-24T11:56:10.980286+0000","last_fullsized":"2024-03-24T11:56:10.980286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:54.848274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:54:54.848274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:11.927198+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978488+0000","last_change":"2024-03-24T11:56:00.086492+0000","last_active":"2024-03-24T11:56:10.978488+0000","last_peered":"2024-03-24T11:56:10.978488+0000","last_clean":"2024-03-24T11:56:10.978488+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:56:10.978488+0000","last_undegraded":"2024-03-24T11:56:10.978488+0000","last_fullsized":"2024-03-24T11:56:10.978488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:00.086396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:00.086396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:01.178550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979604+0000","last_change":"2024-03-24T11:55:09.427153+0000","last_active":"2024-03-24T11:56:10.979604+0000","last_peered":"2024-03-24T11:56:10.979604+0000","last_clean":"2024-03-24T11:56:10.979604+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:56:10.979604+0000","last_undegraded":"2024-03-24T11:56:10.979604+0000","last_fullsized":"2024-03-24T11:56:10.979604+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:09.427033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:09.427033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:11.944518+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990743+0000","last_change":"2024-03-24T11:54:56.987528+0000","last_active":"2024-03-24T11:56:10.990743+0000","last_peered":"2024-03-24T11:56:10.990743+0000","last_clean":"2024-03-24T11:56:10.990743+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:56:10.990743+0000","last_undegraded":"2024-03-24T11:56:10.990743+0000","last_fullsized":"2024-03-24T11:56:10.990743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:54:56.987400+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:53:20.591632+0000","last_clean_scrub_stamp":"2024-03-24T11:54:56.987400+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.974438+0000","last_change":"2024-03-24T11:54:58.045843+0000","last_active":"2024-03-24T11:56:10.974438+0000","last_peered":"2024-03-24T11:56:10.974438+0000","last_clean":"2024-03-24T11:56:10.974438+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:56:10.974438+0000","last_undegraded":"2024-03-24T11:56:10.974438+0000","last_fullsized":"2024-03-24T11:56:10.974438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.045730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.045730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:26.511106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":142,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.981152+0000","last_change":"2024-03-24T11:56:03.549369+0000","last_active":"2024-03-24T11:56:10.981152+0000","last_peered":"2024-03-24T11:56:10.981152+0000","last_clean":"2024-03-24T11:56:10.981152+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:56:10.981152+0000","last_undegraded":"2024-03-24T11:56:10.981152+0000","last_fullsized":"2024-03-24T11:56:10.981152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:03.549293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:03.549293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:10.966750+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979013+0000","last_change":"2024-03-24T11:55:07.333642+0000","last_active":"2024-03-24T11:56:10.979013+0000","last_peered":"2024-03-24T11:56:10.979013+0000","last_clean":"2024-03-24T11:56:10.979013+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:56:10.979013+0000","last_undegraded":"2024-03-24T11:56:10.979013+0000","last_fullsized":"2024-03-24T11:56:10.979013+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:07.333529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:07.333529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:15.496508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990477+0000","last_change":"2024-03-24T11:54:47.842331+0000","last_active":"2024-03-24T11:56:10.990477+0000","last_peered":"2024-03-24T11:56:10.990477+0000","last_clean":"2024-03-24T11:56:10.990477+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:56:10.990477+0000","last_undegraded":"2024-03-24T11:56:10.990477+0000","last_fullsized":"2024-03-24T11:56:10.990477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:54:47.842199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:47.842199+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980387+0000","last_change":"2024-03-24T11:54:58.383503+0000","last_active":"2024-03-24T11:56:10.980387+0000","last_peered":"2024-03-24T11:56:10.980387+0000","last_clean":"2024-03-24T11:56:10.980387+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:56:10.980387+0000","last_undegraded":"2024-03-24T11:56:10.980387+0000","last_fullsized":"2024-03-24T11:56:10.980387+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.383392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.383392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:11.089085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":134,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990935+0000","last_change":"2024-03-24T11:54:15.707976+0000","last_active":"2024-03-24T11:56:10.990935+0000","last_peered":"2024-03-24T11:56:10.990935+0000","last_clean":"2024-03-24T11:56:10.990935+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:56:10.990935+0000","last_undegraded":"2024-03-24T11:56:10.990935+0000","last_fullsized":"2024-03-24T11:56:10.990935+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.707872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.707872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978668+0000","last_change":"2024-03-24T11:54:59.374964+0000","last_active":"2024-03-24T11:56:10.978668+0000","last_peered":"2024-03-24T11:56:10.978668+0000","last_clean":"2024-03-24T11:56:10.978668+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:56:10.978668+0000","last_undegraded":"2024-03-24T11:56:10.978668+0000","last_fullsized":"2024-03-24T11:56:10.978668+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:54:59.374820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:59.374820+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:20.757347+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":194,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.977631+0000","last_change":"2024-03-24T11:55:19.242716+0000","last_active":"2024-03-24T11:56:10.977631+0000","last_peered":"2024-03-24T11:56:10.977631+0000","last_clean":"2024-03-24T11:56:10.977631+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:56:10.977631+0000","last_undegraded":"2024-03-24T11:56:10.977631+0000","last_fullsized":"2024-03-24T11:56:10.977631+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:55:19.242577+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.242577+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:40.357687+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991376+0000","last_change":"2024-03-24T11:54:52.942734+0000","last_active":"2024-03-24T11:56:10.991376+0000","last_peered":"2024-03-24T11:56:10.991376+0000","last_clean":"2024-03-24T11:56:10.991376+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:56:10.991376+0000","last_undegraded":"2024-03-24T11:56:10.991376+0000","last_fullsized":"2024-03-24T11:56:10.991376+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:52.942614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.942614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:14.652492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008519+0000","last_change":"2024-03-24T11:55:14.026391+0000","last_active":"2024-03-24T11:56:11.008519+0000","last_peered":"2024-03-24T11:56:11.008519+0000","last_clean":"2024-03-24T11:56:11.008519+0000","last_became_active":"2024-03-24T11:52:19.384249+0000","last_became_peered":"2024-03-24T11:52:19.384249+0000","last_unstale":"2024-03-24T11:56:11.008519+0000","last_undegraded":"2024-03-24T11:56:11.008519+0000","last_fullsized":"2024-03-24T11:56:11.008519+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:14.026252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.026252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.097029+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.969157+0000","last_change":"2024-03-24T11:54:57.016465+0000","last_active":"2024-03-24T11:56:10.969157+0000","last_peered":"2024-03-24T11:56:10.969157+0000","last_clean":"2024-03-24T11:56:10.969157+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:56:10.969157+0000","last_undegraded":"2024-03-24T11:56:10.969157+0000","last_fullsized":"2024-03-24T11:56:10.969157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:57.016388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:57.016388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"115'22","reported_seq":180,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.992627+0000","last_change":"2024-03-24T11:55:12.618998+0000","last_active":"2024-03-24T11:56:10.992627+0000","last_peered":"2024-03-24T11:56:10.992627+0000","last_clean":"2024-03-24T11:56:10.992627+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:56:10.992627+0000","last_undegraded":"2024-03-24T11:56:10.992627+0000","last_fullsized":"2024-03-24T11:56:10.992627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'22","last_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.618646+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:26.578040+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008072+0000","last_change":"2024-03-24T11:55:18.959374+0000","last_active":"2024-03-24T11:56:11.008072+0000","last_peered":"2024-03-24T11:56:11.008072+0000","last_clean":"2024-03-24T11:56:11.008072+0000","last_became_active":"2024-03-24T11:52:21.390764+0000","last_became_peered":"2024-03-24T11:52:21.390764+0000","last_unstale":"2024-03-24T11:56:11.008072+0000","last_undegraded":"2024-03-24T11:56:11.008072+0000","last_fullsized":"2024-03-24T11:56:11.008072+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.958919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.958919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:20.739685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.976474+0000","last_change":"2024-03-24T11:55:06.054268+0000","last_active":"2024-03-24T11:56:10.976474+0000","last_peered":"2024-03-24T11:56:10.976474+0000","last_clean":"2024-03-24T11:56:10.976474+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:56:10.976474+0000","last_undegraded":"2024-03-24T11:56:10.976474+0000","last_fullsized":"2024-03-24T11:56:10.976474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.054160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.054160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.176617+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":145,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978518+0000","last_change":"2024-03-24T11:54:06.564061+0000","last_active":"2024-03-24T11:56:10.978518+0000","last_peered":"2024-03-24T11:56:10.978518+0000","last_clean":"2024-03-24T11:56:10.978518+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:56:10.978518+0000","last_undegraded":"2024-03-24T11:56:10.978518+0000","last_fullsized":"2024-03-24T11:56:10.978518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:54:06.563966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:06.563966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":165,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.969738+0000","last_change":"2024-03-24T11:54:50.090222+0000","last_active":"2024-03-24T11:56:10.969738+0000","last_peered":"2024-03-24T11:56:10.969738+0000","last_clean":"2024-03-24T11:56:10.969738+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:56:10.969738+0000","last_undegraded":"2024-03-24T11:56:10.969738+0000","last_fullsized":"2024-03-24T11:56:10.969738+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:54:50.090089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:50.090089+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979376+0000","last_change":"2024-03-24T11:54:54.491445+0000","last_active":"2024-03-24T11:56:10.979376+0000","last_peered":"2024-03-24T11:56:10.979376+0000","last_clean":"2024-03-24T11:56:10.979376+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:56:10.979376+0000","last_undegraded":"2024-03-24T11:56:10.979376+0000","last_fullsized":"2024-03-24T11:56:10.979376+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:54:54.491303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:54.491303+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":201,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.969779+0000","last_change":"2024-03-24T11:56:06.931131+0000","last_active":"2024-03-24T11:56:10.969779+0000","last_peered":"2024-03-24T11:56:10.969779+0000","last_clean":"2024-03-24T11:56:10.969779+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:56:10.969779+0000","last_undegraded":"2024-03-24T11:56:10.969779+0000","last_fullsized":"2024-03-24T11:56:10.969779+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:06.931035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.931035+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.652773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980895+0000","last_change":"2024-03-24T11:55:00.754485+0000","last_active":"2024-03-24T11:56:10.980895+0000","last_peered":"2024-03-24T11:56:10.980895+0000","last_clean":"2024-03-24T11:56:10.980895+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:56:10.980895+0000","last_undegraded":"2024-03-24T11:56:10.980895+0000","last_fullsized":"2024-03-24T11:56:10.980895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:00.754339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:00.754339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:14.930276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":159,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.976378+0000","last_change":"2024-03-24T11:53:46.014153+0000","last_active":"2024-03-24T11:56:10.976378+0000","last_peered":"2024-03-24T11:56:10.976378+0000","last_clean":"2024-03-24T11:56:10.976378+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:56:10.976378+0000","last_undegraded":"2024-03-24T11:56:10.976378+0000","last_fullsized":"2024-03-24T11:56:10.976378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_clean_scrub_stamp":"2024-03-24T11:53:46.014065+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.993524+0000","last_change":"2024-03-24T11:54:49.424771+0000","last_active":"2024-03-24T11:56:10.993524+0000","last_peered":"2024-03-24T11:56:10.993524+0000","last_clean":"2024-03-24T11:56:10.993524+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:56:10.993524+0000","last_undegraded":"2024-03-24T11:56:10.993524+0000","last_fullsized":"2024-03-24T11:56:10.993524+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:49.424403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:49.424403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979720+0000","last_change":"2024-03-24T11:56:01.069407+0000","last_active":"2024-03-24T11:56:10.979720+0000","last_peered":"2024-03-24T11:56:10.979720+0000","last_clean":"2024-03-24T11:56:10.979720+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:56:10.979720+0000","last_undegraded":"2024-03-24T11:56:10.979720+0000","last_fullsized":"2024-03-24T11:56:10.979720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:01.069280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.632287+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.069280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:11.863245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.969563+0000","last_change":"2024-03-24T11:56:04.915981+0000","last_active":"2024-03-24T11:56:10.969563+0000","last_peered":"2024-03-24T11:56:10.969563+0000","last_clean":"2024-03-24T11:56:10.969563+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:56:10.969563+0000","last_undegraded":"2024-03-24T11:56:10.969563+0000","last_fullsized":"2024-03-24T11:56:10.969563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:04.915872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:04.915872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:20.330801+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":167,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978792+0000","last_change":"2024-03-24T11:56:05.799020+0000","last_active":"2024-03-24T11:56:10.978792+0000","last_peered":"2024-03-24T11:56:10.978792+0000","last_clean":"2024-03-24T11:56:10.978792+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:56:10.978792+0000","last_undegraded":"2024-03-24T11:56:10.978792+0000","last_fullsized":"2024-03-24T11:56:10.978792+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:05.798510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.798510+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:34.206728+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978321+0000","last_change":"2024-03-24T11:53:51.547818+0000","last_active":"2024-03-24T11:56:10.978321+0000","last_peered":"2024-03-24T11:56:10.978321+0000","last_clean":"2024-03-24T11:56:10.978321+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:56:10.978321+0000","last_undegraded":"2024-03-24T11:56:10.978321+0000","last_fullsized":"2024-03-24T11:56:10.978321+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:51.547680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:51.547680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":142,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.968666+0000","last_change":"2024-03-24T11:54:58.059296+0000","last_active":"2024-03-24T11:56:10.968666+0000","last_peered":"2024-03-24T11:56:10.968666+0000","last_clean":"2024-03-24T11:56:10.968666+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:56:10.968666+0000","last_undegraded":"2024-03-24T11:56:10.968666+0000","last_fullsized":"2024-03-24T11:56:10.968666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.059216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.059216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:21.338010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.982966+0000","last_change":"2024-03-24T11:55:18.619637+0000","last_active":"2024-03-24T11:56:10.982966+0000","last_peered":"2024-03-24T11:56:10.982966+0000","last_clean":"2024-03-24T11:56:10.982966+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:56:10.982966+0000","last_undegraded":"2024-03-24T11:56:10.982966+0000","last_fullsized":"2024-03-24T11:56:10.982966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.619533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.619533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:43.172807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":154,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979380+0000","last_change":"2024-03-24T11:55:00.363697+0000","last_active":"2024-03-24T11:56:10.979380+0000","last_peered":"2024-03-24T11:56:10.979380+0000","last_clean":"2024-03-24T11:56:10.979380+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:56:10.979380+0000","last_undegraded":"2024-03-24T11:56:10.979380+0000","last_fullsized":"2024-03-24T11:56:10.979380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T11:55:00.363559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:00.363559+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.459697+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978423+0000","last_change":"2024-03-24T11:54:51.408148+0000","last_active":"2024-03-24T11:56:10.978423+0000","last_peered":"2024-03-24T11:56:10.978423+0000","last_clean":"2024-03-24T11:56:10.978423+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:56:10.978423+0000","last_undegraded":"2024-03-24T11:56:10.978423+0000","last_fullsized":"2024-03-24T11:56:10.978423+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:51.408018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:51.408018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008706+0000","last_change":"2024-03-24T11:55:56.132313+0000","last_active":"2024-03-24T11:56:11.008706+0000","last_peered":"2024-03-24T11:56:11.008706+0000","last_clean":"2024-03-24T11:56:11.008706+0000","last_became_active":"2024-03-24T11:52:19.377484+0000","last_became_peered":"2024-03-24T11:52:19.377484+0000","last_unstale":"2024-03-24T11:56:11.008706+0000","last_undegraded":"2024-03-24T11:56:11.008706+0000","last_fullsized":"2024-03-24T11:56:11.008706+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:56.132189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:55:56.132189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:01.345699+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008247+0000","last_change":"2024-03-24T11:55:15.993168+0000","last_active":"2024-03-24T11:56:11.008247+0000","last_peered":"2024-03-24T11:56:11.008247+0000","last_clean":"2024-03-24T11:56:11.008247+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:56:11.008247+0000","last_undegraded":"2024-03-24T11:56:11.008247+0000","last_fullsized":"2024-03-24T11:56:11.008247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:15.992998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:15.992998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:24.619769+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":165,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990815+0000","last_change":"2024-03-24T11:56:01.946316+0000","last_active":"2024-03-24T11:56:10.990815+0000","last_peered":"2024-03-24T11:56:10.990815+0000","last_clean":"2024-03-24T11:56:10.990815+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:56:10.990815+0000","last_undegraded":"2024-03-24T11:56:10.990815+0000","last_fullsized":"2024-03-24T11:56:10.990815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:01.946162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.946162+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:10.718956+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.972563+0000","last_change":"2024-03-24T11:55:19.108172+0000","last_active":"2024-03-24T11:56:10.972563+0000","last_peered":"2024-03-24T11:56:10.972563+0000","last_clean":"2024-03-24T11:56:10.972563+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:56:10.972563+0000","last_undegraded":"2024-03-24T11:56:10.972563+0000","last_fullsized":"2024-03-24T11:56:10.972563+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.108015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.299012+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978773+0000","last_change":"2024-03-24T11:55:22.292901+0000","last_active":"2024-03-24T11:56:10.978773+0000","last_peered":"2024-03-24T11:56:10.978773+0000","last_clean":"2024-03-24T11:56:10.978773+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:56:10.978773+0000","last_undegraded":"2024-03-24T11:56:10.978773+0000","last_fullsized":"2024-03-24T11:56:10.978773+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:22.292785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.292785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:22.310604+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.968977+0000","last_change":"2024-03-24T11:55:01.977733+0000","last_active":"2024-03-24T11:56:10.968977+0000","last_peered":"2024-03-24T11:56:10.968977+0000","last_clean":"2024-03-24T11:56:10.968977+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:56:10.968977+0000","last_undegraded":"2024-03-24T11:56:10.968977+0000","last_fullsized":"2024-03-24T11:56:10.968977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.977610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.977610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:17.519759+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.005831+0000","last_change":"2024-03-24T11:54:50.854728+0000","last_active":"2024-03-24T11:56:11.005831+0000","last_peered":"2024-03-24T11:56:11.005831+0000","last_clean":"2024-03-24T11:56:11.005831+0000","last_became_active":"2024-03-24T11:52:17.355956+0000","last_became_peered":"2024-03-24T11:52:17.355956+0000","last_unstale":"2024-03-24T11:56:11.005831+0000","last_undegraded":"2024-03-24T11:56:11.005831+0000","last_fullsized":"2024-03-24T11:56:11.005831+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:54:50.854256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:50.854256+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991482+0000","last_change":"2024-03-24T11:55:11.806686+0000","last_active":"2024-03-24T11:56:10.991482+0000","last_peered":"2024-03-24T11:56:10.991482+0000","last_clean":"2024-03-24T11:56:10.991482+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:56:10.991482+0000","last_undegraded":"2024-03-24T11:56:10.991482+0000","last_fullsized":"2024-03-24T11:56:10.991482+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.806582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.806582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:34.108587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":200,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979842+0000","last_change":"2024-03-24T11:55:01.393587+0000","last_active":"2024-03-24T11:56:10.979842+0000","last_peered":"2024-03-24T11:56:10.979842+0000","last_clean":"2024-03-24T11:56:10.979842+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:56:10.979842+0000","last_undegraded":"2024-03-24T11:56:10.979842+0000","last_fullsized":"2024-03-24T11:56:10.979842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:01.393318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.393318+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:12.523269+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.974138+0000","last_change":"2024-03-24T11:54:05.851953+0000","last_active":"2024-03-24T11:56:10.974138+0000","last_peered":"2024-03-24T11:56:10.974138+0000","last_clean":"2024-03-24T11:56:10.974138+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:56:10.974138+0000","last_undegraded":"2024-03-24T11:56:10.974138+0000","last_fullsized":"2024-03-24T11:56:10.974138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:05.851860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:05.851860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980877+0000","last_change":"2024-03-24T11:55:13.643173+0000","last_active":"2024-03-24T11:56:10.980877+0000","last_peered":"2024-03-24T11:56:10.980877+0000","last_clean":"2024-03-24T11:56:10.980877+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:56:10.980877+0000","last_undegraded":"2024-03-24T11:56:10.980877+0000","last_fullsized":"2024-03-24T11:56:10.980877+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:55:13.643063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:13.643063+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:20.737682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.975795+0000","last_change":"2024-03-24T11:53:55.915626+0000","last_active":"2024-03-24T11:56:10.975795+0000","last_peered":"2024-03-24T11:56:10.975795+0000","last_clean":"2024-03-24T11:56:10.975795+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:56:10.975795+0000","last_undegraded":"2024-03-24T11:56:10.975795+0000","last_fullsized":"2024-03-24T11:56:10.975795+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:55.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:53:55.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":198,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980684+0000","last_change":"2024-03-24T11:54:58.781360+0000","last_active":"2024-03-24T11:56:10.980684+0000","last_peered":"2024-03-24T11:56:10.980684+0000","last_clean":"2024-03-24T11:56:10.980684+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:56:10.980684+0000","last_undegraded":"2024-03-24T11:56:10.980684+0000","last_fullsized":"2024-03-24T11:56:10.980684+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:54:58.781231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.781231+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980458+0000","last_change":"2024-03-24T11:55:21.647704+0000","last_active":"2024-03-24T11:56:10.980458+0000","last_peered":"2024-03-24T11:56:10.980458+0000","last_clean":"2024-03-24T11:56:10.980458+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:56:10.980458+0000","last_undegraded":"2024-03-24T11:56:10.980458+0000","last_fullsized":"2024-03-24T11:56:10.980458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:21.647570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T11:55:21.647570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:30.306492+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":168,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979693+0000","last_change":"2024-03-24T11:55:06.420550+0000","last_active":"2024-03-24T11:56:10.979693+0000","last_peered":"2024-03-24T11:56:10.979693+0000","last_clean":"2024-03-24T11:56:10.979693+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:56:10.979693+0000","last_undegraded":"2024-03-24T11:56:10.979693+0000","last_fullsized":"2024-03-24T11:56:10.979693+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:55:06.420406+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.420406+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.347160+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979543+0000","last_change":"2024-03-24T11:55:11.462967+0000","last_active":"2024-03-24T11:56:10.979543+0000","last_peered":"2024-03-24T11:56:10.979543+0000","last_clean":"2024-03-24T11:56:10.979543+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:56:10.979543+0000","last_undegraded":"2024-03-24T11:56:10.979543+0000","last_fullsized":"2024-03-24T11:56:10.979543+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.462846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.462846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:18.783674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":142,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.982850+0000","last_change":"2024-03-24T11:56:05.489055+0000","last_active":"2024-03-24T11:56:10.982850+0000","last_peered":"2024-03-24T11:56:10.982850+0000","last_clean":"2024-03-24T11:56:10.982850+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:56:10.982850+0000","last_undegraded":"2024-03-24T11:56:10.982850+0000","last_fullsized":"2024-03-24T11:56:10.982850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:05.488932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.488932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:20.205595+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.968763+0000","last_change":"2024-03-24T11:54:48.200814+0000","last_active":"2024-03-24T11:56:10.968763+0000","last_peered":"2024-03-24T11:56:10.968763+0000","last_clean":"2024-03-24T11:56:10.968763+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:56:10.968763+0000","last_undegraded":"2024-03-24T11:56:10.968763+0000","last_fullsized":"2024-03-24T11:56:10.968763+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:48.200693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:48.200693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":157,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980483+0000","last_change":"2024-03-24T11:54:52.361385+0000","last_active":"2024-03-24T11:56:10.980483+0000","last_peered":"2024-03-24T11:56:10.980483+0000","last_clean":"2024-03-24T11:56:10.980483+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:56:10.980483+0000","last_undegraded":"2024-03-24T11:56:10.980483+0000","last_fullsized":"2024-03-24T11:56:10.980483+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:54:52.361260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.361260+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:12.035583+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.992169+0000","last_change":"2024-03-24T11:56:01.567568+0000","last_active":"2024-03-24T11:56:10.992169+0000","last_peered":"2024-03-24T11:56:10.992169+0000","last_clean":"2024-03-24T11:56:10.992169+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:56:10.992169+0000","last_undegraded":"2024-03-24T11:56:10.992169+0000","last_fullsized":"2024-03-24T11:56:10.992169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:01.567421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.567421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.945947+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980575+0000","last_change":"2024-03-24T11:54:15.542707+0000","last_active":"2024-03-24T11:56:10.980575+0000","last_peered":"2024-03-24T11:56:10.980575+0000","last_clean":"2024-03-24T11:56:10.980575+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:56:10.980575+0000","last_undegraded":"2024-03-24T11:56:10.980575+0000","last_fullsized":"2024-03-24T11:56:10.980575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.542625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.542625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.992396+0000","last_change":"2024-03-24T11:55:01.390204+0000","last_active":"2024-03-24T11:56:10.992396+0000","last_peered":"2024-03-24T11:56:10.992396+0000","last_clean":"2024-03-24T11:56:10.992396+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:56:10.992396+0000","last_undegraded":"2024-03-24T11:56:10.992396+0000","last_fullsized":"2024-03-24T11:56:10.992396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.390019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.390019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:30.098357+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"115'13","reported_seq":167,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.976110+0000","last_change":"2024-03-24T11:55:16.094065+0000","last_active":"2024-03-24T11:56:10.976110+0000","last_peered":"2024-03-24T11:56:10.976110+0000","last_clean":"2024-03-24T11:56:10.976110+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:56:10.976110+0000","last_undegraded":"2024-03-24T11:56:10.976110+0000","last_fullsized":"2024-03-24T11:56:10.976110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'13","last_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T11:55:16.093668+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:25.577045+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"129'8024","reported_seq":8098,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.265586+0000","last_change":"2024-03-24T11:54:16.232420+0000","last_active":"2024-03-24T11:56:11.265586+0000","last_peered":"2024-03-24T11:56:11.265586+0000","last_clean":"2024-03-24T11:56:11.265586+0000","last_became_active":"2024-03-24T11:54:16.232098+0000","last_became_peered":"2024-03-24T11:54:16.232098+0000","last_unstale":"2024-03-24T11:56:11.265586+0000","last_undegraded":"2024-03-24T11:56:11.265586+0000","last_fullsized":"2024-03-24T11:56:11.265586+0000","mapping_epoch":97,"log_start":"111'3300","ondisk_log_start":"111'3300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481296384,"num_objects":7344,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8024,"num_write_kb":491776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978877+0000","last_change":"2024-03-24T11:56:02.765493+0000","last_active":"2024-03-24T11:56:10.978877+0000","last_peered":"2024-03-24T11:56:10.978877+0000","last_clean":"2024-03-24T11:56:10.978877+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:56:10.978877+0000","last_undegraded":"2024-03-24T11:56:10.978877+0000","last_fullsized":"2024-03-24T11:56:10.978877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:02.765358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:02.765358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.107895+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991349+0000","last_change":"2024-03-24T11:55:20.861818+0000","last_active":"2024-03-24T11:56:10.991349+0000","last_peered":"2024-03-24T11:56:10.991349+0000","last_clean":"2024-03-24T11:56:10.991349+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:56:10.991349+0000","last_undegraded":"2024-03-24T11:56:10.991349+0000","last_fullsized":"2024-03-24T11:56:10.991349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:55:20.861648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.861648+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:38.570780+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":187,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990554+0000","last_change":"2024-03-24T11:54:48.888639+0000","last_active":"2024-03-24T11:56:10.990554+0000","last_peered":"2024-03-24T11:56:10.990554+0000","last_clean":"2024-03-24T11:56:10.990554+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:56:10.990554+0000","last_undegraded":"2024-03-24T11:56:10.990554+0000","last_fullsized":"2024-03-24T11:56:10.990554+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:54:48.888525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:48.888525+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991896+0000","last_change":"2024-03-24T11:55:19.652898+0000","last_active":"2024-03-24T11:56:10.991896+0000","last_peered":"2024-03-24T11:56:10.991896+0000","last_clean":"2024-03-24T11:56:10.991896+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:56:10.991896+0000","last_undegraded":"2024-03-24T11:56:10.991896+0000","last_fullsized":"2024-03-24T11:56:10.991896+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:19.652641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.652641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:28.881745+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"129'7885","reported_seq":8042,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.126831+0000","last_change":"2024-03-24T11:56:08.680053+0000","last_active":"2024-03-24T11:56:11.126831+0000","last_peered":"2024-03-24T11:56:11.126831+0000","last_clean":"2024-03-24T11:56:11.126831+0000","last_became_active":"2024-03-24T11:54:16.238055+0000","last_became_peered":"2024-03-24T11:54:16.238055+0000","last_unstale":"2024-03-24T11:56:11.126831+0000","last_undegraded":"2024-03-24T11:56:11.126831+0000","last_fullsized":"2024-03-24T11:56:11.126831+0000","mapping_epoch":97,"log_start":"108'2600","ondisk_log_start":"108'2600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7741","last_scrub_stamp":"2024-03-24T11:56:08.679818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.679818+0000","objects_scrubbed":7448,"log_size":5285,"log_dups_size":0,"ondisk_log_size":5285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.903434+0000","scrub_duration":1667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475463680,"num_objects":7255,"num_object_clones":0,"num_object_copies":21765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7885,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.972801+0000","last_change":"2024-03-24T11:54:04.892146+0000","last_active":"2024-03-24T11:56:10.972801+0000","last_peered":"2024-03-24T11:56:10.972801+0000","last_clean":"2024-03-24T11:56:10.972801+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:56:10.972801+0000","last_undegraded":"2024-03-24T11:56:10.972801+0000","last_fullsized":"2024-03-24T11:56:10.972801+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:04.892038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:04.892038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008983+0000","last_change":"2024-03-24T11:55:04.005537+0000","last_active":"2024-03-24T11:56:11.008983+0000","last_peered":"2024-03-24T11:56:11.008983+0000","last_clean":"2024-03-24T11:56:11.008983+0000","last_became_active":"2024-03-24T11:52:15.138792+0000","last_became_peered":"2024-03-24T11:52:15.138792+0000","last_unstale":"2024-03-24T11:56:11.008983+0000","last_undegraded":"2024-03-24T11:56:11.008983+0000","last_fullsized":"2024-03-24T11:56:11.008983+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:04.005410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:55:04.005410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:19.205017+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.005968+0000","last_change":"2024-03-24T11:55:01.917263+0000","last_active":"2024-03-24T11:56:11.005968+0000","last_peered":"2024-03-24T11:56:11.005968+0000","last_clean":"2024-03-24T11:56:11.005968+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:56:11.005968+0000","last_undegraded":"2024-03-24T11:56:11.005968+0000","last_fullsized":"2024-03-24T11:56:11.005968+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:55:01.917120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.917120+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.443473+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":191,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.974327+0000","last_change":"2024-03-24T11:54:00.896739+0000","last_active":"2024-03-24T11:56:10.974327+0000","last_peered":"2024-03-24T11:56:10.974327+0000","last_clean":"2024-03-24T11:56:10.974327+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:56:10.974327+0000","last_undegraded":"2024-03-24T11:56:10.974327+0000","last_fullsized":"2024-03-24T11:56:10.974327+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:54:00.896635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:00.896635+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"129'8079","reported_seq":8236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.115435+0000","last_change":"2024-03-24T11:56:10.140756+0000","last_active":"2024-03-24T11:56:11.115435+0000","last_peered":"2024-03-24T11:56:11.115435+0000","last_clean":"2024-03-24T11:56:11.115435+0000","last_became_active":"2024-03-24T11:54:16.240514+0000","last_became_peered":"2024-03-24T11:54:16.240514+0000","last_unstale":"2024-03-24T11:56:11.115435+0000","last_undegraded":"2024-03-24T11:56:11.115435+0000","last_fullsized":"2024-03-24T11:56:11.115435+0000","mapping_epoch":97,"log_start":"108'1900","ondisk_log_start":"108'1900","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'8025","last_scrub_stamp":"2024-03-24T11:56:10.140421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.140421+0000","objects_scrubbed":7499,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:13.331670+0000","scrub_duration":2191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483983360,"num_objects":7385,"num_object_clones":0,"num_object_copies":22155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":494848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978030+0000","last_change":"2024-03-24T11:55:12.389096+0000","last_active":"2024-03-24T11:56:10.978030+0000","last_peered":"2024-03-24T11:56:10.978030+0000","last_clean":"2024-03-24T11:56:10.978030+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:56:10.978030+0000","last_undegraded":"2024-03-24T11:56:10.978030+0000","last_fullsized":"2024-03-24T11:56:10.978030+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:12.388835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.388835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:32.363194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980555+0000","last_change":"2024-03-24T11:54:52.822234+0000","last_active":"2024-03-24T11:56:10.980555+0000","last_peered":"2024-03-24T11:56:10.980555+0000","last_clean":"2024-03-24T11:56:10.980555+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:56:10.980555+0000","last_undegraded":"2024-03-24T11:56:10.980555+0000","last_fullsized":"2024-03-24T11:56:10.980555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:52.822135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.822135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:12.399116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":186,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980449+0000","last_change":"2024-03-24T11:54:07.842648+0000","last_active":"2024-03-24T11:56:10.980449+0000","last_peered":"2024-03-24T11:56:10.980449+0000","last_clean":"2024-03-24T11:56:10.980449+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:56:10.980449+0000","last_undegraded":"2024-03-24T11:56:10.980449+0000","last_fullsized":"2024-03-24T11:56:10.980449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-03-24T11:54:07.842555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:07.842555+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.984196+0000","last_change":"2024-03-24T11:56:06.187599+0000","last_active":"2024-03-24T11:56:10.984196+0000","last_peered":"2024-03-24T11:56:10.984196+0000","last_clean":"2024-03-24T11:56:10.984196+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:56:10.984196+0000","last_undegraded":"2024-03-24T11:56:10.984196+0000","last_fullsized":"2024-03-24T11:56:10.984196+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:06.187195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.187195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:29.724378+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"129'7899","reported_seq":7971,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.359508+0000","last_change":"2024-03-24T11:54:16.238411+0000","last_active":"2024-03-24T11:56:11.359508+0000","last_peered":"2024-03-24T11:56:11.359508+0000","last_clean":"2024-03-24T11:56:11.359508+0000","last_became_active":"2024-03-24T11:54:16.238056+0000","last_became_peered":"2024-03-24T11:54:16.238056+0000","last_unstale":"2024-03-24T11:56:11.359508+0000","last_undegraded":"2024-03-24T11:56:11.359508+0000","last_fullsized":"2024-03-24T11:56:11.359508+0000","mapping_epoch":97,"log_start":"108'2400","ondisk_log_start":"108'2400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5499,"log_dups_size":0,"ondisk_log_size":5499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471400448,"num_objects":7193,"num_object_clones":0,"num_object_copies":21579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7899,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979714+0000","last_change":"2024-03-24T11:55:57.171256+0000","last_active":"2024-03-24T11:56:10.979714+0000","last_peered":"2024-03-24T11:56:10.979714+0000","last_clean":"2024-03-24T11:56:10.979714+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:56:10.979714+0000","last_undegraded":"2024-03-24T11:56:10.979714+0000","last_fullsized":"2024-03-24T11:56:10.979714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:57.171146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:57.171146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:09.649463+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":391,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.005433+0000","last_change":"2024-03-24T11:55:22.048179+0000","last_active":"2024-03-24T11:56:11.005433+0000","last_peered":"2024-03-24T11:56:11.005433+0000","last_clean":"2024-03-24T11:56:11.005433+0000","last_became_active":"2024-03-24T11:52:15.144954+0000","last_became_peered":"2024-03-24T11:52:15.144954+0000","last_unstale":"2024-03-24T11:56:11.005433+0000","last_undegraded":"2024-03-24T11:56:11.005433+0000","last_fullsized":"2024-03-24T11:56:11.005433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:55:22.048036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.048036+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:46.092889+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":230,"num_read_kb":225,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991704+0000","last_change":"2024-03-24T11:56:10.706814+0000","last_active":"2024-03-24T11:56:10.991704+0000","last_peered":"2024-03-24T11:56:10.991704+0000","last_clean":"2024-03-24T11:56:10.991704+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:56:10.991704+0000","last_undegraded":"2024-03-24T11:56:10.991704+0000","last_fullsized":"2024-03-24T11:56:10.991704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.706685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.706685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:10.842111+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":174,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008728+0000","last_change":"2024-03-24T11:56:04.340339+0000","last_active":"2024-03-24T11:56:11.008728+0000","last_peered":"2024-03-24T11:56:11.008728+0000","last_clean":"2024-03-24T11:56:11.008728+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:56:11.008728+0000","last_undegraded":"2024-03-24T11:56:11.008728+0000","last_fullsized":"2024-03-24T11:56:11.008728+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-03-24T11:56:04.340193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:04.340193+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:09.770652+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.993028+0000","last_change":"2024-03-24T11:55:13.600385+0000","last_active":"2024-03-24T11:56:10.993028+0000","last_peered":"2024-03-24T11:56:10.993028+0000","last_clean":"2024-03-24T11:56:10.993028+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:56:10.993028+0000","last_undegraded":"2024-03-24T11:56:10.993028+0000","last_fullsized":"2024-03-24T11:56:10.993028+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T11:55:13.600133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:19.274997+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"129'7655","reported_seq":7729,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.336179+0000","last_change":"2024-03-24T11:54:16.238815+0000","last_active":"2024-03-24T11:56:11.336179+0000","last_peered":"2024-03-24T11:56:11.336179+0000","last_clean":"2024-03-24T11:56:11.336179+0000","last_became_active":"2024-03-24T11:54:16.238515+0000","last_became_peered":"2024-03-24T11:54:16.238515+0000","last_unstale":"2024-03-24T11:56:11.336179+0000","last_undegraded":"2024-03-24T11:56:11.336179+0000","last_fullsized":"2024-03-24T11:56:11.336179+0000","mapping_epoch":97,"log_start":"108'2200","ondisk_log_start":"108'2200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462487552,"num_objects":7057,"num_object_clones":0,"num_object_copies":21171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7655,"num_write_kb":470784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.981464+0000","last_change":"2024-03-24T11:55:10.692203+0000","last_active":"2024-03-24T11:56:10.981464+0000","last_peered":"2024-03-24T11:56:10.981464+0000","last_clean":"2024-03-24T11:56:10.981464+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:56:10.981464+0000","last_undegraded":"2024-03-24T11:56:10.981464+0000","last_fullsized":"2024-03-24T11:56:10.981464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:10.692130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:55:10.692130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:22.802403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":161,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980941+0000","last_change":"2024-03-24T11:55:56.666658+0000","last_active":"2024-03-24T11:56:10.980941+0000","last_peered":"2024-03-24T11:56:10.980941+0000","last_clean":"2024-03-24T11:56:10.980941+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:56:10.980941+0000","last_undegraded":"2024-03-24T11:56:10.980941+0000","last_fullsized":"2024-03-24T11:56:10.980941+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T11:55:56.665991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:56.665991+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:19.338388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.981184+0000","last_change":"2024-03-24T11:55:20.647830+0000","last_active":"2024-03-24T11:56:10.981184+0000","last_peered":"2024-03-24T11:56:10.981184+0000","last_clean":"2024-03-24T11:56:10.981184+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:56:10.981184+0000","last_undegraded":"2024-03-24T11:56:10.981184+0000","last_fullsized":"2024-03-24T11:56:10.981184+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.647693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.647693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:37.648168+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":407,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978543+0000","last_change":"2024-03-24T11:56:05.931697+0000","last_active":"2024-03-24T11:56:10.978543+0000","last_peered":"2024-03-24T11:56:10.978543+0000","last_clean":"2024-03-24T11:56:10.978543+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:56:10.978543+0000","last_undegraded":"2024-03-24T11:56:10.978543+0000","last_fullsized":"2024-03-24T11:56:10.978543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:56:05.931549+0000","last_deep_scrub":"71'119","last_deep_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.931549+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.615604+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":216,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.969986+0000","last_change":"2024-03-24T11:55:22.095371+0000","last_active":"2024-03-24T11:56:10.969986+0000","last_peered":"2024-03-24T11:56:10.969986+0000","last_clean":"2024-03-24T11:56:10.969986+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:56:10.969986+0000","last_undegraded":"2024-03-24T11:56:10.969986+0000","last_fullsized":"2024-03-24T11:56:10.969986+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:55:22.095232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.095232+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:31.457374+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979224+0000","last_change":"2024-03-24T11:55:04.414736+0000","last_active":"2024-03-24T11:56:10.979224+0000","last_peered":"2024-03-24T11:56:10.979224+0000","last_clean":"2024-03-24T11:56:10.979224+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:56:10.979224+0000","last_undegraded":"2024-03-24T11:56:10.979224+0000","last_fullsized":"2024-03-24T11:56:10.979224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:04.414651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:04.414651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:31.408133+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"129'7930","reported_seq":8021,"reported_epoch":129,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T11:56:11.164613+0000","last_change":"2024-03-24T11:56:10.724690+0000","last_active":"2024-03-24T11:56:11.164613+0000","last_peered":"2024-03-24T11:56:11.164613+0000","last_clean":"2024-03-24T11:56:11.164613+0000","last_became_active":"2024-03-24T11:54:16.236927+0000","last_became_peered":"2024-03-24T11:54:16.236927+0000","last_unstale":"2024-03-24T11:56:11.164613+0000","last_undegraded":"2024-03-24T11:56:11.164613+0000","last_fullsized":"2024-03-24T11:56:11.164613+0000","mapping_epoch":97,"log_start":"109'3000","ondisk_log_start":"109'3000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":1399,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477888512,"num_objects":7292,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.010554+0000","last_change":"2024-03-24T11:55:14.967693+0000","last_active":"2024-03-24T11:56:11.010554+0000","last_peered":"2024-03-24T11:56:11.010554+0000","last_clean":"2024-03-24T11:56:11.010554+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:56:11.010554+0000","last_undegraded":"2024-03-24T11:56:11.010554+0000","last_fullsized":"2024-03-24T11:56:11.010554+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:14.967137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.967137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:24.422604+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.974092+0000","last_change":"2024-03-24T11:53:52.992618+0000","last_active":"2024-03-24T11:56:10.974092+0000","last_peered":"2024-03-24T11:56:10.974092+0000","last_clean":"2024-03-24T11:56:10.974092+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:56:10.974092+0000","last_undegraded":"2024-03-24T11:56:10.974092+0000","last_fullsized":"2024-03-24T11:56:10.974092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:53:52.992540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:53:52.992540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978405+0000","last_change":"2024-03-24T11:54:49.540232+0000","last_active":"2024-03-24T11:56:10.978405+0000","last_peered":"2024-03-24T11:56:10.978405+0000","last_clean":"2024-03-24T11:56:10.978405+0000","last_became_active":"2024-03-24T11:52:15.146310+0000","last_became_peered":"2024-03-24T11:52:15.146310+0000","last_unstale":"2024-03-24T11:56:10.978405+0000","last_undegraded":"2024-03-24T11:56:10.978405+0000","last_fullsized":"2024-03-24T11:56:10.978405+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:49.540127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:49.540127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":165,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.975915+0000","last_change":"2024-03-24T11:55:14.029361+0000","last_active":"2024-03-24T11:56:10.975915+0000","last_peered":"2024-03-24T11:56:10.975915+0000","last_clean":"2024-03-24T11:56:10.975915+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:56:10.975915+0000","last_undegraded":"2024-03-24T11:56:10.975915+0000","last_fullsized":"2024-03-24T11:56:10.975915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:55:14.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.029279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:26.539716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":142,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.981362+0000","last_change":"2024-03-24T11:56:08.456477+0000","last_active":"2024-03-24T11:56:10.981362+0000","last_peered":"2024-03-24T11:56:10.981362+0000","last_clean":"2024-03-24T11:56:10.981362+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:56:10.981362+0000","last_undegraded":"2024-03-24T11:56:10.981362+0000","last_fullsized":"2024-03-24T11:56:10.981362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:08.456346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.456346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:13.785080+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"129'8109","reported_seq":8183,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.019236+0000","last_change":"2024-03-24T11:54:16.237179+0000","last_active":"2024-03-24T11:56:11.019236+0000","last_peered":"2024-03-24T11:56:11.019236+0000","last_clean":"2024-03-24T11:56:11.019236+0000","last_became_active":"2024-03-24T11:54:16.236857+0000","last_became_peered":"2024-03-24T11:54:16.236857+0000","last_unstale":"2024-03-24T11:56:11.019236+0000","last_undegraded":"2024-03-24T11:56:11.019236+0000","last_fullsized":"2024-03-24T11:56:11.019236+0000","mapping_epoch":97,"log_start":"108'2400","ondisk_log_start":"108'2400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5709,"log_dups_size":0,"ondisk_log_size":5709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480182272,"num_objects":7327,"num_object_clones":0,"num_object_copies":21981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8109,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":147,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.005386+0000","last_change":"2024-03-24T11:56:10.262911+0000","last_active":"2024-03-24T11:56:11.005386+0000","last_peered":"2024-03-24T11:56:11.005386+0000","last_clean":"2024-03-24T11:56:11.005386+0000","last_became_active":"2024-03-24T11:52:21.851069+0000","last_became_peered":"2024-03-24T11:52:21.851069+0000","last_unstale":"2024-03-24T11:56:11.005386+0000","last_undegraded":"2024-03-24T11:56:11.005386+0000","last_fullsized":"2024-03-24T11:56:11.005386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.262776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.262776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:12.921856+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979810+0000","last_change":"2024-03-24T11:54:51.499930+0000","last_active":"2024-03-24T11:56:10.979810+0000","last_peered":"2024-03-24T11:56:10.979810+0000","last_clean":"2024-03-24T11:56:10.979810+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:56:10.979810+0000","last_undegraded":"2024-03-24T11:56:10.979810+0000","last_fullsized":"2024-03-24T11:56:10.979810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:51.499817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:51.499817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:17.253146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":183,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.974038+0000","last_change":"2024-03-24T11:56:05.227501+0000","last_active":"2024-03-24T11:56:10.974038+0000","last_peered":"2024-03-24T11:56:10.974038+0000","last_clean":"2024-03-24T11:56:10.974038+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:56:10.974038+0000","last_undegraded":"2024-03-24T11:56:10.974038+0000","last_fullsized":"2024-03-24T11:56:10.974038+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:56:05.227347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.227347+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:12.047195+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":142,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.969252+0000","last_change":"2024-03-24T11:55:15.058768+0000","last_active":"2024-03-24T11:56:10.969252+0000","last_peered":"2024-03-24T11:56:10.969252+0000","last_clean":"2024-03-24T11:56:10.969252+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:56:10.969252+0000","last_undegraded":"2024-03-24T11:56:10.969252+0000","last_fullsized":"2024-03-24T11:56:10.969252+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:15.058688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T11:55:15.058688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:33.054204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"129'7766","reported_seq":7859,"reported_epoch":129,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T11:56:11.301778+0000","last_change":"2024-03-24T11:56:10.744179+0000","last_active":"2024-03-24T11:56:11.301778+0000","last_peered":"2024-03-24T11:56:11.301778+0000","last_clean":"2024-03-24T11:56:11.301778+0000","last_became_active":"2024-03-24T11:54:16.238176+0000","last_became_peered":"2024-03-24T11:54:16.238176+0000","last_unstale":"2024-03-24T11:56:11.301778+0000","last_undegraded":"2024-03-24T11:56:11.301778+0000","last_fullsized":"2024-03-24T11:56:11.301778+0000","mapping_epoch":97,"log_start":"109'3000","ondisk_log_start":"109'3000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":1599,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463863808,"num_objects":7078,"num_object_clones":0,"num_object_copies":21234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990773+0000","last_change":"2024-03-24T11:54:55.993898+0000","last_active":"2024-03-24T11:56:10.990773+0000","last_peered":"2024-03-24T11:56:10.990773+0000","last_clean":"2024-03-24T11:56:10.990773+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:56:10.990773+0000","last_undegraded":"2024-03-24T11:56:10.990773+0000","last_fullsized":"2024-03-24T11:56:10.990773+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:55.993770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:55.993770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008189+0000","last_change":"2024-03-24T11:55:02.950666+0000","last_active":"2024-03-24T11:56:11.008189+0000","last_peered":"2024-03-24T11:56:11.008189+0000","last_clean":"2024-03-24T11:56:11.008189+0000","last_became_active":"2024-03-24T11:52:15.142974+0000","last_became_peered":"2024-03-24T11:52:15.142974+0000","last_unstale":"2024-03-24T11:56:11.008189+0000","last_undegraded":"2024-03-24T11:56:11.008189+0000","last_fullsized":"2024-03-24T11:56:11.008189+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:55:02.950528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:02.950528+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:22.293270+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":180,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979232+0000","last_change":"2024-03-24T11:54:57.356969+0000","last_active":"2024-03-24T11:56:10.979232+0000","last_peered":"2024-03-24T11:56:10.979232+0000","last_clean":"2024-03-24T11:56:10.979232+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:56:10.979232+0000","last_undegraded":"2024-03-24T11:56:10.979232+0000","last_fullsized":"2024-03-24T11:56:10.979232+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:54:57.356797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:57.356797+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:27.206630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":142,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990662+0000","last_change":"2024-03-24T11:56:10.062814+0000","last_active":"2024-03-24T11:56:10.990662+0000","last_peered":"2024-03-24T11:56:10.990662+0000","last_clean":"2024-03-24T11:56:10.990662+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:56:10.990662+0000","last_undegraded":"2024-03-24T11:56:10.990662+0000","last_fullsized":"2024-03-24T11:56:10.990662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.062643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.062643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:31.233410+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"129'7759","reported_seq":7916,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.287365+0000","last_change":"2024-03-24T11:56:06.946412+0000","last_active":"2024-03-24T11:56:11.287365+0000","last_peered":"2024-03-24T11:56:11.287365+0000","last_clean":"2024-03-24T11:56:11.287365+0000","last_became_active":"2024-03-24T11:54:16.235994+0000","last_became_peered":"2024-03-24T11:54:16.235994+0000","last_unstale":"2024-03-24T11:56:11.287365+0000","last_undegraded":"2024-03-24T11:56:11.287365+0000","last_fullsized":"2024-03-24T11:56:11.287365+0000","mapping_epoch":97,"log_start":"110'3000","ondisk_log_start":"110'3000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7491","last_scrub_stamp":"2024-03-24T11:56:06.946102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.946102+0000","objects_scrubbed":7377,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:18.268280+0000","scrub_duration":3152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463798272,"num_objects":7077,"num_object_clones":0,"num_object_copies":21231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7759,"num_write_kb":474752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978808+0000","last_change":"2024-03-24T11:55:06.364864+0000","last_active":"2024-03-24T11:56:10.978808+0000","last_peered":"2024-03-24T11:56:10.978808+0000","last_clean":"2024-03-24T11:56:10.978808+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:56:10.978808+0000","last_undegraded":"2024-03-24T11:56:10.978808+0000","last_fullsized":"2024-03-24T11:56:10.978808+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.364777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.364777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:20.281583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.969648+0000","last_change":"2024-03-24T11:54:49.113470+0000","last_active":"2024-03-24T11:56:10.969648+0000","last_peered":"2024-03-24T11:56:10.969648+0000","last_clean":"2024-03-24T11:56:10.969648+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:56:10.969648+0000","last_undegraded":"2024-03-24T11:56:10.969648+0000","last_fullsized":"2024-03-24T11:56:10.969648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:49.113375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:54:49.113375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":174,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990504+0000","last_change":"2024-03-24T11:55:05.889871+0000","last_active":"2024-03-24T11:56:10.990504+0000","last_peered":"2024-03-24T11:56:10.990504+0000","last_clean":"2024-03-24T11:56:10.990504+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:56:10.990504+0000","last_undegraded":"2024-03-24T11:56:10.990504+0000","last_fullsized":"2024-03-24T11:56:10.990504+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:55:05.889771+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:55:05.889771+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:29.514305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":192,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979145+0000","last_change":"2024-03-24T11:54:13.589433+0000","last_active":"2024-03-24T11:56:10.979145+0000","last_peered":"2024-03-24T11:56:10.979145+0000","last_clean":"2024-03-24T11:56:10.979145+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:56:10.979145+0000","last_undegraded":"2024-03-24T11:56:10.979145+0000","last_fullsized":"2024-03-24T11:56:10.979145+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:54:13.589346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:13.589346+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"129'7861","reported_seq":7936,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.148019+0000","last_change":"2024-03-24T11:54:16.242004+0000","last_active":"2024-03-24T11:56:11.148019+0000","last_peered":"2024-03-24T11:56:11.148019+0000","last_clean":"2024-03-24T11:56:11.148019+0000","last_became_active":"2024-03-24T11:54:16.241724+0000","last_became_peered":"2024-03-24T11:54:16.241724+0000","last_unstale":"2024-03-24T11:56:11.148019+0000","last_undegraded":"2024-03-24T11:56:11.148019+0000","last_fullsized":"2024-03-24T11:56:11.148019+0000","mapping_epoch":97,"log_start":"107'1700","ondisk_log_start":"107'1700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470286360,"num_objects":7177,"num_object_clones":0,"num_object_copies":21531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7861,"num_write_kb":481153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.979597+0000","last_change":"2024-03-24T11:55:55.660561+0000","last_active":"2024-03-24T11:56:10.979597+0000","last_peered":"2024-03-24T11:56:10.979597+0000","last_clean":"2024-03-24T11:56:10.979597+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:56:10.979597+0000","last_undegraded":"2024-03-24T11:56:10.979597+0000","last_fullsized":"2024-03-24T11:56:10.979597+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:55.660368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:55.660368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:14.298173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":142,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980889+0000","last_change":"2024-03-24T11:55:03.730546+0000","last_active":"2024-03-24T11:56:10.980889+0000","last_peered":"2024-03-24T11:56:10.980889+0000","last_clean":"2024-03-24T11:56:10.980889+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:56:10.980889+0000","last_undegraded":"2024-03-24T11:56:10.980889+0000","last_fullsized":"2024-03-24T11:56:10.980889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:03.730436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:03.730436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:33.413314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978602+0000","last_change":"2024-03-24T11:54:52.448522+0000","last_active":"2024-03-24T11:56:10.978602+0000","last_peered":"2024-03-24T11:56:10.978602+0000","last_clean":"2024-03-24T11:56:10.978602+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:56:10.978602+0000","last_undegraded":"2024-03-24T11:56:10.978602+0000","last_fullsized":"2024-03-24T11:56:10.978602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:52.448412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.448412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":165,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991383+0000","last_change":"2024-03-24T11:56:06.009518+0000","last_active":"2024-03-24T11:56:10.991383+0000","last_peered":"2024-03-24T11:56:10.991383+0000","last_clean":"2024-03-24T11:56:10.991383+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:56:10.991383+0000","last_undegraded":"2024-03-24T11:56:10.991383+0000","last_fullsized":"2024-03-24T11:56:10.991383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:06.009394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.009394+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:12.022411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"129'7855","reported_seq":7929,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.140972+0000","last_change":"2024-03-24T11:54:16.237411+0000","last_active":"2024-03-24T11:56:11.140972+0000","last_peered":"2024-03-24T11:56:11.140972+0000","last_clean":"2024-03-24T11:56:11.140972+0000","last_became_active":"2024-03-24T11:54:16.237102+0000","last_became_peered":"2024-03-24T11:54:16.237102+0000","last_unstale":"2024-03-24T11:56:11.140972+0000","last_undegraded":"2024-03-24T11:56:11.140972+0000","last_fullsized":"2024-03-24T11:56:11.140972+0000","mapping_epoch":97,"log_start":"107'1700","ondisk_log_start":"107'1700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474021888,"num_objects":7233,"num_object_clones":0,"num_object_copies":21699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":482816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980509+0000","last_change":"2024-03-24T11:55:11.673583+0000","last_active":"2024-03-24T11:56:10.980509+0000","last_peered":"2024-03-24T11:56:10.980509+0000","last_clean":"2024-03-24T11:56:10.980509+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:56:10.980509+0000","last_undegraded":"2024-03-24T11:56:10.980509+0000","last_fullsized":"2024-03-24T11:56:10.980509+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.673492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.673492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:16.257227+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":201,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991273+0000","last_change":"2024-03-24T11:55:02.895074+0000","last_active":"2024-03-24T11:56:10.991273+0000","last_peered":"2024-03-24T11:56:10.991273+0000","last_clean":"2024-03-24T11:56:10.991273+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:56:10.991273+0000","last_undegraded":"2024-03-24T11:56:10.991273+0000","last_fullsized":"2024-03-24T11:56:10.991273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:02.894942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:02.894942+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:25.378002+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.005608+0000","last_change":"2024-03-24T11:55:58.143201+0000","last_active":"2024-03-24T11:56:11.005608+0000","last_peered":"2024-03-24T11:56:11.005608+0000","last_clean":"2024-03-24T11:56:11.005608+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:56:11.005608+0000","last_undegraded":"2024-03-24T11:56:11.005608+0000","last_fullsized":"2024-03-24T11:56:11.005608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:58.143072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:58.143072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:02.908198+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":172,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008629+0000","last_change":"2024-03-24T11:54:59.949653+0000","last_active":"2024-03-24T11:56:11.008629+0000","last_peered":"2024-03-24T11:56:11.008629+0000","last_clean":"2024-03-24T11:56:11.008629+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:56:11.008629+0000","last_undegraded":"2024-03-24T11:56:11.008629+0000","last_fullsized":"2024-03-24T11:56:11.008629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:54:59.949501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:59.949501+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:16.451264+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"129'7851","reported_seq":7925,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.356178+0000","last_change":"2024-03-24T11:54:16.239589+0000","last_active":"2024-03-24T11:56:11.356178+0000","last_peered":"2024-03-24T11:56:11.356178+0000","last_clean":"2024-03-24T11:56:11.356178+0000","last_became_active":"2024-03-24T11:54:16.239274+0000","last_became_peered":"2024-03-24T11:54:16.239274+0000","last_unstale":"2024-03-24T11:56:11.356178+0000","last_undegraded":"2024-03-24T11:56:11.356178+0000","last_fullsized":"2024-03-24T11:56:11.356178+0000","mapping_epoch":97,"log_start":"108'2300","ondisk_log_start":"108'2300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5551,"log_dups_size":0,"ondisk_log_size":5551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467468288,"num_objects":7133,"num_object_clones":0,"num_object_copies":21399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991165+0000","last_change":"2024-03-24T11:55:01.911844+0000","last_active":"2024-03-24T11:56:10.991165+0000","last_peered":"2024-03-24T11:56:10.991165+0000","last_clean":"2024-03-24T11:56:10.991165+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:56:10.991165+0000","last_undegraded":"2024-03-24T11:56:10.991165+0000","last_fullsized":"2024-03-24T11:56:10.991165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:55:01.911663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.911663+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":200,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.997411+0000","last_change":"2024-03-24T11:54:58.454553+0000","last_active":"2024-03-24T11:56:10.997411+0000","last_peered":"2024-03-24T11:56:10.997411+0000","last_clean":"2024-03-24T11:56:10.997411+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:56:10.997411+0000","last_undegraded":"2024-03-24T11:56:10.997411+0000","last_fullsized":"2024-03-24T11:56:10.997411+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:54:58.454433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.454433+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.005670+0000","last_change":"2024-03-24T11:56:00.196860+0000","last_active":"2024-03-24T11:56:11.005670+0000","last_peered":"2024-03-24T11:56:11.005670+0000","last_clean":"2024-03-24T11:56:11.005670+0000","last_became_active":"2024-03-24T11:52:15.134326+0000","last_became_peered":"2024-03-24T11:52:15.134326+0000","last_unstale":"2024-03-24T11:56:11.005670+0000","last_undegraded":"2024-03-24T11:56:11.005670+0000","last_fullsized":"2024-03-24T11:56:11.005670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:00.196663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_clean_scrub_stamp":"2024-03-24T11:56:00.196663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:08.616618+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":161,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.990828+0000","last_change":"2024-03-24T11:55:18.793369+0000","last_active":"2024-03-24T11:56:10.990828+0000","last_peered":"2024-03-24T11:56:10.990828+0000","last_clean":"2024-03-24T11:56:10.990828+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:56:10.990828+0000","last_undegraded":"2024-03-24T11:56:10.990828+0000","last_fullsized":"2024-03-24T11:56:10.990828+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:55:18.793281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.793281+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:23.906014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"129'7882","reported_seq":8043,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.275821+0000","last_change":"2024-03-24T11:56:06.470793+0000","last_active":"2024-03-24T11:56:11.275821+0000","last_peered":"2024-03-24T11:56:11.275821+0000","last_clean":"2024-03-24T11:56:11.275821+0000","last_became_active":"2024-03-24T11:54:16.236885+0000","last_became_peered":"2024-03-24T11:54:16.236885+0000","last_unstale":"2024-03-24T11:56:11.275821+0000","last_undegraded":"2024-03-24T11:56:11.275821+0000","last_fullsized":"2024-03-24T11:56:11.275821+0000","mapping_epoch":97,"log_start":"106'1300","ondisk_log_start":"106'1300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7612","last_scrub_stamp":"2024-03-24T11:56:06.470609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.470609+0000","objects_scrubbed":7541,"log_size":6582,"log_dups_size":0,"ondisk_log_size":6582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:21.968168+0000","scrub_duration":2905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476446720,"num_objects":7270,"num_object_clones":0,"num_object_copies":21810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7882,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.992876+0000","last_change":"2024-03-24T11:55:20.629992+0000","last_active":"2024-03-24T11:56:10.992876+0000","last_peered":"2024-03-24T11:56:10.992876+0000","last_clean":"2024-03-24T11:56:10.992876+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:56:10.992876+0000","last_undegraded":"2024-03-24T11:56:10.992876+0000","last_fullsized":"2024-03-24T11:56:10.992876+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.629800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.629800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:31.421523+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.009093+0000","last_change":"2024-03-24T11:56:06.310590+0000","last_active":"2024-03-24T11:56:11.009093+0000","last_peered":"2024-03-24T11:56:11.009093+0000","last_clean":"2024-03-24T11:56:11.009093+0000","last_became_active":"2024-03-24T11:52:19.387115+0000","last_became_peered":"2024-03-24T11:52:19.387115+0000","last_unstale":"2024-03-24T11:56:11.009093+0000","last_undegraded":"2024-03-24T11:56:11.009093+0000","last_fullsized":"2024-03-24T11:56:11.009093+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:06.310357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.310357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:29.666932+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":134,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.999274+0000","last_change":"2024-03-24T11:54:59.405029+0000","last_active":"2024-03-24T11:56:10.999274+0000","last_peered":"2024-03-24T11:56:10.999274+0000","last_clean":"2024-03-24T11:56:10.999274+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:56:10.999274+0000","last_undegraded":"2024-03-24T11:56:10.999274+0000","last_fullsized":"2024-03-24T11:56:10.999274+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:54:59.404740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:59.404740+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.460597+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"116'25","reported_seq":182,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.991154+0000","last_change":"2024-03-24T11:55:19.864480+0000","last_active":"2024-03-24T11:56:10.991154+0000","last_peered":"2024-03-24T11:56:10.991154+0000","last_clean":"2024-03-24T11:56:10.991154+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:56:10.991154+0000","last_undegraded":"2024-03-24T11:56:10.991154+0000","last_fullsized":"2024-03-24T11:56:10.991154+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'25","last_scrub_stamp":"2024-03-24T11:55:19.864301+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T11:55:19.864301+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.864301+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:45.201869+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"129'7928","reported_seq":8002,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.309612+0000","last_change":"2024-03-24T11:54:16.237670+0000","last_active":"2024-03-24T11:56:11.309612+0000","last_peered":"2024-03-24T11:56:11.309612+0000","last_clean":"2024-03-24T11:56:11.309612+0000","last_became_active":"2024-03-24T11:54:16.237374+0000","last_became_peered":"2024-03-24T11:54:16.237374+0000","last_unstale":"2024-03-24T11:56:11.309612+0000","last_undegraded":"2024-03-24T11:56:11.309612+0000","last_fullsized":"2024-03-24T11:56:11.309612+0000","mapping_epoch":97,"log_start":"108'2400","ondisk_log_start":"108'2400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476315648,"num_objects":7268,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7928,"num_write_kb":486272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978300+0000","last_change":"2024-03-24T11:55:18.299756+0000","last_active":"2024-03-24T11:56:10.978300+0000","last_peered":"2024-03-24T11:56:10.978300+0000","last_clean":"2024-03-24T11:56:10.978300+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:56:10.978300+0000","last_undegraded":"2024-03-24T11:56:10.978300+0000","last_fullsized":"2024-03-24T11:56:10.978300+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.299637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:40.446628+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.999527+0000","last_change":"2024-03-24T11:55:07.488338+0000","last_active":"2024-03-24T11:56:10.999527+0000","last_peered":"2024-03-24T11:56:10.999527+0000","last_clean":"2024-03-24T11:56:10.999527+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:56:10.999527+0000","last_undegraded":"2024-03-24T11:56:10.999527+0000","last_fullsized":"2024-03-24T11:56:10.999527+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:07.488042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:07.488042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:22.154511+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.992426+0000","last_change":"2024-03-24T11:54:51.317621+0000","last_active":"2024-03-24T11:56:10.992426+0000","last_peered":"2024-03-24T11:56:10.992426+0000","last_clean":"2024-03-24T11:56:10.992426+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:56:10.992426+0000","last_undegraded":"2024-03-24T11:56:10.992426+0000","last_fullsized":"2024-03-24T11:56:10.992426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:51.317368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:51.317368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:13.404690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":177,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.969054+0000","last_change":"2024-03-24T11:55:19.105549+0000","last_active":"2024-03-24T11:56:10.969054+0000","last_peered":"2024-03-24T11:56:10.969054+0000","last_clean":"2024-03-24T11:56:10.969054+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:56:10.969054+0000","last_undegraded":"2024-03-24T11:56:10.969054+0000","last_fullsized":"2024-03-24T11:56:10.969054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:55:19.105397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.105397+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:27.881078+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"129'7921","reported_seq":8077,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.251894+0000","last_change":"2024-03-24T11:56:05.111342+0000","last_active":"2024-03-24T11:56:11.251894+0000","last_peered":"2024-03-24T11:56:11.251894+0000","last_clean":"2024-03-24T11:56:11.251894+0000","last_became_active":"2024-03-24T11:54:16.237483+0000","last_became_peered":"2024-03-24T11:54:16.237483+0000","last_unstale":"2024-03-24T11:56:11.251894+0000","last_undegraded":"2024-03-24T11:56:11.251894+0000","last_fullsized":"2024-03-24T11:56:11.251894+0000","mapping_epoch":97,"log_start":"106'1300","ondisk_log_start":"106'1300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7549","last_scrub_stamp":"2024-03-24T11:56:05.111031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.111031+0000","objects_scrubbed":7494,"log_size":6621,"log_dups_size":0,"ondisk_log_size":6621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:05.209536+0000","scrub_duration":2574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475201536,"num_objects":7251,"num_object_clones":0,"num_object_copies":21753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7921,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.974337+0000","last_change":"2024-03-24T11:55:01.085505+0000","last_active":"2024-03-24T11:56:10.974337+0000","last_peered":"2024-03-24T11:56:10.974337+0000","last_clean":"2024-03-24T11:56:10.974337+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:56:10.974337+0000","last_undegraded":"2024-03-24T11:56:10.974337+0000","last_fullsized":"2024-03-24T11:56:10.974337+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.085396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.085396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.992190+0000","last_change":"2024-03-24T11:55:22.642429+0000","last_active":"2024-03-24T11:56:10.992190+0000","last_peered":"2024-03-24T11:56:10.992190+0000","last_clean":"2024-03-24T11:56:10.992190+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:56:10.992190+0000","last_undegraded":"2024-03-24T11:56:10.992190+0000","last_fullsized":"2024-03-24T11:56:10.992190+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:22.642148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.642148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:24.428410+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:11.008405+0000","last_change":"2024-03-24T11:55:06.933106+0000","last_active":"2024-03-24T11:56:11.008405+0000","last_peered":"2024-03-24T11:56:11.008405+0000","last_clean":"2024-03-24T11:56:11.008405+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:56:11.008405+0000","last_undegraded":"2024-03-24T11:56:11.008405+0000","last_fullsized":"2024-03-24T11:56:11.008405+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.932966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.932966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:17.361036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.974226+0000","last_change":"2024-03-24T11:56:02.340688+0000","last_active":"2024-03-24T11:56:10.974226+0000","last_peered":"2024-03-24T11:56:10.974226+0000","last_clean":"2024-03-24T11:56:10.974226+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:56:10.974226+0000","last_undegraded":"2024-03-24T11:56:10.974226+0000","last_fullsized":"2024-03-24T11:56:10.974226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:02.340507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:02.340507+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:06.085699+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"129'7952","reported_seq":8092,"reported_epoch":129,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T11:56:11.232769+0000","last_change":"2024-03-24T11:56:09.220906+0000","last_active":"2024-03-24T11:56:11.232769+0000","last_peered":"2024-03-24T11:56:11.232769+0000","last_clean":"2024-03-24T11:56:11.232769+0000","last_became_active":"2024-03-24T11:54:16.239338+0000","last_became_peered":"2024-03-24T11:54:16.239338+0000","last_unstale":"2024-03-24T11:56:11.232769+0000","last_undegraded":"2024-03-24T11:56:11.232769+0000","last_fullsized":"2024-03-24T11:56:11.232769+0000","mapping_epoch":97,"log_start":"109'2800","ondisk_log_start":"109'2800","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":6199,"log_size":5152,"log_dups_size":0,"ondisk_log_size":5152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472776704,"num_objects":7214,"num_object_clones":0,"num_object_copies":21642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7952,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.980353+0000","last_change":"2024-03-24T11:55:17.618704+0000","last_active":"2024-03-24T11:56:10.980353+0000","last_peered":"2024-03-24T11:56:10.980353+0000","last_clean":"2024-03-24T11:56:10.980353+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:56:10.980353+0000","last_undegraded":"2024-03-24T11:56:10.980353+0000","last_fullsized":"2024-03-24T11:56:10.980353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:17.618606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T11:55:17.618606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:42.294065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.992299+0000","last_change":"2024-03-24T11:55:04.412885+0000","last_active":"2024-03-24T11:56:10.992299+0000","last_peered":"2024-03-24T11:56:10.992299+0000","last_clean":"2024-03-24T11:56:10.992299+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:56:10.992299+0000","last_undegraded":"2024-03-24T11:56:10.992299+0000","last_fullsized":"2024-03-24T11:56:10.992299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:04.412611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:04.412611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:19.594594+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":171,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.981369+0000","last_change":"2024-03-24T11:55:19.645322+0000","last_active":"2024-03-24T11:56:10.981369+0000","last_peered":"2024-03-24T11:56:10.981369+0000","last_clean":"2024-03-24T11:56:10.981369+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:56:10.981369+0000","last_undegraded":"2024-03-24T11:56:10.981369+0000","last_fullsized":"2024-03-24T11:56:10.981369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:55:19.645176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.645176+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:39.822493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":199,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-03-24T11:56:10.978324+0000","last_change":"2024-03-24T11:55:02.415044+0000","last_active":"2024-03-24T11:56:10.978324+0000","last_peered":"2024-03-24T11:56:10.978324+0000","last_clean":"2024-03-24T11:56:10.978324+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:56:10.978324+0000","last_undegraded":"2024-03-24T11:56:10.978324+0000","last_fullsized":"2024-03-24T11:56:10.978324+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:02.414931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:02.414931+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:30.803168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7572881432,"num_objects":115554,"num_object_clones":0,"num_object_copies":346662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126356,"num_write_kb":7741057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22722523136,"data_stored":22722510920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89056,"ondisk_log_size":89056,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":225,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":444,"ondisk_log_size":444,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005107,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2383372,"kb_used_data":2311936,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91361780,"statfs":{"total":95995035648,"available":93554462720,"internally_reserved":0,"allocated":2367422464,"data_stored":2367029344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332154,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2397964,"kb_used_data":2326528,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91347188,"statfs":{"total":95995035648,"available":93539520512,"internally_reserved":0,"allocated":2382364672,"data_stored":2381971394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2858964,"kb_used_data":2791688,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":90886188,"statfs":{"total":95995035648,"available":93067456512,"internally_reserved":0,"allocated":2858688512,"data_stored":2858286077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3810396,"kb_used_data":3702736,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":89934756,"statfs":{"total":95995035648,"available":92093190144,"internally_reserved":0,"allocated":3791601664,"data_stored":3791197904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2848396,"kb_used_data":2785024,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":90896756,"statfs":{"total":95995035648,"available":93078278144,"internally_reserved":0,"allocated":2851864576,"data_stored":2851470729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117064,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4271260,"kb_used_data":4167764,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":89473892,"statfs":{"total":95995035648,"available":91621265408,"internally_reserved":0,"allocated":4267790336,"data_stored":4267376439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2388452,"kb_used_data":2312852,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91356700,"statfs":{"total":95995035648,"available":93549260800,"internally_reserved":0,"allocated":2368360448,"data_stored":2367952589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659198,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1954892,"kb_used_data":1879232,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91790260,"statfs":{"total":95995035648,"available":93993226240,"internally_reserved":0,"allocated":1924333568,"data_stored":1923939448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3783659520,"data_stored":3783655448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2835746816,"data_stored":2835742744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2838429696,"data_stored":2838429696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4250992640,"data_stored":4250992640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2365390848,"data_stored":2365390848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1912471552,"data_stored":1912471552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2376863744,"data_stored":2376859672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2358968320,"data_stored":2358968320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:56:17.455 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-24T11:56:17.455 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:56:17.455 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-24T11:56:17.455 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T11:56:17.455 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T11:56:17.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-24T11:56:17.473 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:17.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:17.576 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:17.616 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:17.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:17.639 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:56:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:56:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:17.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:17.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:17.842 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:17.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:17.923 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:17.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:18.024 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:18.058 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:18.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:18.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:18.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[42243]: from='client.14964 v1:172.21.15.64:0/1687719211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[42243]: 7.d scrub starts 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[42243]: 4.f deep-scrub starts 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[42243]: 4.f deep-scrub ok 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[30223]: from='client.14964 v1:172.21.15.64:0/1687719211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[30223]: 7.d scrub starts 2024-03-24T11:56:18.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:18.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:18.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:18.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[30223]: 4.f deep-scrub starts 2024-03-24T11:56:18.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:18 smithi064 ceph-mon[30223]: 4.f deep-scrub ok 2024-03-24T11:56:18.308 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:18.333 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:18.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:18 smithi193 ceph-mon[33593]: from='client.14964 v1:172.21.15.64:0/1687719211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:18 smithi193 ceph-mon[33593]: 7.d scrub starts 2024-03-24T11:56:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:18 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:18 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:18 smithi193 ceph-mon[33593]: 4.f deep-scrub starts 2024-03-24T11:56:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:18 smithi193 ceph-mon[33593]: 4.f deep-scrub ok 2024-03-24T11:56:18.434 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:18.466 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:18.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:18.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:18.607 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:18.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:18.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:18.744 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:18.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:18.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:18.873 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:18.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:18.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:19.015 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:19.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:19.117 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:19.145 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:19.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:19.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:19.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:19.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:19 smithi064 ceph-mon[30223]: pgmap v153: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T11:56:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:19 smithi064 ceph-mon[42243]: pgmap v153: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T11:56:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:19 smithi193 ceph-mon[33593]: pgmap v153: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T11:56:19.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:19.414 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:19.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:19.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:19.553 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:19.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:19.654 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:19.684 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:19.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:19.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:19.815 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:19.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:19.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:19.948 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:19.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:20.050 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:20.083 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:20.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:20.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:20.225 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:20.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:20.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[42243]: 7.d scrub ok 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[42243]: 4.4 scrub starts 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[42243]: 4.4 scrub ok 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[30223]: 7.d scrub ok 2024-03-24T11:56:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[30223]: 4.4 scrub starts 2024-03-24T11:56:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[30223]: 4.4 scrub ok 2024-03-24T11:56:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T11:56:20.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:20 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T11:56:20.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:20 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T11:56:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:20 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T11:56:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:20 smithi193 ceph-mon[33593]: 7.d scrub ok 2024-03-24T11:56:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:20 smithi193 ceph-mon[33593]: 4.4 scrub starts 2024-03-24T11:56:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:20 smithi193 ceph-mon[33593]: 4.4 scrub ok 2024-03-24T11:56:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:20 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T11:56:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:20 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T11:56:20.356 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:20.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:20.458 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:20.493 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:20.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:20.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:20.624 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:20.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:20.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:20.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:20.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:20.895 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:20.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:20.997 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:21.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:21.030 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:21.110 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:21.128 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:21.157 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:21.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[30223]: pgmap v154: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:56:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T11:56:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T11:56:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T11:56:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T11:56:21.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[42243]: pgmap v154: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:56:21.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T11:56:21.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T11:56:21.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T11:56:21.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:21 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T11:56:21.260 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:21.292 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:21.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:21 smithi193 ceph-mon[33593]: pgmap v154: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:56:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:21 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T11:56:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:21 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T11:56:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:21 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T11:56:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:21 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T11:56:21.394 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:21.418 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:21.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:21.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:21.555 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:21.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:21.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:21.696 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:21.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:21.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:21.847 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:21.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:21.866 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:56:21.866 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T11:56:21.866 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:56:21.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:21.980 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:21.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:22.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:22.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:22 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T11:56:22.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:22 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T11:56:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:22 smithi193 ceph-mon[33593]: 2.8 deep-scrub starts 2024-03-24T11:56:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:22 smithi193 ceph-mon[33593]: 2.8 deep-scrub ok 2024-03-24T11:56:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:22 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T11:56:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:22 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T11:56:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:22 smithi193 ceph-mon[33593]: pgmap v155: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 967 op/s 2024-03-24T11:56:22.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:22 smithi193 ceph-mon[33593]: 7.0 scrub starts 2024-03-24T11:56:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:22.133 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:22.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:22.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T11:56:22.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T11:56:22.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[42243]: 2.8 deep-scrub starts 2024-03-24T11:56:22.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[42243]: 2.8 deep-scrub ok 2024-03-24T11:56:22.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T11:56:22.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T11:56:22.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[42243]: pgmap v155: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 967 op/s 2024-03-24T11:56:22.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[42243]: 7.0 scrub starts 2024-03-24T11:56:22.184 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T11:56:22.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T11:56:22.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[30223]: 2.8 deep-scrub starts 2024-03-24T11:56:22.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[30223]: 2.8 deep-scrub ok 2024-03-24T11:56:22.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T11:56:22.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T11:56:22.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[30223]: pgmap v155: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 967 op/s 2024-03-24T11:56:22.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:22 smithi064 ceph-mon[30223]: 7.0 scrub starts 2024-03-24T11:56:22.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:22.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:22.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:22.365 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:22.393 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:22.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:22.490 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:56:22.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:22.535 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:56:22.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:22.536 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:22.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:22.584 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:56:22.631 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:56:22.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:22.671 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:56:22.680 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:22.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:22.711 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:56:22.750 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:56:22.782 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:22.787 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:56:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:56:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:56:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:56:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:56:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:56:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:56:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:56:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:56:22.812 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:22.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:22.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:22.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:22.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:23.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:23.098 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:23.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:23.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:23.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:23.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:23.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T11:56:23.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T11:56:23.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T11:56:23.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T11:56:23.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1222827554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T11:56:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T11:56:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T11:56:23.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T11:56:23.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:23 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1222827554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:23 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T11:56:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:23 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T11:56:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:23 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T11:56:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:23 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T11:56:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:23 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1222827554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:23.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:23.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:23.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:23.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:23.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:23.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:23.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:23.668 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:23.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:23.772 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:23.799 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:23.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:23.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:23.927 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:23.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:24.029 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:24.045 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T11:56:24.045 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T11:56:24.045 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[30223]: 7.0 scrub ok 2024-03-24T11:56:24.056 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:24.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:24.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:24.211 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:24.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:24.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[30223]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 912 B/s rd, 0 B/s wr, 831 op/s 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[30223]: 5.17 scrub starts 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[30223]: 5.17 scrub ok 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[42243]: 7.0 scrub ok 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[42243]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 912 B/s rd, 0 B/s wr, 831 op/s 2024-03-24T11:56:24.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T11:56:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T11:56:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[42243]: 5.17 scrub starts 2024-03-24T11:56:24.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:24 smithi064 ceph-mon[42243]: 5.17 scrub ok 2024-03-24T11:56:24.312 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:24 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T11:56:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:24 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T11:56:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:24 smithi193 ceph-mon[33593]: 7.0 scrub ok 2024-03-24T11:56:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:24 smithi193 ceph-mon[33593]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 912 B/s rd, 0 B/s wr, 831 op/s 2024-03-24T11:56:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:24 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T11:56:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:24 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T11:56:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:24 smithi193 ceph-mon[33593]: 5.17 scrub starts 2024-03-24T11:56:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:24 smithi193 ceph-mon[33593]: 5.17 scrub ok 2024-03-24T11:56:24.340 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:24.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:24.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:24.481 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:24.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:24.583 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:24.613 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:24.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:24.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:24.757 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:24.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:24.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:24.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:24.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:24.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:25.041 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:25.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:25.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:25.179 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:25.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:25.281 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T11:56:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T11:56:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 2.1d deep-scrub starts 2024-03-24T11:56:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 2.1d deep-scrub ok 2024-03-24T11:56:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 4.1b deep-scrub starts 2024-03-24T11:56:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 4.1b deep-scrub ok 2024-03-24T11:56:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 6.13 scrub starts 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[30223]: 6.13 scrub ok 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 2.1d deep-scrub starts 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 2.1d deep-scrub ok 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 4.1b deep-scrub starts 2024-03-24T11:56:25.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 4.1b deep-scrub ok 2024-03-24T11:56:25.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T11:56:25.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T11:56:25.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T11:56:25.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T11:56:25.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 6.13 scrub starts 2024-03-24T11:56:25.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:25 smithi064 ceph-mon[42243]: 6.13 scrub ok 2024-03-24T11:56:25.311 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:25.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 2.1d deep-scrub starts 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 2.1d deep-scrub ok 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 4.1b deep-scrub starts 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 4.1b deep-scrub ok 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 6.13 scrub starts 2024-03-24T11:56:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:25 smithi193 ceph-mon[33593]: 6.13 scrub ok 2024-03-24T11:56:25.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:25.451 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:25.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:25.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:25.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:56:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:25.590 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:25.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:25.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:25.738 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:25.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:25.839 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:25.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:25.877 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:25.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:25.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:26.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:26.118 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:26.143 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:26.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:26.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:26.291 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:26.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: pgmap v157: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 777 op/s 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: 6.1d scrub starts 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: 6.1d scrub ok 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: 6.19 scrub starts 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: 6.19 scrub ok 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: 6.11 scrub starts 2024-03-24T11:56:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-mon[33593]: 6.11 scrub ok 2024-03-24T11:56:26.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: pgmap v157: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 777 op/s 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: 6.1d scrub starts 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: 6.1d scrub ok 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: 6.19 scrub starts 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: 6.19 scrub ok 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: 6.11 scrub starts 2024-03-24T11:56:26.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[30223]: 6.11 scrub ok 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: pgmap v157: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 777 op/s 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: 6.1d scrub starts 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: 6.1d scrub ok 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: 6.19 scrub starts 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: 6.19 scrub ok 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: 6.11 scrub starts 2024-03-24T11:56:26.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:26 smithi064 ceph-mon[42243]: 6.11 scrub ok 2024-03-24T11:56:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:26.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:26.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:26.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:26.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:26.714 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:26.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:26.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:26.863 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:26.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:26.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:26.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:27.006 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:27.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:27.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:27.146 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:27.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:27.248 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:27.276 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:27.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:27.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:56:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:56:27.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T11:56:27.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T11:56:27.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T11:56:27.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T11:56:27.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T11:56:27.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T11:56:27.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T11:56:27.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T11:56:27.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:27 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:27.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T11:56:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T11:56:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:27.414 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:27.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:27.515 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:27.544 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:27.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:27.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:27.656 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:56:27.656 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:56:27.656 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-24T11:56:27.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T11:56:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:27.682 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:27.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:27.703 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:56:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:56:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:27.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:27.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:27.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:27.970 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:27.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:28.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:28.097 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:28.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T11:56:28.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:28.228 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:28.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:28.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:28.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T11:56:28.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T11:56:28.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: 2.11 deep-scrub starts 2024-03-24T11:56:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: 2.11 deep-scrub ok 2024-03-24T11:56:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1519372223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:56:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: from='client.34835 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:56:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:56:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T11:56:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T11:56:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T11:56:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:28 smithi193 ceph-mon[33593]: 6.1f scrub ok 2024-03-24T11:56:28.367 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:28.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:28.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:28.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:28.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:28.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T11:56:28.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T11:56:28.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: 2.11 deep-scrub starts 2024-03-24T11:56:28.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: 2.11 deep-scrub ok 2024-03-24T11:56:28.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1519372223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:56:28.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: from='client.34835 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[30223]: 6.1f scrub ok 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: 2.11 deep-scrub starts 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: 2.11 deep-scrub ok 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1519372223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: from='client.34835 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:56:28.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:56:28.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T11:56:28.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T11:56:28.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T11:56:28.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:28 smithi064 ceph-mon[42243]: 6.1f scrub ok 2024-03-24T11:56:28.598 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:28.624 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:28.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:28.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:28.753 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:28.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:28.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:28.908 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:28.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:29.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:29.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:29.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:29.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:29.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[30223]: 6.0 scrub starts 2024-03-24T11:56:29.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[30223]: 6.0 scrub ok 2024-03-24T11:56:29.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[30223]: 4.c deep-scrub starts 2024-03-24T11:56:29.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[30223]: 4.c deep-scrub ok 2024-03-24T11:56:29.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:29.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[30223]: from='client.34835 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:56:29.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[30223]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:56:29.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[42243]: 6.0 scrub starts 2024-03-24T11:56:29.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[42243]: 6.0 scrub ok 2024-03-24T11:56:29.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[42243]: 4.c deep-scrub starts 2024-03-24T11:56:29.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[42243]: 4.c deep-scrub ok 2024-03-24T11:56:29.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:29.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[42243]: from='client.34835 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:56:29.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:29 smithi064 ceph-mon[42243]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:56:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:29.169 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:29.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:29.271 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:29.299 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:29.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:29 smithi193 ceph-mon[33593]: 6.0 scrub starts 2024-03-24T11:56:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:29 smithi193 ceph-mon[33593]: 6.0 scrub ok 2024-03-24T11:56:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:29 smithi193 ceph-mon[33593]: 4.c deep-scrub starts 2024-03-24T11:56:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:29 smithi193 ceph-mon[33593]: 4.c deep-scrub ok 2024-03-24T11:56:29.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:29 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:29.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:29 smithi193 ceph-mon[33593]: from='client.34835 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:56:29.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:29 smithi193 ceph-mon[33593]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:56:29.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:29.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:29.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:29.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:29.584 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:29.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:29.687 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:29.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:29.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:29.868 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:29.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:29.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:30.027 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:30.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:30.110 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:56:30.110 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:56:30.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:30.179 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:30.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:30.280 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:30.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:30 smithi064 ceph-mon[30223]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-03-24T11:56:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:30 smithi064 ceph-mon[42243]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-03-24T11:56:30.308 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:30.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:30.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:30 smithi193 ceph-mon[33593]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-03-24T11:56:30.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:30.441 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:30.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:30.542 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:30.571 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:30.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:30.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:30.722 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:30.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:30.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:30.874 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:30.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:30.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:31.014 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:31.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:31.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:31.115 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:31.197 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T11:56:31.229 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:31.231 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:31.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:31.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:31.364 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:31.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:31.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:31.488 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:31.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:31.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:31.546 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:56:31.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:31.625 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:31.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:31.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:31.754 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:31.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:31.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:31.879 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:31 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:31.879 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:31 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:31.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:31 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:31.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:31 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:31.883 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:31.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:31.903 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:31 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:31.903 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:31 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:31.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:32.013 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:32.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:32.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:32.156 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:32.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:32.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:32.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:32.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:32.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:32.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:32.416 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:32.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:32.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:32.547 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:32.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:32.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:32.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:32.674 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:32.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:32.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:32.805 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:32.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:32.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:32.941 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:32.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:33.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:32 smithi064 ceph-mon[30223]: pgmap v161: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 877 op/s 2024-03-24T11:56:33.002 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:32 smithi064 ceph-mon[42243]: pgmap v161: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 877 op/s 2024-03-24T11:56:33.043 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:33.072 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:33.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:33.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:32 smithi193 ceph-mon[33593]: pgmap v161: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 877 op/s 2024-03-24T11:56:33.174 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:33.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:33.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:33.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:33.359 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:33.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:33.460 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:33.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:33.492 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:33.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:33.593 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:33.621 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:33.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:33.723 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:33.756 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:33.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:33.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:33.904 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:33.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:34.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:34.054 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:34.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:34.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:34.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:34.296 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:34.320 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:34.348 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:34.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:34.386 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:56:34.386 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:56:34.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:34.499 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:34.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:34.600 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:34.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:34.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:34.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:34.766 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:34.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:34.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:34 smithi064 ceph-mon[30223]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 866 op/s 2024-03-24T11:56:34.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:34 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:34.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:34 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:56:34.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:34 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:34 smithi064 ceph-mon[42243]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 866 op/s 2024-03-24T11:56:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:34 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:34 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:56:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:34 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:34.868 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:34.894 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:34.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:34.997 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:35.026 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:35.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:34 smithi193 ceph-mon[33593]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 866 op/s 2024-03-24T11:56:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:34 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:34 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:56:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:34 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:35.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:35.182 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:35.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:35.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:35.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T11:56:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:35.312 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:35.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:35.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:35.453 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:35.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:35.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:35.601 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:35.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:35.610 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:56:35.615 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:56:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 131, 2024-03-24T11:56:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T11:56:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-03-24T11:46:40.657065+0000", 2024-03-24T11:56:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-03-24T11:56:28.060165+0000", 2024-03-24T11:56:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-03-24T11:52:01.590830+0000", 2024-03-24T11:56:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-03-24T11:51:43.667453+0000", 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5801984, 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 18, 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T11:56:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 7, 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:49:56.369674+0000", 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T11:56:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-03-24T11:56:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "22", 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T11:56:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T11:56:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T11:56:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:56:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:56:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:56:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:56:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T11:56:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T11:56:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T11:56:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T11:56:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 7.8899998664855957, 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 7.8899998664855957, 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.37999999523162842, 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 3, 2024-03-24T11:56:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 3, 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".rgw.root", 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:13.871974+0000", 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T11:56:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T11:56:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "68", 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:56:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T11:56:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:56:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T11:56:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.5, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.5, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.5, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.5, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "datapool", 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:14.117018+0000", 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-03-24T11:56:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 3, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 3, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 3, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 3, 2024-03-24T11:56:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 3, 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "71", 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T11:56:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 2, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 71, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:56:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:56:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.6500000953674316, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.6500000953674316, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-24T11:56:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 4, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.log", 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:15.453744+0000", 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T11:56:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T11:56:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "70", 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T11:56:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:56:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T11:56:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.25, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.25, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.25, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.25, 2024-03-24T11:56:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 5, 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.control", 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:17.611004+0000", 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T11:56:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T11:56:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "72", 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T11:56:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T11:56:35.645 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T11:56:35.646 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T11:56:35.647 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T11:56:35.647 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T11:56:35.647 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T11:56:35.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.647 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T11:56:35.649 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T11:56:35.649 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.25, 2024-03-24T11:56:35.649 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.25, 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.25, 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.25, 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 6, 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.meta", 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:19.701416+0000", 2024-03-24T11:56:35.650 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T11:56:35.651 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:56:35.651 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:56:35.651 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:56:35.651 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:56:35.651 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:56:35.651 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:56:35.651 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:56:35.651 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:56:35.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:56:35.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:56:35.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:56:35.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:56:35.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:56:35.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:56:35.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:56:35.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:56:35.653 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "74", 2024-03-24T11:56:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T11:56:35.655 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T11:56:35.656 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_bias": 4 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.657 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.75, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.75, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 7, 2024-03-24T11:56:35.658 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "unique_pool_0", 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:54:14.779650+0000", 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:56:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 16, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 16, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 16, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 16, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 16, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "103", 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T11:56:35.660 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T11:56:35.661 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T11:56:35.662 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "rados": {} 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2, 2024-03-24T11:56:35.663 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2, 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d", 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T11:56:35.664 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 8, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 97, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-03-24T11:56:35.665 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/2675992571", 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6802/2675992571", 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6804/2675992571", 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6803/2675992571", 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9", 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T11:56:35.668 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 97, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T11:56:35.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.671 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6805/841667271", 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6806/841667271", 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6808/841667271", 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6807/841667271", 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T11:56:35.672 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b", 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 18, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 97, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-03-24T11:56:35.675 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/435147390", 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6810/435147390", 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6812/435147390", 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6811/435147390", 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.676 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479", 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 27, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 97, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6813/3355318038", 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6814/3355318038", 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6816/3355318038", 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6815/3355318038", 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b", 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 33, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 97, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T11:56:35.681 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6800", 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6801", 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T11:56:35.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6803", 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6802", 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6800/3851339176", 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6801/3851339176", 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3851339176", 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3851339176", 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T11:56:35.684 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6", 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 41, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 97, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T11:56:35.685 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6804", 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1625306337 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6805", 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1625306337 2024-03-24T11:56:35.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6807", 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1625306337 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6806", 2024-03-24T11:56:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1625306337 2024-03-24T11:56:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6804/1625306337", 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6805/1625306337", 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6807/1625306337", 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6806/1625306337", 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996", 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T11:56:35.689 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 51, 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 97, 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6808", 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T11:56:35.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6809", 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.691 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6811", 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6810", 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.692 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6808/2408237278", 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6809/2408237278", 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6811/2408237278", 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6810/2408237278", 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015", 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T11:56:35.693 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 61, 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 97, 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T11:56:35.694 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6812", 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6813", 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T11:56:35.695 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6815", 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6814", 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6812/2649989609", 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6813/2649989609", 2024-03-24T11:56:35.697 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6815/2649989609", 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6814/2649989609", 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T11:56:35.698 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T11:55:24.599632+0000", 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T11:55:44.834305+0000", 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T11:56:35.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T11:55:55.654778+0000", 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T11:56:35.700 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T11:55:19.345998+0000", 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T11:56:35.701 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T11:55:18.819821+0000", 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T11:56:35.702 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T11:56:11.616067+0000", 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T11:56:35.703 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T11:55:21.525981+0000", 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T11:56:09.668197+0000", 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.704 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [], 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [ 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.b", 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.4", 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 5 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.706 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.9", 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.1d", 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T11:56:35.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": { 2024-03-24T11:56:35.708 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/3538503390": "2024-03-25T11:52:55.117233+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1277626303": "2024-03-25T11:52:55.117233+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3407451558": "2024-03-25T11:52:55.117233+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3753895773": "2024-03-25T11:52:55.117233+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/842059688": "2024-03-25T11:47:36.035658+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1610565583": "2024-03-25T11:47:36.035658+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/2730170969": "2024-03-25T11:47:08.661692+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3257265142": "2024-03-25T11:47:08.661692+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1459540658": "2024-03-25T11:52:55.117233+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/629947599": "2024-03-25T11:47:08.661692+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/520335082": "2024-03-25T11:47:08.661692+0000", 2024-03-24T11:56:35.709 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3351953651": "2024-03-25T11:52:55.117233+0000", 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/4027721305": "2024-03-25T11:47:36.035658+0000", 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/4028269665": "2024-03-25T11:47:36.035658+0000" 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:56:35.710 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:56:35.711 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:56:35.712 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:35.737 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:35.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:35.838 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:35.868 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:35.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:35.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:36.010 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:36.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:36.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/797175143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:36.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1987730425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:56:36.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/797175143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:36.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1987730425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:56:36.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/797175143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:36.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1987730425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:56:36.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:36.143 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:36.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:36.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:36.278 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:36.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:36.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:36.408 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:36.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:36.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:36.542 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:36.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:36.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:36.683 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:36.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:36.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:36.827 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:36.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:36 smithi193 ceph-mon[33593]: pgmap v163: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 866 op/s 2024-03-24T11:56:36.929 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:36.959 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:36.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:36 smithi064 ceph-mon[30223]: pgmap v163: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 866 op/s 2024-03-24T11:56:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:36 smithi064 ceph-mon[42243]: pgmap v163: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 866 op/s 2024-03-24T11:56:37.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:37.097 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:37.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:37.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:37.230 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:37.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:37.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:56:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:56:37.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:37.363 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:37.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:37.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:37.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:37.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:37.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:37.642 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:37.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:37.703 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:56:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:56:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:37.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:37.787 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:37.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:37.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:37.892 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:37 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:37.916 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:37.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:38.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:37 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:37 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:38.055 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:38.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:38.060 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:56:38.060 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T11:56:38.060 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T11:56:38.060 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T11:56:38.060 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:56:38.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:38.185 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:38.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:38.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:38.331 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:38.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:38.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:38.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:38.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:38.603 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:38.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:38.705 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:38.732 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:38.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:38.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:38.861 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:38.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:38.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T11:56:38.963 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:39.005 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:39.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:39.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:38 smithi064 ceph-mon[30223]: pgmap v164: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-03-24T11:56:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:38 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:56:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:38 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:56:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:38 smithi064 ceph-mon[30223]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:56:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:38 smithi064 ceph-mon[42243]: pgmap v164: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-03-24T11:56:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:38 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:56:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:38 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:56:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:38 smithi064 ceph-mon[42243]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:56:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:38 smithi193 ceph-mon[33593]: pgmap v164: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-03-24T11:56:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:38 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:56:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:38 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:56:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:38 smithi193 ceph-mon[33593]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:56:39.107 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:39.137 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:39.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:39.239 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:39.269 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:39.281 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:39.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:39.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:39.413 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:39.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:39.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:39.566 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:39.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:39.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:39.711 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:39.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:39.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:39.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:39.852 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:39.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:39.953 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:39.983 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:39.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:40.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:40.117 INFO:tasks.thrashosds.thrasher:j is {'epoch': 131, 'fsid': '06f23456-e9d4-11ee-95cd-87774f69a715', 'created': '2024-03-24T11:46:40.657065+0000', 'modified': '2024-03-24T11:56:28.060165+0000', 'last_up_change': '2024-03-24T11:52:01.590830+0000', 'last_in_change': '2024-03-24T11:51:43.667453+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:49:56.369674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T11:52:13.871974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-24T11:52:14.117018+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T11:52:15.453744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T11:52:17.611004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T11:52:19.701416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-24T11:54:14.779650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'bc14a64f-9e6c-41f1-901e-c49fabffb53d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 2675992571}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6802', 'nonce': 2675992571}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6804', 'nonce': 2675992571}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 2675992571}]}, 'public_addr': '172.21.15.64:6801/2675992571', 'cluster_addr': '172.21.15.64:6802/2675992571', 'heartbeat_back_addr': '172.21.15.64:6804/2675992571', 'heartbeat_front_addr': '172.21.15.64:6803/2675992571', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '74587332-f0b2-4905-9888-7bdb604cc0e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 841667271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6806', 'nonce': 841667271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6808', 'nonce': 841667271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 841667271}]}, 'public_addr': '172.21.15.64:6805/841667271', 'cluster_addr': '172.21.15.64:6806/841667271', 'heartbeat_back_addr': '172.21.15.64:6808/841667271', 'heartbeat_front_addr': '172.21.15.64:6807/841667271', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a0c0ed85-9f33-4198-a1f2-447eb17c364b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 435147390}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6810', 'nonce': 435147390}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6812', 'nonce': 435147390}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 435147390}]}, 'public_addr': '172.21.15.64:6809/435147390', 'cluster_addr': '172.21.15.64:6810/435147390', 'heartbeat_back_addr': '172.21.15.64:6812/435147390', 'heartbeat_front_addr': '172.21.15.64:6811/435147390', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1396f66c-3e0c-4a01-b0a7-6f763989a479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3355318038}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6814', 'nonce': 3355318038}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6816', 'nonce': 3355318038}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3355318038}]}, 'public_addr': '172.21.15.64:6813/3355318038', 'cluster_addr': '172.21.15.64:6814/3355318038', 'heartbeat_back_addr': '172.21.15.64:6816/3355318038', 'heartbeat_front_addr': '172.21.15.64:6815/3355318038', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50d65d8b-3e90-46a4-b19d-50c3015af40b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3851339176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3851339176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3851339176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3851339176}]}, 'public_addr': '172.21.15.193:6800/3851339176', 'cluster_addr': '172.21.15.193:6801/3851339176', 'heartbeat_back_addr': '172.21.15.193:6803/3851339176', 'heartbeat_front_addr': '172.21.15.193:6802/3851339176', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '50c5d50e-7e68-487c-9f68-1abb8c927ff6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 1625306337}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 1625306337}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 1625306337}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 1625306337}]}, 'public_addr': '172.21.15.193:6804/1625306337', 'cluster_addr': '172.21.15.193:6805/1625306337', 'heartbeat_back_addr': '172.21.15.193:6807/1625306337', 'heartbeat_front_addr': '172.21.15.193:6806/1625306337', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7e74fe39-5707-4c90-ad38-1083cd7a6996', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 2408237278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 2408237278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 2408237278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 2408237278}]}, 'public_addr': '172.21.15.193:6808/2408237278', 'cluster_addr': '172.21.15.193:6809/2408237278', 'heartbeat_back_addr': '172.21.15.193:6811/2408237278', 'heartbeat_front_addr': '172.21.15.193:6810/2408237278', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e568df64-cc9a-43f8-a432-8628d1da1015', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 2649989609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 2649989609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 2649989609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 2649989609}]}, 'public_addr': '172.21.15.193:6812/2649989609', 'cluster_addr': '172.21.15.193:6813/2649989609', 'heartbeat_back_addr': '172.21.15.193:6815/2649989609', 'heartbeat_front_addr': '172.21.15.193:6814/2649989609', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:55:24.599632+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:55:44.834305+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:55:55.654778+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:55:19.345998+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:55:18.819821+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:56:11.616067+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:55:21.525981+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:56:09.668197+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.b', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.64:6800/3538503390': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/1277626303': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3407451558': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3753895773': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/842059688': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/1610565583': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/2730170969': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3257265142': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/1459540658': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/629947599': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:6800/520335082': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3351953651': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:6800/4027721305': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/4028269665': '2024-03-25T11:47:36.035658+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T11:56:40.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:56:40.132 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:40.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:40.236 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:40.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:40.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:40.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:40.409 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:40.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:40.510 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:40.538 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:40.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:40.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:40.694 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:40.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:40.785 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:40.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:40.863 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:40.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:40.951 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:40 smithi193 ceph-mon[33593]: pgmap v166: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-03-24T11:56:40.951 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:40.951 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:40.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:41.007 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:41.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:40 smithi064 ceph-mon[30223]: pgmap v166: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-03-24T11:56:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:40 smithi064 ceph-mon[42243]: pgmap v166: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-03-24T11:56:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:56:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:41.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:41.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:56:41.160 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:41.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:41.260 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28181 7.1 7:85082842:::benchmark_data_smithi064_87013_object28180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.184089+0000", 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 114.075061662, 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15663042499999999, 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.261 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28181 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184089+0000", 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184089+0000", 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184091+0000", 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.779e-06 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184155+0000", 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3891000000000002e-05 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.263 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184169+0000", 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4538e-05 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184178+0000", 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4549999999999995e-06 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184228+0000", 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0105000000000002e-05 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.264 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184349+0000", 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000121484 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184491+0000", 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000142249 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.265 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185428+0000", 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00093699699999999996 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185644+0000", 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000215598 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.340609+0000", 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.154965294 2024-03-24T11:56:41.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.340670+0000", 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0247000000000001e-05 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.340719+0000", 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9787999999999998e-05 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.267 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28188 7.1 7:83e3e525:::benchmark_data_smithi064_87013_object28187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.187122+0000", 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 114.07202857599999, 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15375007500000001, 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28188 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.268 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187122+0000", 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187122+0000", 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187124+0000", 2024-03-24T11:56:41.269 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2340000000000001e-06 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187157+0000", 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2870999999999997e-05 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187168+0000", 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0453e-05 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.270 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187176+0000", 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0749999999999998e-06 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187231+0000", 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5271e-05 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187347+0000", 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000116072 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.271 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187480+0000", 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000133459 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188587+0000", 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001106744 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188921+0000", 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033416300000000001 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.272 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.340773+0000", 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15185157599999999 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.340837+0000", 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4060000000000007e-05 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.340872+0000", 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5097e-05 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28608 7.1 7:8632daf5:::benchmark_data_smithi064_87013_object28607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.552941+0000", 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 113.706209203, 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.156585635, 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.274 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28608 2024-03-24T11:56:41.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:41.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.276 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.276 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552941+0000", 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552941+0000", 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552944+0000", 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.402e-06 2024-03-24T11:56:41.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.553000+0000", 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6465000000000002e-05 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.553014+0000", 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3621000000000001e-05 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.553021+0000", 2024-03-24T11:56:41.278 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5129999999999999e-06 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.553083+0000", 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1413000000000003e-05 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.553192+0000", 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000109574 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.279 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.553322+0000", 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012961599999999999 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.554405+0000", 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0010830499999999999 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.554664+0000", 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025932599999999999 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.280 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.709455+0000", 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.154790488 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.709495+0000", 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0000999999999999e-05 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.709527+0000", 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2165999999999997e-05 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:29493 7.1 7:8248f508:::benchmark_data_smithi064_87013_object29492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:48.327595+0000", 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.931555843, 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.070035564999999994, 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 29493 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.282 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.327595+0000", 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.327595+0000", 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.327597+0000", 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.832e-06 2024-03-24T11:56:41.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.327635+0000", 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8090000000000003e-05 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.327644+0000", 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2059999999999996e-06 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.327652+0000", 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4139999999999994e-06 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.284 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.327755+0000", 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010259600000000001 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.327897+0000", 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014229499999999999 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.328057+0000", 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000159381 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.285 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.330110+0000", 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002053259 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.330153+0000", 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3099000000000002e-05 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.397522+0000", 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.067368738999999997 2024-03-24T11:56:41.286 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.397582+0000", 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0260000000000002e-05 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.397630+0000", 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8393999999999998e-05 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.287 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:30170 7.1 7:80158f74:::benchmark_data_smithi064_87013_object30169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:48.766313+0000", 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.492837106, 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.088261519999999996, 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 30170 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766313+0000", 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766313+0000", 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766316+0000", 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.334e-06 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766369+0000", 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3236000000000003e-05 2024-03-24T11:56:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766391+0000", 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2220000000000001e-05 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766402+0000", 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0919e-05 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766483+0000", 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0587000000000006e-05 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766619+0000", 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013592500000000001 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766773+0000", 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015464799999999999 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.769229+0000", 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024561539999999999 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.772713+0000", 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034833109999999998 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.854446+0000", 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.081733389000000004 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.854519+0000", 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2669999999999994e-05 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.854575+0000", 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6127e-05 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:30177 7.1 7:88715fe8:::benchmark_data_smithi064_87013_object30176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:48.767173+0000", 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.49197727400001, 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.091999545000000002, 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 30177 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.767173+0000", 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.767173+0000", 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.767175+0000", 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.392e-06 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.767268+0000", 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3832999999999994e-05 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.767281+0000", 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3003000000000001e-05 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.767291+0000", 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8840000000000006e-06 2024-03-24T11:56:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.768337+0000", 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001045374 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.768443+0000", 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010604500000000001 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.768526+0000", 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2824000000000005e-05 2024-03-24T11:56:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.769322+0000", 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00079641 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.772743+0000", 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034212740000000002 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.859052+0000", 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.086308985000000005 2024-03-24T11:56:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.859125+0000", 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2422000000000006e-05 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.859173+0000", 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8099000000000001e-05 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:30185 7.1 7:8ca600e5:::benchmark_data_smithi064_87013_object30184:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:48.773179+0000", 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.48597182500001, 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.086135866000000005, 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 30185 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773179+0000", 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773179+0000", 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773181+0000", 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.897e-06 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773276+0000", 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5489999999999995e-05 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773288+0000", 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2041e-05 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773298+0000", 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4380000000000001e-06 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773336+0000", 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8337000000000002e-05 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773421+0000", 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5498e-05 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.773522+0000", 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010056900000000001 2024-03-24T11:56:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.774431+0000", 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00090929900000000002 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.775192+0000", 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00076031999999999998 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.859226+0000", 2024-03-24T11:56:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.084034763999999998 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.859278+0000", 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1156999999999998e-05 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.859315+0000", 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7055999999999999e-05 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:32554 7.1 7:811f6832:::benchmark_data_smithi064_87013_object32553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:50.250421+0000", 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 111.008729604, 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.11539584, 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 32554 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250421+0000", 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250421+0000", 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250425+0000", 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6710000000000001e-06 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250475+0000", 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0529999999999999e-05 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250487+0000", 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1806e-05 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250494+0000", 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6100000000000002e-06 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250548+0000", 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4453999999999997e-05 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250647+0000", 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8913000000000007e-05 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.250806+0000", 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000159178 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.252042+0000", 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001236174 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.252353+0000", 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00031084299999999999 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.365717+0000", 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.113364214 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.365771+0000", 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3295999999999999e-05 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:50.365817+0000", 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6150999999999999e-05 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:65812 7.1 7:8435938e:::benchmark_data_smithi064_87013_object65811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:12.703855+0000", 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 88.555295849000004, 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.153696952, 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 65812 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.703855+0000", 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.703855+0000", 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.703856+0000", 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.204e-06 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.703915+0000", 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9580000000000002e-05 2024-03-24T11:56:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.703925+0000", 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.645e-06 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.703933+0000", 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3149999999999992e-06 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.705855+0000", 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001921568 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.705991+0000", 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000135969 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.706140+0000", 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000149235 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.708385+0000", 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022447729999999998 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.708723+0000", 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033800300000000002 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.857409+0000", 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14868572499999999 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.857518+0000", 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000109202 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.857552+0000", 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3732999999999997e-05 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:65823 7.1 7:8a4675ff:::benchmark_data_smithi064_87013_object65822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:12.712694+0000", 2024-03-24T11:56:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 88.546456403999997, 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.153577143, 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 65823 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.712694+0000", 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.712694+0000", 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.712696+0000", 2024-03-24T11:56:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.672e-06 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.712745+0000", 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9296999999999999e-05 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.712757+0000", 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2276000000000001e-05 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.712763+0000", 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7960000000000001e-06 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.712825+0000", 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2125000000000006e-05 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.712950+0000", 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012507299999999999 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.713098+0000", 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000147544 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.715234+0000", 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021356309999999998 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.715546+0000", 2024-03-24T11:56:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00031233600000000003 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.866160+0000", 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.150614043 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.866222+0000", 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2356999999999999e-05 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.866271+0000", 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8992999999999997e-05 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67053 7.1 7:84c631cb:::benchmark_data_smithi064_87013_object67052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:13.872043+0000", 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 87.387107165000003, 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.178221296, 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67053 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872043+0000", 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872043+0000", 2024-03-24T11:56:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872045+0000", 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7489999999999999e-06 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872095+0000", 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0309000000000002e-05 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872108+0000", 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2211999999999999e-05 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872116+0000", 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1079999999999993e-06 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872168+0000", 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2188999999999998e-05 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872270+0000", 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010171 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.872369+0000", 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9735999999999995e-05 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.874594+0000", 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022248789999999999 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875003+0000", 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00040823399999999999 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.050194+0000", 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.175191128 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.050230+0000", 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6862e-05 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.050265+0000", 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4180000000000001e-05 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67665 7.1 7:8e5a15c9:::benchmark_data_smithi064_87013_object67664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.677074+0000", 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 86.582076125, 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.070045505999999993, 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67665 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677074+0000", 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677074+0000", 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677076+0000", 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0849999999999999e-06 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677114+0000", 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7517e-05 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677127+0000", 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3259e-05 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677136+0000", 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7530000000000007e-06 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677195+0000", 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8875000000000001e-05 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677387+0000", 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019194500000000001 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.677549+0000", 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016174000000000001 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.679807+0000", 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022584250000000001 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.680975+0000", 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0011682350000000001 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.747038+0000", 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.066063064000000005 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.747078+0000", 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9881e-05 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.747120+0000", 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1727000000000002e-05 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:80086 7.1 7:8acef5a3:::benchmark_data_smithi064_87013_object80085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:25.446669+0000", 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 75.812481276, 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.070611590000000002, 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 80086 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.446669+0000", 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.446669+0000", 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.446671+0000", 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9889999999999999e-06 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.446725+0000", 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3411000000000001e-05 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.446737+0000", 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2374999999999999e-05 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.446744+0000", 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3130000000000001e-06 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.446806+0000", 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.135e-05 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.446925+0000", 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000119108 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.447043+0000", 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011773 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.449248+0000", 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002205238 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.449785+0000", 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000537273 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.517175+0000", 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.067389493999999994 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.517234+0000", 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9483e-05 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:25.517281+0000", 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6826000000000002e-05 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:104083 7.1 7:83a67717:::benchmark_data_smithi064_87013_object104082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:48.312050+0000", 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 52.947100829999997, 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.070067993999999995, 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 104083 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312050+0000", 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312050+0000", 2024-03-24T11:56:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312061+0000", 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1194e-05 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312097+0000", 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5716000000000002e-05 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312115+0000", 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8581e-05 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312125+0000", 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0144e-05 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312197+0000", 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1594000000000006e-05 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312336+0000", 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013954899999999999 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.312488+0000", 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015160099999999999 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.314876+0000", 2024-03-24T11:56:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002388172 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.315257+0000", 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00038024599999999999 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.382021+0000", 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.066764360999999994 2024-03-24T11:56:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.382076+0000", 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5244000000000002e-05 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:48.382118+0000", 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1591999999999998e-05 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:128892 7.1 7:8f53199d:::benchmark_data_smithi064_87013_object7935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:13.811724+0000", 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 27.447426171, 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.107988625, 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 128892 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811724+0000", 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811724+0000", 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811727+0000", 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.249e-06 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811732+0000", 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8449999999999999e-06 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811744+0000", 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1965e-05 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811753+0000", 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2919999999999996e-06 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811808+0000", 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5368000000000003e-05 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811887+0000", 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9212e-05 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.811963+0000", 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6018000000000003e-05 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.814534+0000", 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025711089999999998 2024-03-24T11:56:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.819298+0000", 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0047640499999999997 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.919603+0000", 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.10030480999999999 2024-03-24T11:56:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.919665+0000", 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2031000000000003e-05 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.919713+0000", 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7676000000000002e-05 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:128902 7.1 7:88ce70c6:::benchmark_data_smithi064_87013_object7945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:13.821168+0000", 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 27.437982488999999, 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.10972404199999999, 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 128902 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821168+0000", 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821168+0000", 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821170+0000", 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8500000000000001e-06 2024-03-24T11:56:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821175+0000", 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3800000000000002e-06 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821184+0000", 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9840000000000007e-06 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821190+0000", 2024-03-24T11:56:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8309999999999997e-06 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821227+0000", 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7159000000000001e-05 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821310+0000", 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3086999999999999e-05 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.821382+0000", 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2075000000000004e-05 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.823910+0000", 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002527278 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.824825+0000", 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00091512099999999997 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.930784+0000", 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.105959461 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.930847+0000", 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2669999999999995e-05 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.930892+0000", 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5145999999999998e-05 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:128907 7.1 7:8c3f34e1:::benchmark_data_smithi064_87013_object7950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:13.825195+0000", 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 27.433955522000002, 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.10589225200000001, 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 128907 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825195+0000", 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825195+0000", 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825197+0000", 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.725e-06 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825201+0000", 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6639999999999997e-06 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825212+0000", 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0543999999999999e-05 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825219+0000", 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.075e-06 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825256+0000", 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7394999999999997e-05 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825327+0000", 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0792999999999998e-05 2024-03-24T11:56:41.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.825391+0000", 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4059000000000005e-05 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.827072+0000", 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016810359999999999 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.828519+0000", 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001447088 2024-03-24T11:56:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.930954+0000", 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.102434148 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.931038+0000", 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3956999999999996e-05 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.931087+0000", 2024-03-24T11:56:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9768000000000002e-05 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:128922 7.1 7:870f45f2:::benchmark_data_smithi064_87013_object7965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:13.842033+0000", 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 27.417117873999999, 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.091590166000000001, 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.371 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 128922 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842033+0000", 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842033+0000", 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842035+0000", 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9139999999999998e-06 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842041+0000", 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2299999999999996e-06 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842050+0000", 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4690000000000003e-06 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842056+0000", 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3480000000000003e-06 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842107+0000", 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1362000000000001e-05 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842204+0000", 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6984999999999994e-05 2024-03-24T11:56:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.842277+0000", 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3143000000000001e-05 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.844331+0000", 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020538319999999998 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.844689+0000", 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00035813300000000001 2024-03-24T11:56:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.933494+0000", 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.088805086000000005 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.933582+0000", 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7495000000000002e-05 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.933623+0000", 2024-03-24T11:56:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1168999999999998e-05 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:129068 7.1 7:8e4ad8ea:::benchmark_data_smithi064_87013_object8111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:14.072647+0000", 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 27.186503985000002, 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.070547024, 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 129068 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072647+0000", 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072647+0000", 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072648+0000", 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.717e-06 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072653+0000", 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9400000000000001e-06 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072662+0000", 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6039999999999993e-06 2024-03-24T11:56:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072667+0000", 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6420000000000004e-06 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072711+0000", 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3979e-05 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072810+0000", 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8528000000000003e-05 2024-03-24T11:56:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.072878+0000", 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8324000000000005e-05 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.075092+0000", 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022139149999999999 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.080576+0000", 2024-03-24T11:56:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054837180000000003 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.143075+0000", 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.062499091999999999 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.143140+0000", 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4790999999999997e-05 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.143194+0000", 2024-03-24T11:56:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3773999999999997e-05 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:133531 7.1 7:8e01053e:::benchmark_data_smithi064_87013_object12574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:18.841302+0000", 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 22.417848831000001, 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.093300986000000002, 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:56:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 133531 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841302+0000", 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841302+0000", 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841303+0000", 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.53e-06 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841307+0000", 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1459999999999998e-06 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841317+0000", 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0088000000000001e-05 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841325+0000", 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4030000000000001e-06 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841369+0000", 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4403000000000002e-05 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841473+0000", 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000103757 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:56:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.841546+0000", 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2484999999999995e-05 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.843849+0000", 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002303846 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.844385+0000", 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00053584300000000004 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.934503+0000", 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.090118180000000006 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.934555+0000", 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1486999999999997e-05 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:18.934603+0000", 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7818000000000002e-05 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:41.389 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:56:41.389 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:41.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:41.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:41.543 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:41.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:41.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:41.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:41.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:41.778 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:41.781 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T11:56:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:41.805 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:41.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:41.906 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:41.937 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:41.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:42.038 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:42.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:41 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2182975772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:42.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:41 smithi064 ceph-mon[30223]: from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:41 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2182975772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:41 smithi064 ceph-mon[42243]: from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:42.069 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:42.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:42.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:41 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2182975772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:41 smithi193 ceph-mon[33593]: from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:56:42.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:42.211 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:42.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:42.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:42.342 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:42.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:42.444 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:42.475 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:42.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:42.577 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:42.606 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:42.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:42.708 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:42.739 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:42.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:42.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:42.883 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:42.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:42.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:42 smithi064 ceph-mon[30223]: pgmap v167: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 840 op/s 2024-03-24T11:56:42.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:42 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:42.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:42 smithi064 ceph-mon[30223]: from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:56:42.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:42 smithi064 ceph-mon[30223]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:56:42.909 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:42 smithi064 ceph-mon[42243]: pgmap v167: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 840 op/s 2024-03-24T11:56:42.909 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:42 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:42.909 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:42 smithi064 ceph-mon[42243]: from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:56:42.909 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:42 smithi064 ceph-mon[42243]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:56:42.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:43.015 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:43.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:42 smithi193 ceph-mon[33593]: pgmap v167: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 840 op/s 2024-03-24T11:56:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:42 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:42 smithi193 ceph-mon[33593]: from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:56:43.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:42 smithi193 ceph-mon[33593]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:56:43.116 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:43.145 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:43.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:43.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:43.275 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:43.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:43.328 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T11:56:43.377 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:43.412 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:43.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:43.454 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:43.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:43.567 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:43.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:43.662 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:43.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:43.711 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:43.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:43.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:43.858 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:43.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:43.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:56:43.977 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:43 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:56:43.977 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:43 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:56:43.977 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:43 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:56:43.977 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:43 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:56:43.977 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:43 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:56:43.977 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:43 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:56:43.978 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:43 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:56:43.978 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:43 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:56:43.985 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:56:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:44.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:44.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:43 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:56:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:43 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:56:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:43 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:56:44.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:43 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:56:44.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:44.141 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:44.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:44.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:44.269 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:44.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:44.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:44.411 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:44.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:44.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:44.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:44.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:44.661 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:44.691 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:44.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:44.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:44.833 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:44.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:44 smithi193 ceph-mon[33593]: pgmap v169: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:56:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:44 smithi193 ceph-mon[33593]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:56:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:44 smithi193 ceph-mon[33593]: from='client.24871 v1:172.21.15.64:0/2812900819' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:44.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:44.980 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:44.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:45.081 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:45.113 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:45.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:45.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:44 smithi064 ceph-mon[30223]: pgmap v169: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:56:45.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:44 smithi064 ceph-mon[30223]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:56:45.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:44 smithi064 ceph-mon[30223]: from='client.24871 v1:172.21.15.64:0/2812900819' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:45.185 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:44 smithi064 ceph-mon[42243]: pgmap v169: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:56:45.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:44 smithi064 ceph-mon[42243]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:56:45.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:44 smithi064 ceph-mon[42243]: from='client.24871 v1:172.21.15.64:0/2812900819' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:56:45.215 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:45.230 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-03-24T11:56:43.810000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5830220150,"num_objects":89152,"num_object_clones":0,"num_object_copies":267456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89152,"num_whiteouts":0,"num_read":1188,"num_read_kb":1034,"num_write":153571,"num_write_kb":7743518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89610,"ondisk_log_size":89610,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17914664,"kb_used_data":17184872,"kb_used_omap":12,"kb_used_meta":729651,"kb_avail":732046552,"statfs":{"total":767960285184,"available":749615669248,"internally_reserved":0,"allocated":17597308928,"data_stored":17594415946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":747163184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,3,7,7,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-296026112,"num_objects":-4517,"num_object_clones":0,"num_object_copies":-13551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4517,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.611212"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777325+0000","last_change":"2024-03-24T11:55:59.687133+0000","last_active":"2024-03-24T11:56:37.777325+0000","last_peered":"2024-03-24T11:56:37.777325+0000","last_clean":"2024-03-24T11:56:37.777325+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:56:37.777325+0000","last_undegraded":"2024-03-24T11:56:37.777325+0000","last_fullsized":"2024-03-24T11:56:37.777325+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:59.686983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:59.686983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.251542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781747+0000","last_change":"2024-03-24T11:56:26.035534+0000","last_active":"2024-03-24T11:56:37.781747+0000","last_peered":"2024-03-24T11:56:37.781747+0000","last_clean":"2024-03-24T11:56:37.781747+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:56:37.781747+0000","last_undegraded":"2024-03-24T11:56:37.781747+0000","last_fullsized":"2024-03-24T11:56:37.781747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:26.035422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:56:26.035422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.390126+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.785404+0000","last_change":"2024-03-24T11:54:13.817576+0000","last_active":"2024-03-24T11:56:37.785404+0000","last_peered":"2024-03-24T11:56:37.785404+0000","last_clean":"2024-03-24T11:56:37.785404+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:56:37.785404+0000","last_undegraded":"2024-03-24T11:56:37.785404+0000","last_fullsized":"2024-03-24T11:56:37.785404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T11:54:13.817437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"117'22","reported_seq":186,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779428+0000","last_change":"2024-03-24T11:55:21.286888+0000","last_active":"2024-03-24T11:56:37.779428+0000","last_peered":"2024-03-24T11:56:37.779428+0000","last_clean":"2024-03-24T11:56:37.779428+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:56:37.779428+0000","last_undegraded":"2024-03-24T11:56:37.779428+0000","last_fullsized":"2024-03-24T11:56:37.779428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'22","last_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T11:55:21.286727+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779444+0000","last_change":"2024-03-24T11:55:12.810702+0000","last_active":"2024-03-24T11:56:37.779444+0000","last_peered":"2024-03-24T11:56:37.779444+0000","last_clean":"2024-03-24T11:56:37.779444+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:56:37.779444+0000","last_undegraded":"2024-03-24T11:56:37.779444+0000","last_fullsized":"2024-03-24T11:56:37.779444+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:12.810626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.810626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.916616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779686+0000","last_change":"2024-03-24T11:56:03.305225+0000","last_active":"2024-03-24T11:56:37.779686+0000","last_peered":"2024-03-24T11:56:37.779686+0000","last_clean":"2024-03-24T11:56:37.779686+0000","last_became_active":"2024-03-24T11:52:19.378282+0000","last_became_peered":"2024-03-24T11:52:19.378282+0000","last_unstale":"2024-03-24T11:56:37.779686+0000","last_undegraded":"2024-03-24T11:56:37.779686+0000","last_fullsized":"2024-03-24T11:56:37.779686+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:03.305072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:03.305072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:17.590871+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779807+0000","last_change":"2024-03-24T11:55:20.239169+0000","last_active":"2024-03-24T11:56:37.779807+0000","last_peered":"2024-03-24T11:56:37.779807+0000","last_clean":"2024-03-24T11:56:37.779807+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:56:37.779807+0000","last_undegraded":"2024-03-24T11:56:37.779807+0000","last_fullsized":"2024-03-24T11:56:37.779807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.239058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.239058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781902+0000","last_change":"2024-03-24T11:56:08.033498+0000","last_active":"2024-03-24T11:56:37.781902+0000","last_peered":"2024-03-24T11:56:37.781902+0000","last_clean":"2024-03-24T11:56:37.781902+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:56:37.781902+0000","last_undegraded":"2024-03-24T11:56:37.781902+0000","last_fullsized":"2024-03-24T11:56:37.781902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:08.033370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.033370+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:20.438822+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":152,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774784+0000","last_change":"2024-03-24T11:56:25.437491+0000","last_active":"2024-03-24T11:56:37.774784+0000","last_peered":"2024-03-24T11:56:37.774784+0000","last_clean":"2024-03-24T11:56:37.774784+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:56:37.774784+0000","last_undegraded":"2024-03-24T11:56:37.774784+0000","last_fullsized":"2024-03-24T11:56:37.774784+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.437410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.437410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.581445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780753+0000","last_change":"2024-03-24T11:56:00.086492+0000","last_active":"2024-03-24T11:56:37.780753+0000","last_peered":"2024-03-24T11:56:37.780753+0000","last_clean":"2024-03-24T11:56:37.780753+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:56:37.780753+0000","last_undegraded":"2024-03-24T11:56:37.780753+0000","last_fullsized":"2024-03-24T11:56:37.780753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:00.086396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:00.086396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:01.178550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781320+0000","last_change":"2024-03-24T11:56:23.689326+0000","last_active":"2024-03-24T11:56:37.781320+0000","last_peered":"2024-03-24T11:56:37.781320+0000","last_clean":"2024-03-24T11:56:37.781320+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:56:37.781320+0000","last_undegraded":"2024-03-24T11:56:37.781320+0000","last_fullsized":"2024-03-24T11:56:37.781320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:23.689192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:23.689192+0000","last_clean_scrub_stamp":"2024-03-24T11:56:23.689192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:24.558926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":172,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779092+0000","last_change":"2024-03-24T11:56:24.027679+0000","last_active":"2024-03-24T11:56:37.779092+0000","last_peered":"2024-03-24T11:56:37.779092+0000","last_clean":"2024-03-24T11:56:37.779092+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:56:37.779092+0000","last_undegraded":"2024-03-24T11:56:37.779092+0000","last_fullsized":"2024-03-24T11:56:37.779092+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T11:56:24.027577+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:46.704448+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":137,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780713+0000","last_change":"2024-03-24T11:54:58.045843+0000","last_active":"2024-03-24T11:56:37.780713+0000","last_peered":"2024-03-24T11:56:37.780713+0000","last_clean":"2024-03-24T11:56:37.780713+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:56:37.780713+0000","last_undegraded":"2024-03-24T11:56:37.780713+0000","last_fullsized":"2024-03-24T11:56:37.780713+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.045730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.045730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774946+0000","last_change":"2024-03-24T11:56:03.549369+0000","last_active":"2024-03-24T11:56:37.774946+0000","last_peered":"2024-03-24T11:56:37.774946+0000","last_clean":"2024-03-24T11:56:37.774946+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:56:37.774946+0000","last_undegraded":"2024-03-24T11:56:37.774946+0000","last_fullsized":"2024-03-24T11:56:37.774946+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:03.549293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:03.549293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:10.966750+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781094+0000","last_change":"2024-03-24T11:55:07.333642+0000","last_active":"2024-03-24T11:56:37.781094+0000","last_peered":"2024-03-24T11:56:37.781094+0000","last_clean":"2024-03-24T11:56:37.781094+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:56:37.781094+0000","last_undegraded":"2024-03-24T11:56:37.781094+0000","last_fullsized":"2024-03-24T11:56:37.781094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:07.333529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:07.333529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.726773+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":176,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.778841+0000","last_change":"2024-03-24T11:56:15.991404+0000","last_active":"2024-03-24T11:56:37.778841+0000","last_peered":"2024-03-24T11:56:37.778841+0000","last_clean":"2024-03-24T11:56:37.778841+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:56:37.778841+0000","last_undegraded":"2024-03-24T11:56:37.778841+0000","last_fullsized":"2024-03-24T11:56:37.778841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:15.991241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:15.991241+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:42.423985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":155,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777912+0000","last_change":"2024-03-24T11:56:27.778229+0000","last_active":"2024-03-24T11:56:37.777912+0000","last_peered":"2024-03-24T11:56:37.777912+0000","last_clean":"2024-03-24T11:56:37.777912+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:56:37.777912+0000","last_undegraded":"2024-03-24T11:56:37.777912+0000","last_fullsized":"2024-03-24T11:56:37.777912+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:27.777985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:27.777985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:54.716158+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779249+0000","last_change":"2024-03-24T11:56:11.992907+0000","last_active":"2024-03-24T11:56:37.779249+0000","last_peered":"2024-03-24T11:56:37.779249+0000","last_clean":"2024-03-24T11:56:37.779249+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:56:37.779249+0000","last_undegraded":"2024-03-24T11:56:37.779249+0000","last_fullsized":"2024-03-24T11:56:37.779249+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:11.992773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:11.992773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.465543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":164,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780778+0000","last_change":"2024-03-24T11:54:59.374964+0000","last_active":"2024-03-24T11:56:37.780778+0000","last_peered":"2024-03-24T11:56:37.780778+0000","last_clean":"2024-03-24T11:56:37.780778+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:56:37.780778+0000","last_undegraded":"2024-03-24T11:56:37.780778+0000","last_fullsized":"2024-03-24T11:56:37.780778+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:54:59.374820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:59.374820+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":200,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777617+0000","last_change":"2024-03-24T11:55:19.242716+0000","last_active":"2024-03-24T11:56:37.777617+0000","last_peered":"2024-03-24T11:56:37.777617+0000","last_clean":"2024-03-24T11:56:37.777617+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:56:37.777617+0000","last_undegraded":"2024-03-24T11:56:37.777617+0000","last_fullsized":"2024-03-24T11:56:37.777617+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:55:19.242577+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.242577+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:40.357687+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779694+0000","last_change":"2024-03-24T11:56:25.010309+0000","last_active":"2024-03-24T11:56:37.779694+0000","last_peered":"2024-03-24T11:56:37.779694+0000","last_clean":"2024-03-24T11:56:37.779694+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:56:37.779694+0000","last_undegraded":"2024-03-24T11:56:37.779694+0000","last_fullsized":"2024-03-24T11:56:37.779694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.010190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.010190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:54.549909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776459+0000","last_change":"2024-03-24T11:55:14.026391+0000","last_active":"2024-03-24T11:56:37.776459+0000","last_peered":"2024-03-24T11:56:37.776459+0000","last_clean":"2024-03-24T11:56:37.776459+0000","last_became_active":"2024-03-24T11:52:19.384249+0000","last_became_peered":"2024-03-24T11:52:19.384249+0000","last_unstale":"2024-03-24T11:56:37.776459+0000","last_undegraded":"2024-03-24T11:56:37.776459+0000","last_fullsized":"2024-03-24T11:56:37.776459+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:14.026252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.026252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:39.442153+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":164,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777387+0000","last_change":"2024-03-24T11:56:25.722636+0000","last_active":"2024-03-24T11:56:37.777387+0000","last_peered":"2024-03-24T11:56:37.777387+0000","last_clean":"2024-03-24T11:56:37.777387+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:56:37.777387+0000","last_undegraded":"2024-03-24T11:56:37.777387+0000","last_fullsized":"2024-03-24T11:56:37.777387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.722560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.722560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:28.956369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"115'22","reported_seq":186,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775721+0000","last_change":"2024-03-24T11:55:12.618998+0000","last_active":"2024-03-24T11:56:37.775721+0000","last_peered":"2024-03-24T11:56:37.775721+0000","last_clean":"2024-03-24T11:56:37.775721+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:56:37.775721+0000","last_undegraded":"2024-03-24T11:56:37.775721+0000","last_fullsized":"2024-03-24T11:56:37.775721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'22","last_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.618646+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:42.680810+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776102+0000","last_change":"2024-03-24T11:56:25.369507+0000","last_active":"2024-03-24T11:56:37.776102+0000","last_peered":"2024-03-24T11:56:37.776102+0000","last_clean":"2024-03-24T11:56:37.776102+0000","last_became_active":"2024-03-24T11:52:21.390764+0000","last_became_peered":"2024-03-24T11:52:21.390764+0000","last_unstale":"2024-03-24T11:56:37.776102+0000","last_undegraded":"2024-03-24T11:56:37.776102+0000","last_fullsized":"2024-03-24T11:56:37.776102+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.369074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.369074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.196833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.782498+0000","last_change":"2024-03-24T11:55:06.054268+0000","last_active":"2024-03-24T11:56:37.782498+0000","last_peered":"2024-03-24T11:56:37.782498+0000","last_clean":"2024-03-24T11:56:37.782498+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:56:37.782498+0000","last_undegraded":"2024-03-24T11:56:37.782498+0000","last_fullsized":"2024-03-24T11:56:37.782498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.054160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.054160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779069+0000","last_change":"2024-03-24T11:56:14.756794+0000","last_active":"2024-03-24T11:56:37.779069+0000","last_peered":"2024-03-24T11:56:37.779069+0000","last_clean":"2024-03-24T11:56:37.779069+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:56:37.779069+0000","last_undegraded":"2024-03-24T11:56:37.779069+0000","last_fullsized":"2024-03-24T11:56:37.779069+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:56:14.756549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.756549+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:44.749261+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.784517+0000","last_change":"2024-03-24T11:56:19.749277+0000","last_active":"2024-03-24T11:56:37.784517+0000","last_peered":"2024-03-24T11:56:37.784517+0000","last_clean":"2024-03-24T11:56:37.784517+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:56:37.784517+0000","last_undegraded":"2024-03-24T11:56:37.784517+0000","last_fullsized":"2024-03-24T11:56:37.784517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:19.749138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:19.749138+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:33.204161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781450+0000","last_change":"2024-03-24T11:56:22.692993+0000","last_active":"2024-03-24T11:56:37.781450+0000","last_peered":"2024-03-24T11:56:37.781450+0000","last_clean":"2024-03-24T11:56:37.781450+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:56:37.781450+0000","last_undegraded":"2024-03-24T11:56:37.781450+0000","last_fullsized":"2024-03-24T11:56:37.781450+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:56:22.692822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:22.692822+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:44.367501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":214,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.784496+0000","last_change":"2024-03-24T11:56:06.931131+0000","last_active":"2024-03-24T11:56:37.784496+0000","last_peered":"2024-03-24T11:56:37.784496+0000","last_clean":"2024-03-24T11:56:37.784496+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:56:37.784496+0000","last_undegraded":"2024-03-24T11:56:37.784496+0000","last_fullsized":"2024-03-24T11:56:37.784496+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:06.931035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.931035+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.652773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776482+0000","last_change":"2024-03-24T11:55:00.754485+0000","last_active":"2024-03-24T11:56:37.776482+0000","last_peered":"2024-03-24T11:56:37.776482+0000","last_clean":"2024-03-24T11:56:37.776482+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:56:37.776482+0000","last_undegraded":"2024-03-24T11:56:37.776482+0000","last_fullsized":"2024-03-24T11:56:37.776482+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:00.754339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:00.754339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.782580+0000","last_change":"2024-03-24T11:56:15.211513+0000","last_active":"2024-03-24T11:56:37.782580+0000","last_peered":"2024-03-24T11:56:37.782580+0000","last_clean":"2024-03-24T11:56:37.782580+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:56:37.782580+0000","last_undegraded":"2024-03-24T11:56:37.782580+0000","last_fullsized":"2024-03-24T11:56:37.782580+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T11:56:15.211165+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_clean_scrub_stamp":"2024-03-24T11:56:15.211165+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.992778+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780924+0000","last_change":"2024-03-24T11:56:24.748650+0000","last_active":"2024-03-24T11:56:37.780924+0000","last_peered":"2024-03-24T11:56:37.780924+0000","last_clean":"2024-03-24T11:56:37.780924+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:56:37.780924+0000","last_undegraded":"2024-03-24T11:56:37.780924+0000","last_fullsized":"2024-03-24T11:56:37.780924+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:24.748528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:24.748528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.615827+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.783958+0000","last_change":"2024-03-24T11:56:01.069407+0000","last_active":"2024-03-24T11:56:37.783958+0000","last_peered":"2024-03-24T11:56:37.783958+0000","last_clean":"2024-03-24T11:56:37.783958+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:56:37.783958+0000","last_undegraded":"2024-03-24T11:56:37.783958+0000","last_fullsized":"2024-03-24T11:56:37.783958+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:01.069280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.632287+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.069280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:11.863245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.784320+0000","last_change":"2024-03-24T11:56:04.915981+0000","last_active":"2024-03-24T11:56:37.784320+0000","last_peered":"2024-03-24T11:56:37.784320+0000","last_clean":"2024-03-24T11:56:37.784320+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:56:37.784320+0000","last_undegraded":"2024-03-24T11:56:37.784320+0000","last_fullsized":"2024-03-24T11:56:37.784320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:04.915872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:04.915872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:20.330801+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":173,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779340+0000","last_change":"2024-03-24T11:56:05.799020+0000","last_active":"2024-03-24T11:56:37.779340+0000","last_peered":"2024-03-24T11:56:37.779340+0000","last_clean":"2024-03-24T11:56:37.779340+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:56:37.779340+0000","last_undegraded":"2024-03-24T11:56:37.779340+0000","last_fullsized":"2024-03-24T11:56:37.779340+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:05.798510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.798510+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:34.206728+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780676+0000","last_change":"2024-03-24T11:56:14.695577+0000","last_active":"2024-03-24T11:56:37.780676+0000","last_peered":"2024-03-24T11:56:37.780676+0000","last_clean":"2024-03-24T11:56:37.780676+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:56:37.780676+0000","last_undegraded":"2024-03-24T11:56:37.780676+0000","last_fullsized":"2024-03-24T11:56:37.780676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:14.695456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.695456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:17.493129+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777217+0000","last_change":"2024-03-24T11:54:58.059296+0000","last_active":"2024-03-24T11:56:37.777217+0000","last_peered":"2024-03-24T11:56:37.777217+0000","last_clean":"2024-03-24T11:56:37.777217+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:56:37.777217+0000","last_undegraded":"2024-03-24T11:56:37.777217+0000","last_fullsized":"2024-03-24T11:56:37.777217+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.059216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.059216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.612729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775425+0000","last_change":"2024-03-24T11:55:18.619637+0000","last_active":"2024-03-24T11:56:37.775425+0000","last_peered":"2024-03-24T11:56:37.775425+0000","last_clean":"2024-03-24T11:56:37.775425+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:56:37.775425+0000","last_undegraded":"2024-03-24T11:56:37.775425+0000","last_fullsized":"2024-03-24T11:56:37.775425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.619533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.619533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:43.172807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":160,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.773752+0000","last_change":"2024-03-24T11:55:00.363697+0000","last_active":"2024-03-24T11:56:37.773752+0000","last_peered":"2024-03-24T11:56:37.773752+0000","last_clean":"2024-03-24T11:56:37.773752+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:56:37.773752+0000","last_undegraded":"2024-03-24T11:56:37.773752+0000","last_fullsized":"2024-03-24T11:56:37.773752+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T11:55:00.363559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:00.363559+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.778929+0000","last_change":"2024-03-24T11:56:25.759157+0000","last_active":"2024-03-24T11:56:37.778929+0000","last_peered":"2024-03-24T11:56:37.778929+0000","last_clean":"2024-03-24T11:56:37.778929+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:56:37.778929+0000","last_undegraded":"2024-03-24T11:56:37.778929+0000","last_fullsized":"2024-03-24T11:56:37.778929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.759010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.759010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:37.953711+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776973+0000","last_change":"2024-03-24T11:55:56.132313+0000","last_active":"2024-03-24T11:56:37.776973+0000","last_peered":"2024-03-24T11:56:37.776973+0000","last_clean":"2024-03-24T11:56:37.776973+0000","last_became_active":"2024-03-24T11:52:19.377484+0000","last_became_peered":"2024-03-24T11:52:19.377484+0000","last_unstale":"2024-03-24T11:56:37.776973+0000","last_undegraded":"2024-03-24T11:56:37.776973+0000","last_fullsized":"2024-03-24T11:56:37.776973+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:56.132189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:55:56.132189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:01.345699+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776246+0000","last_change":"2024-03-24T11:55:15.993168+0000","last_active":"2024-03-24T11:56:37.776246+0000","last_peered":"2024-03-24T11:56:37.776246+0000","last_clean":"2024-03-24T11:56:37.776246+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:56:37.776246+0000","last_undegraded":"2024-03-24T11:56:37.776246+0000","last_fullsized":"2024-03-24T11:56:37.776246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:15.992998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:15.992998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:39.442239+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779242+0000","last_change":"2024-03-24T11:56:01.946316+0000","last_active":"2024-03-24T11:56:37.779242+0000","last_peered":"2024-03-24T11:56:37.779242+0000","last_clean":"2024-03-24T11:56:37.779242+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:56:37.779242+0000","last_undegraded":"2024-03-24T11:56:37.779242+0000","last_fullsized":"2024-03-24T11:56:37.779242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:01.946162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.946162+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:10.718956+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780278+0000","last_change":"2024-03-24T11:55:19.108172+0000","last_active":"2024-03-24T11:56:37.780278+0000","last_peered":"2024-03-24T11:56:37.780278+0000","last_clean":"2024-03-24T11:56:37.780278+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:56:37.780278+0000","last_undegraded":"2024-03-24T11:56:37.780278+0000","last_fullsized":"2024-03-24T11:56:37.780278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.108015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780713+0000","last_change":"2024-03-24T11:55:22.292901+0000","last_active":"2024-03-24T11:56:37.780713+0000","last_peered":"2024-03-24T11:56:37.780713+0000","last_clean":"2024-03-24T11:56:37.780713+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:56:37.780713+0000","last_undegraded":"2024-03-24T11:56:37.780713+0000","last_fullsized":"2024-03-24T11:56:37.780713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:22.292785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.292785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777459+0000","last_change":"2024-03-24T11:55:01.977733+0000","last_active":"2024-03-24T11:56:37.777459+0000","last_peered":"2024-03-24T11:56:37.777459+0000","last_clean":"2024-03-24T11:56:37.777459+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:56:37.777459+0000","last_undegraded":"2024-03-24T11:56:37.777459+0000","last_fullsized":"2024-03-24T11:56:37.777459+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.977610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.977610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.612713+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":177,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776788+0000","last_change":"2024-03-24T11:56:11.308420+0000","last_active":"2024-03-24T11:56:37.776788+0000","last_peered":"2024-03-24T11:56:37.776788+0000","last_clean":"2024-03-24T11:56:37.776788+0000","last_became_active":"2024-03-24T11:52:17.355956+0000","last_became_peered":"2024-03-24T11:52:17.355956+0000","last_unstale":"2024-03-24T11:56:37.776788+0000","last_undegraded":"2024-03-24T11:56:37.776788+0000","last_fullsized":"2024-03-24T11:56:37.776788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:11.308278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:11.308278+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:22.594130+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779972+0000","last_change":"2024-03-24T11:55:11.806686+0000","last_active":"2024-03-24T11:56:37.779972+0000","last_peered":"2024-03-24T11:56:37.779972+0000","last_clean":"2024-03-24T11:56:37.779972+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:56:37.779972+0000","last_undegraded":"2024-03-24T11:56:37.779972+0000","last_fullsized":"2024-03-24T11:56:37.779972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.806582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.806582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.916642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":212,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777780+0000","last_change":"2024-03-24T11:55:01.393587+0000","last_active":"2024-03-24T11:56:37.777780+0000","last_peered":"2024-03-24T11:56:37.777780+0000","last_clean":"2024-03-24T11:56:37.777780+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:56:37.777780+0000","last_undegraded":"2024-03-24T11:56:37.777780+0000","last_fullsized":"2024-03-24T11:56:37.777780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:01.393318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.393318+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:43.750966+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781602+0000","last_change":"2024-03-24T11:56:24.140811+0000","last_active":"2024-03-24T11:56:37.781602+0000","last_peered":"2024-03-24T11:56:37.781602+0000","last_clean":"2024-03-24T11:56:37.781602+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:56:37.781602+0000","last_undegraded":"2024-03-24T11:56:37.781602+0000","last_fullsized":"2024-03-24T11:56:37.781602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:24.140700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:24.140700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:51.459650+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775399+0000","last_change":"2024-03-24T11:55:13.643173+0000","last_active":"2024-03-24T11:56:37.775399+0000","last_peered":"2024-03-24T11:56:37.775399+0000","last_clean":"2024-03-24T11:56:37.775399+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:56:37.775399+0000","last_undegraded":"2024-03-24T11:56:37.775399+0000","last_fullsized":"2024-03-24T11:56:37.775399+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:55:13.643063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:13.643063+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.782068+0000","last_change":"2024-03-24T11:56:14.264963+0000","last_active":"2024-03-24T11:56:37.782068+0000","last_peered":"2024-03-24T11:56:37.782068+0000","last_clean":"2024-03-24T11:56:37.782068+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:56:37.782068+0000","last_undegraded":"2024-03-24T11:56:37.782068+0000","last_fullsized":"2024-03-24T11:56:37.782068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:14.264656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.264656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:38.580881+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":218,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775231+0000","last_change":"2024-03-24T11:56:24.399112+0000","last_active":"2024-03-24T11:56:37.775231+0000","last_peered":"2024-03-24T11:56:37.775231+0000","last_clean":"2024-03-24T11:56:37.775231+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:56:37.775231+0000","last_undegraded":"2024-03-24T11:56:37.775231+0000","last_fullsized":"2024-03-24T11:56:37.775231+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:24.398998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:24.398998+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:49.258784+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775094+0000","last_change":"2024-03-24T11:55:21.647704+0000","last_active":"2024-03-24T11:56:37.775094+0000","last_peered":"2024-03-24T11:56:37.775094+0000","last_clean":"2024-03-24T11:56:37.775094+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:56:37.775094+0000","last_undegraded":"2024-03-24T11:56:37.775094+0000","last_fullsized":"2024-03-24T11:56:37.775094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:21.647570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T11:55:21.647570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777597+0000","last_change":"2024-03-24T11:55:06.420550+0000","last_active":"2024-03-24T11:56:37.777597+0000","last_peered":"2024-03-24T11:56:37.777597+0000","last_clean":"2024-03-24T11:56:37.777597+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:56:37.777597+0000","last_undegraded":"2024-03-24T11:56:37.777597+0000","last_fullsized":"2024-03-24T11:56:37.777597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:55:06.420406+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.420406+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781233+0000","last_change":"2024-03-24T11:55:11.462967+0000","last_active":"2024-03-24T11:56:37.781233+0000","last_peered":"2024-03-24T11:56:37.781233+0000","last_clean":"2024-03-24T11:56:37.781233+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:56:37.781233+0000","last_undegraded":"2024-03-24T11:56:37.781233+0000","last_fullsized":"2024-03-24T11:56:37.781233+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.462846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.462846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.726853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775254+0000","last_change":"2024-03-24T11:56:05.489055+0000","last_active":"2024-03-24T11:56:37.775254+0000","last_peered":"2024-03-24T11:56:37.775254+0000","last_clean":"2024-03-24T11:56:37.775254+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:56:37.775254+0000","last_undegraded":"2024-03-24T11:56:37.775254+0000","last_fullsized":"2024-03-24T11:56:37.775254+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:05.488932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.488932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:20.205595+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":164,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777328+0000","last_change":"2024-03-24T11:56:26.760375+0000","last_active":"2024-03-24T11:56:37.777328+0000","last_peered":"2024-03-24T11:56:37.777328+0000","last_clean":"2024-03-24T11:56:37.777328+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:56:37.777328+0000","last_undegraded":"2024-03-24T11:56:37.777328+0000","last_fullsized":"2024-03-24T11:56:37.777328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T11:56:26.760297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:48.927348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.778013+0000","last_change":"2024-03-24T11:54:52.361385+0000","last_active":"2024-03-24T11:56:37.778013+0000","last_peered":"2024-03-24T11:56:37.778013+0000","last_clean":"2024-03-24T11:56:37.778013+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:56:37.778013+0000","last_undegraded":"2024-03-24T11:56:37.778013+0000","last_fullsized":"2024-03-24T11:56:37.778013+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:54:52.361260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.361260+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775287+0000","last_change":"2024-03-24T11:56:01.567568+0000","last_active":"2024-03-24T11:56:37.775287+0000","last_peered":"2024-03-24T11:56:37.775287+0000","last_clean":"2024-03-24T11:56:37.775287+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:56:37.775287+0000","last_undegraded":"2024-03-24T11:56:37.775287+0000","last_fullsized":"2024-03-24T11:56:37.775287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:01.567421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.567421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.945947+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.778107+0000","last_change":"2024-03-24T11:56:23.736925+0000","last_active":"2024-03-24T11:56:37.778107+0000","last_peered":"2024-03-24T11:56:37.778107+0000","last_clean":"2024-03-24T11:56:37.778107+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:56:37.778107+0000","last_undegraded":"2024-03-24T11:56:37.778107+0000","last_fullsized":"2024-03-24T11:56:37.778107+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:23.736791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:23.736791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:46.576335+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.773297+0000","last_change":"2024-03-24T11:55:01.390204+0000","last_active":"2024-03-24T11:56:37.773297+0000","last_peered":"2024-03-24T11:56:37.773297+0000","last_clean":"2024-03-24T11:56:37.773297+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:56:37.773297+0000","last_undegraded":"2024-03-24T11:56:37.773297+0000","last_fullsized":"2024-03-24T11:56:37.773297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.390019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.390019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:42.680893+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"115'13","reported_seq":173,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.782263+0000","last_change":"2024-03-24T11:55:16.094065+0000","last_active":"2024-03-24T11:56:37.782263+0000","last_peered":"2024-03-24T11:56:37.782263+0000","last_clean":"2024-03-24T11:56:37.782263+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:56:37.782263+0000","last_undegraded":"2024-03-24T11:56:37.782263+0000","last_fullsized":"2024-03-24T11:56:37.782263+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'13","last_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T11:55:16.093668+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"132'13322","reported_seq":13561,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T11:56:41.645843+0000","last_change":"2024-03-24T11:56:16.744237+0000","last_active":"2024-03-24T11:56:41.645843+0000","last_peered":"2024-03-24T11:56:41.645843+0000","last_clean":"2024-03-24T11:56:41.645843+0000","last_became_active":"2024-03-24T11:54:16.232098+0000","last_became_peered":"2024-03-24T11:54:16.232098+0000","last_unstale":"2024-03-24T11:56:41.645843+0000","last_undegraded":"2024-03-24T11:56:41.645843+0000","last_fullsized":"2024-03-24T11:56:41.645843+0000","mapping_epoch":97,"log_start":"130'8602","ondisk_log_start":"130'8602","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":3822,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 24s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369426432,"num_objects":5637,"num_object_clones":0,"num_object_copies":16911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9731,"num_write_kb":491776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780819+0000","last_change":"2024-03-24T11:56:02.765493+0000","last_active":"2024-03-24T11:56:37.780819+0000","last_peered":"2024-03-24T11:56:37.780819+0000","last_clean":"2024-03-24T11:56:37.780819+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:56:37.780819+0000","last_undegraded":"2024-03-24T11:56:37.780819+0000","last_fullsized":"2024-03-24T11:56:37.780819+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:02.765358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:02.765358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.107895+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779783+0000","last_change":"2024-03-24T11:55:20.861818+0000","last_active":"2024-03-24T11:56:37.779783+0000","last_peered":"2024-03-24T11:56:37.779783+0000","last_clean":"2024-03-24T11:56:37.779783+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:56:37.779783+0000","last_undegraded":"2024-03-24T11:56:37.779783+0000","last_fullsized":"2024-03-24T11:56:37.779783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:55:20.861648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.861648+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:38.570780+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":202,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779160+0000","last_change":"2024-03-24T11:56:14.948071+0000","last_active":"2024-03-24T11:56:37.779160+0000","last_peered":"2024-03-24T11:56:37.779160+0000","last_clean":"2024-03-24T11:56:37.779160+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:56:37.779160+0000","last_undegraded":"2024-03-24T11:56:37.779160+0000","last_fullsized":"2024-03-24T11:56:37.779160+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:56:14.947940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.947940+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:16.043116+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":151,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.773311+0000","last_change":"2024-03-24T11:55:19.652898+0000","last_active":"2024-03-24T11:56:37.773311+0000","last_peered":"2024-03-24T11:56:37.773311+0000","last_clean":"2024-03-24T11:56:37.773311+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:56:37.773311+0000","last_undegraded":"2024-03-24T11:56:37.773311+0000","last_fullsized":"2024-03-24T11:56:37.773311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:19.652641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.652641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:42.680842+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"132'9586","reported_seq":9750,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.155059+0000","last_change":"2024-03-24T11:56:08.680053+0000","last_active":"2024-03-24T11:56:41.155059+0000","last_peered":"2024-03-24T11:56:41.155059+0000","last_clean":"2024-03-24T11:56:41.155059+0000","last_became_active":"2024-03-24T11:54:16.238055+0000","last_became_peered":"2024-03-24T11:54:16.238055+0000","last_unstale":"2024-03-24T11:56:41.155059+0000","last_undegraded":"2024-03-24T11:56:41.155059+0000","last_fullsized":"2024-03-24T11:56:41.155059+0000","mapping_epoch":97,"log_start":"115'4300","ondisk_log_start":"115'4300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7741","last_scrub_stamp":"2024-03-24T11:56:08.679818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.679818+0000","objects_scrubbed":7448,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.903434+0000","scrub_duration":1667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363986944,"num_objects":5554,"num_object_clones":0,"num_object_copies":16662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9586,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780383+0000","last_change":"2024-03-24T11:56:27.229810+0000","last_active":"2024-03-24T11:56:37.780383+0000","last_peered":"2024-03-24T11:56:37.780383+0000","last_clean":"2024-03-24T11:56:37.780383+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:56:37.780383+0000","last_undegraded":"2024-03-24T11:56:37.780383+0000","last_fullsized":"2024-03-24T11:56:37.780383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:27.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:27.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:42.352370+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774188+0000","last_change":"2024-03-24T11:56:23.429848+0000","last_active":"2024-03-24T11:56:37.774188+0000","last_peered":"2024-03-24T11:56:37.774188+0000","last_clean":"2024-03-24T11:56:37.774188+0000","last_became_active":"2024-03-24T11:52:15.138792+0000","last_became_peered":"2024-03-24T11:52:15.138792+0000","last_unstale":"2024-03-24T11:56:37.774188+0000","last_undegraded":"2024-03-24T11:56:37.774188+0000","last_fullsized":"2024-03-24T11:56:37.774188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:23.429712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:56:23.429712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.052978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776884+0000","last_change":"2024-03-24T11:55:01.917263+0000","last_active":"2024-03-24T11:56:37.776884+0000","last_peered":"2024-03-24T11:56:37.776884+0000","last_clean":"2024-03-24T11:56:37.776884+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:56:37.776884+0000","last_undegraded":"2024-03-24T11:56:37.776884+0000","last_fullsized":"2024-03-24T11:56:37.776884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T11:55:01.917120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.917120+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:39.442186+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":212,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781813+0000","last_change":"2024-03-24T11:56:25.188171+0000","last_active":"2024-03-24T11:56:37.781813+0000","last_peered":"2024-03-24T11:56:37.781813+0000","last_clean":"2024-03-24T11:56:37.781813+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:56:37.781813+0000","last_undegraded":"2024-03-24T11:56:37.781813+0000","last_fullsized":"2024-03-24T11:56:37.781813+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:25.188045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.188045+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.612880+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"132'9743","reported_seq":9907,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.115106+0000","last_change":"2024-03-24T11:56:10.140756+0000","last_active":"2024-03-24T11:56:41.115106+0000","last_peered":"2024-03-24T11:56:41.115106+0000","last_clean":"2024-03-24T11:56:41.115106+0000","last_became_active":"2024-03-24T11:54:16.240514+0000","last_became_peered":"2024-03-24T11:54:16.240514+0000","last_unstale":"2024-03-24T11:56:41.115106+0000","last_undegraded":"2024-03-24T11:56:41.115106+0000","last_fullsized":"2024-03-24T11:56:41.115106+0000","mapping_epoch":97,"log_start":"112'3600","ondisk_log_start":"112'3600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'8025","last_scrub_stamp":"2024-03-24T11:56:10.140421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.140421+0000","objects_scrubbed":7499,"log_size":6143,"log_dups_size":0,"ondisk_log_size":6143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:13.331670+0000","scrub_duration":2191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374931456,"num_objects":5721,"num_object_clones":0,"num_object_copies":17163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9743,"num_write_kb":494848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779930+0000","last_change":"2024-03-24T11:55:12.389096+0000","last_active":"2024-03-24T11:56:37.779930+0000","last_peered":"2024-03-24T11:56:37.779930+0000","last_clean":"2024-03-24T11:56:37.779930+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:56:37.779930+0000","last_undegraded":"2024-03-24T11:56:37.779930+0000","last_fullsized":"2024-03-24T11:56:37.779930+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:12.388835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.388835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:43.751173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775137+0000","last_change":"2024-03-24T11:54:52.822234+0000","last_active":"2024-03-24T11:56:37.775137+0000","last_peered":"2024-03-24T11:56:37.775137+0000","last_clean":"2024-03-24T11:56:37.775137+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:56:37.775137+0000","last_undegraded":"2024-03-24T11:56:37.775137+0000","last_fullsized":"2024-03-24T11:56:37.775137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:52.822135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:52.822135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":206,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775063+0000","last_change":"2024-03-24T11:56:19.300291+0000","last_active":"2024-03-24T11:56:37.775063+0000","last_peered":"2024-03-24T11:56:37.775063+0000","last_clean":"2024-03-24T11:56:37.775063+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:56:37.775063+0000","last_undegraded":"2024-03-24T11:56:37.775063+0000","last_fullsized":"2024-03-24T11:56:37.775063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-03-24T11:56:19.300145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:19.300145+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.340159+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.782018+0000","last_change":"2024-03-24T11:56:06.187599+0000","last_active":"2024-03-24T11:56:37.782018+0000","last_peered":"2024-03-24T11:56:37.782018+0000","last_clean":"2024-03-24T11:56:37.782018+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:56:37.782018+0000","last_undegraded":"2024-03-24T11:56:37.782018+0000","last_fullsized":"2024-03-24T11:56:37.782018+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:06.187195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.187195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:29.724378+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"132'9593","reported_seq":9674,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.398964+0000","last_change":"2024-03-24T11:54:16.238411+0000","last_active":"2024-03-24T11:56:41.398964+0000","last_peered":"2024-03-24T11:56:41.398964+0000","last_clean":"2024-03-24T11:56:41.398964+0000","last_became_active":"2024-03-24T11:54:16.238056+0000","last_became_peered":"2024-03-24T11:54:16.238056+0000","last_unstale":"2024-03-24T11:56:41.398964+0000","last_undegraded":"2024-03-24T11:56:41.398964+0000","last_fullsized":"2024-03-24T11:56:41.398964+0000","mapping_epoch":97,"log_start":"115'4100","ondisk_log_start":"115'4100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:45.404254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360382464,"num_objects":5499,"num_object_clones":0,"num_object_copies":16497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9593,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781477+0000","last_change":"2024-03-24T11:55:57.171256+0000","last_active":"2024-03-24T11:56:37.781477+0000","last_peered":"2024-03-24T11:56:37.781477+0000","last_clean":"2024-03-24T11:56:37.781477+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:56:37.781477+0000","last_undegraded":"2024-03-24T11:56:37.781477+0000","last_fullsized":"2024-03-24T11:56:37.781477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:57.171146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:57.171146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:09.649463+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":428,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:40.241177+0000","last_change":"2024-03-24T11:55:22.048179+0000","last_active":"2024-03-24T11:56:40.241177+0000","last_peered":"2024-03-24T11:56:40.241177+0000","last_clean":"2024-03-24T11:56:40.241177+0000","last_became_active":"2024-03-24T11:52:15.144954+0000","last_became_peered":"2024-03-24T11:52:15.144954+0000","last_unstale":"2024-03-24T11:56:40.241177+0000","last_undegraded":"2024-03-24T11:56:40.241177+0000","last_fullsized":"2024-03-24T11:56:40.241177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:55:22.048036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.048036+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:46.092889+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":260,"num_read_kb":255,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775018+0000","last_change":"2024-03-24T11:56:10.706814+0000","last_active":"2024-03-24T11:56:37.775018+0000","last_peered":"2024-03-24T11:56:37.775018+0000","last_clean":"2024-03-24T11:56:37.775018+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:56:37.775018+0000","last_undegraded":"2024-03-24T11:56:37.775018+0000","last_fullsized":"2024-03-24T11:56:37.775018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.706685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.706685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:10.842111+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":181,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774714+0000","last_change":"2024-03-24T11:56:04.340339+0000","last_active":"2024-03-24T11:56:37.774714+0000","last_peered":"2024-03-24T11:56:37.774714+0000","last_clean":"2024-03-24T11:56:37.774714+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:56:37.774714+0000","last_undegraded":"2024-03-24T11:56:37.774714+0000","last_fullsized":"2024-03-24T11:56:37.774714+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-03-24T11:56:04.340193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:04.340193+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:09.770652+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776684+0000","last_change":"2024-03-24T11:56:20.561269+0000","last_active":"2024-03-24T11:56:37.776684+0000","last_peered":"2024-03-24T11:56:37.776684+0000","last_clean":"2024-03-24T11:56:37.776684+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:56:37.776684+0000","last_undegraded":"2024-03-24T11:56:37.776684+0000","last_fullsized":"2024-03-24T11:56:37.776684+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:20.561139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T11:56:20.561139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:33.363698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"132'9224","reported_seq":9305,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.407491+0000","last_change":"2024-03-24T11:54:16.238815+0000","last_active":"2024-03-24T11:56:41.407491+0000","last_peered":"2024-03-24T11:56:41.407491+0000","last_clean":"2024-03-24T11:56:41.407491+0000","last_became_active":"2024-03-24T11:54:16.238515+0000","last_became_peered":"2024-03-24T11:54:16.238515+0000","last_unstale":"2024-03-24T11:56:41.407491+0000","last_undegraded":"2024-03-24T11:56:41.407491+0000","last_fullsized":"2024-03-24T11:56:41.407491+0000","mapping_epoch":97,"log_start":"114'3700","ondisk_log_start":"114'3700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:45.401840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359661568,"num_objects":5488,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":470784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775674+0000","last_change":"2024-03-24T11:55:10.692203+0000","last_active":"2024-03-24T11:56:37.775674+0000","last_peered":"2024-03-24T11:56:37.775674+0000","last_clean":"2024-03-24T11:56:37.775674+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:56:37.775674+0000","last_undegraded":"2024-03-24T11:56:37.775674+0000","last_fullsized":"2024-03-24T11:56:37.775674+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:10.692130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:55:10.692130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:44.394776+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.786922+0000","last_change":"2024-03-24T11:55:56.666658+0000","last_active":"2024-03-24T11:56:37.786922+0000","last_peered":"2024-03-24T11:56:37.786922+0000","last_clean":"2024-03-24T11:56:37.786922+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:56:37.786922+0000","last_undegraded":"2024-03-24T11:56:37.786922+0000","last_fullsized":"2024-03-24T11:56:37.786922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T11:55:56.665991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:56.665991+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:19.338388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775543+0000","last_change":"2024-03-24T11:55:20.647830+0000","last_active":"2024-03-24T11:56:37.775543+0000","last_peered":"2024-03-24T11:56:37.775543+0000","last_clean":"2024-03-24T11:56:37.775543+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:56:37.775543+0000","last_undegraded":"2024-03-24T11:56:37.775543+0000","last_fullsized":"2024-03-24T11:56:37.775543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.647693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.647693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":413,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.783759+0000","last_change":"2024-03-24T11:56:05.931697+0000","last_active":"2024-03-24T11:56:37.783759+0000","last_peered":"2024-03-24T11:56:37.783759+0000","last_clean":"2024-03-24T11:56:37.783759+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:56:37.783759+0000","last_undegraded":"2024-03-24T11:56:37.783759+0000","last_fullsized":"2024-03-24T11:56:37.783759+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:56:05.931549+0000","last_deep_scrub":"71'119","last_deep_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.931549+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.615604+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":229,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777589+0000","last_change":"2024-03-24T11:55:22.095371+0000","last_active":"2024-03-24T11:56:37.777589+0000","last_peered":"2024-03-24T11:56:37.777589+0000","last_clean":"2024-03-24T11:56:37.777589+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:56:37.777589+0000","last_undegraded":"2024-03-24T11:56:37.777589+0000","last_fullsized":"2024-03-24T11:56:37.777589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:55:22.095232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.095232+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:43.523039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781303+0000","last_change":"2024-03-24T11:55:04.414736+0000","last_active":"2024-03-24T11:56:37.781303+0000","last_peered":"2024-03-24T11:56:37.781303+0000","last_clean":"2024-03-24T11:56:37.781303+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:56:37.781303+0000","last_undegraded":"2024-03-24T11:56:37.781303+0000","last_fullsized":"2024-03-24T11:56:37.781303+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:04.414651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:04.414651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.726945+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"132'9597","reported_seq":9758,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.099691+0000","last_change":"2024-03-24T11:56:12.814952+0000","last_active":"2024-03-24T11:56:41.099691+0000","last_peered":"2024-03-24T11:56:41.099691+0000","last_clean":"2024-03-24T11:56:41.099691+0000","last_became_active":"2024-03-24T11:54:16.236927+0000","last_became_peered":"2024-03-24T11:54:16.236927+0000","last_unstale":"2024-03-24T11:56:41.099691+0000","last_undegraded":"2024-03-24T11:56:41.099691+0000","last_fullsized":"2024-03-24T11:56:41.099691+0000","mapping_epoch":97,"log_start":"116'4600","ondisk_log_start":"116'4600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8029","last_scrub_stamp":"2024-03-24T11:56:12.814717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:12.814717+0000","objects_scrubbed":7259,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:22.656858+0000","scrub_duration":2088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368640000,"num_objects":5625,"num_object_clones":0,"num_object_copies":16875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9597,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774292+0000","last_change":"2024-03-24T11:55:14.967693+0000","last_active":"2024-03-24T11:56:37.774292+0000","last_peered":"2024-03-24T11:56:37.774292+0000","last_clean":"2024-03-24T11:56:37.774292+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:56:37.774292+0000","last_undegraded":"2024-03-24T11:56:37.774292+0000","last_fullsized":"2024-03-24T11:56:37.774292+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:14.967137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.967137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:39.442213+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781008+0000","last_change":"2024-03-24T11:56:26.209516+0000","last_active":"2024-03-24T11:56:37.781008+0000","last_peered":"2024-03-24T11:56:37.781008+0000","last_clean":"2024-03-24T11:56:37.781008+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:56:37.781008+0000","last_undegraded":"2024-03-24T11:56:37.781008+0000","last_fullsized":"2024-03-24T11:56:37.781008+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:26.209386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:26.209386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.139031+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.783727+0000","last_change":"2024-03-24T11:56:19.706222+0000","last_active":"2024-03-24T11:56:37.783727+0000","last_peered":"2024-03-24T11:56:37.783727+0000","last_clean":"2024-03-24T11:56:37.783727+0000","last_became_active":"2024-03-24T11:52:15.146310+0000","last_became_peered":"2024-03-24T11:52:15.146310+0000","last_unstale":"2024-03-24T11:56:37.783727+0000","last_undegraded":"2024-03-24T11:56:37.783727+0000","last_fullsized":"2024-03-24T11:56:37.783727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:19.706115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:19.706115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:42.732583+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":172,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.782163+0000","last_change":"2024-03-24T11:55:14.029361+0000","last_active":"2024-03-24T11:56:37.782163+0000","last_peered":"2024-03-24T11:56:37.782163+0000","last_clean":"2024-03-24T11:56:37.782163+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:56:37.782163+0000","last_undegraded":"2024-03-24T11:56:37.782163+0000","last_fullsized":"2024-03-24T11:56:37.782163+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:55:14.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.029279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775596+0000","last_change":"2024-03-24T11:56:08.456477+0000","last_active":"2024-03-24T11:56:37.775596+0000","last_peered":"2024-03-24T11:56:37.775596+0000","last_clean":"2024-03-24T11:56:37.775596+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:56:37.775596+0000","last_undegraded":"2024-03-24T11:56:37.775596+0000","last_fullsized":"2024-03-24T11:56:37.775596+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:08.456346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.456346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:13.785080+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"132'9769","reported_seq":9924,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.134858+0000","last_change":"2024-03-24T11:56:22.846505+0000","last_active":"2024-03-24T11:56:41.134858+0000","last_peered":"2024-03-24T11:56:41.134858+0000","last_clean":"2024-03-24T11:56:41.134858+0000","last_became_active":"2024-03-24T11:54:16.236857+0000","last_became_peered":"2024-03-24T11:54:16.236857+0000","last_unstale":"2024-03-24T11:56:41.134858+0000","last_undegraded":"2024-03-24T11:56:41.134858+0000","last_fullsized":"2024-03-24T11:56:41.134858+0000","mapping_epoch":97,"log_start":"114'4100","ondisk_log_start":"114'4100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8794","last_scrub_stamp":"2024-03-24T11:56:22.846262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:22.846262+0000","objects_scrubbed":6680,"log_size":5669,"log_dups_size":0,"ondisk_log_size":5669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.664107+0000","scrub_duration":1492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371392512,"num_objects":5667,"num_object_clones":0,"num_object_copies":17001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9769,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779666+0000","last_change":"2024-03-24T11:56:10.262911+0000","last_active":"2024-03-24T11:56:37.779666+0000","last_peered":"2024-03-24T11:56:37.779666+0000","last_clean":"2024-03-24T11:56:37.779666+0000","last_became_active":"2024-03-24T11:52:21.851069+0000","last_became_peered":"2024-03-24T11:52:21.851069+0000","last_unstale":"2024-03-24T11:56:37.779666+0000","last_undegraded":"2024-03-24T11:56:37.779666+0000","last_fullsized":"2024-03-24T11:56:37.779666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.262776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.262776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:12.921856+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781571+0000","last_change":"2024-03-24T11:54:51.499930+0000","last_active":"2024-03-24T11:56:37.781571+0000","last_peered":"2024-03-24T11:56:37.781571+0000","last_clean":"2024-03-24T11:56:37.781571+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:56:37.781571+0000","last_undegraded":"2024-03-24T11:56:37.781571+0000","last_fullsized":"2024-03-24T11:56:37.781571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:51.499817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:51.499817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.726824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":190,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781508+0000","last_change":"2024-03-24T11:56:05.227501+0000","last_active":"2024-03-24T11:56:37.781508+0000","last_peered":"2024-03-24T11:56:37.781508+0000","last_clean":"2024-03-24T11:56:37.781508+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:56:37.781508+0000","last_undegraded":"2024-03-24T11:56:37.781508+0000","last_fullsized":"2024-03-24T11:56:37.781508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:56:05.227347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.227347+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:12.047195+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777434+0000","last_change":"2024-03-24T11:55:15.058768+0000","last_active":"2024-03-24T11:56:37.777434+0000","last_peered":"2024-03-24T11:56:37.777434+0000","last_clean":"2024-03-24T11:56:37.777434+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:56:37.777434+0000","last_undegraded":"2024-03-24T11:56:37.777434+0000","last_fullsized":"2024-03-24T11:56:37.777434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:15.058688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T11:55:15.058688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.612804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"132'9467","reported_seq":9626,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.623141+0000","last_change":"2024-03-24T11:56:13.177828+0000","last_active":"2024-03-24T11:56:41.623141+0000","last_peered":"2024-03-24T11:56:41.623141+0000","last_clean":"2024-03-24T11:56:41.623141+0000","last_became_active":"2024-03-24T11:54:16.238176+0000","last_became_peered":"2024-03-24T11:54:16.238176+0000","last_unstale":"2024-03-24T11:56:41.623141+0000","last_undegraded":"2024-03-24T11:56:41.623141+0000","last_fullsized":"2024-03-24T11:56:41.623141+0000","mapping_epoch":97,"log_start":"117'4700","ondisk_log_start":"117'4700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'7875","last_scrub_stamp":"2024-03-24T11:56:13.177489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:13.177489+0000","objects_scrubbed":7037,"log_size":4767,"log_dups_size":0,"ondisk_log_size":4767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:13.371285+0000","scrub_duration":2432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352387072,"num_objects":5377,"num_object_clones":0,"num_object_copies":16131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779161+0000","last_change":"2024-03-24T11:56:22.106831+0000","last_active":"2024-03-24T11:56:37.779161+0000","last_peered":"2024-03-24T11:56:37.779161+0000","last_clean":"2024-03-24T11:56:37.779161+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:56:37.779161+0000","last_undegraded":"2024-03-24T11:56:37.779161+0000","last_fullsized":"2024-03-24T11:56:37.779161+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:22.106699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:22.106699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:43.372371+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774379+0000","last_change":"2024-03-24T11:56:27.426637+0000","last_active":"2024-03-24T11:56:37.774379+0000","last_peered":"2024-03-24T11:56:37.774379+0000","last_clean":"2024-03-24T11:56:37.774379+0000","last_became_active":"2024-03-24T11:52:15.142974+0000","last_became_peered":"2024-03-24T11:52:15.142974+0000","last_unstale":"2024-03-24T11:56:37.774379+0000","last_undegraded":"2024-03-24T11:56:37.774379+0000","last_fullsized":"2024-03-24T11:56:37.774379+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:56:27.426497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:27.426497+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:40.245207+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.773610+0000","last_change":"2024-03-24T11:54:57.356969+0000","last_active":"2024-03-24T11:56:37.773610+0000","last_peered":"2024-03-24T11:56:37.773610+0000","last_clean":"2024-03-24T11:56:37.773610+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:56:37.773610+0000","last_undegraded":"2024-03-24T11:56:37.773610+0000","last_fullsized":"2024-03-24T11:56:37.773610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:54:57.356797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:57.356797+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779217+0000","last_change":"2024-03-24T11:56:10.062814+0000","last_active":"2024-03-24T11:56:37.779217+0000","last_peered":"2024-03-24T11:56:37.779217+0000","last_clean":"2024-03-24T11:56:37.779217+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:56:37.779217+0000","last_undegraded":"2024-03-24T11:56:37.779217+0000","last_fullsized":"2024-03-24T11:56:37.779217+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.062643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.062643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:31.233410+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"132'9343","reported_seq":9503,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.645472+0000","last_change":"2024-03-24T11:56:06.946412+0000","last_active":"2024-03-24T11:56:41.645472+0000","last_peered":"2024-03-24T11:56:41.645472+0000","last_clean":"2024-03-24T11:56:41.645472+0000","last_became_active":"2024-03-24T11:54:16.235994+0000","last_became_peered":"2024-03-24T11:54:16.235994+0000","last_unstale":"2024-03-24T11:56:41.645472+0000","last_undegraded":"2024-03-24T11:56:41.645472+0000","last_fullsized":"2024-03-24T11:56:41.645472+0000","mapping_epoch":97,"log_start":"117'4600","ondisk_log_start":"117'4600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7491","last_scrub_stamp":"2024-03-24T11:56:06.946102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.946102+0000","objects_scrubbed":7377,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:18.268280+0000","scrub_duration":3152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359989248,"num_objects":5493,"num_object_clones":0,"num_object_copies":16479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":474752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781098+0000","last_change":"2024-03-24T11:55:06.364864+0000","last_active":"2024-03-24T11:56:37.781098+0000","last_peered":"2024-03-24T11:56:37.781098+0000","last_clean":"2024-03-24T11:56:37.781098+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:56:37.781098+0000","last_undegraded":"2024-03-24T11:56:37.781098+0000","last_fullsized":"2024-03-24T11:56:37.781098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.364777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.364777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.726883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.784424+0000","last_change":"2024-03-24T11:54:49.113470+0000","last_active":"2024-03-24T11:56:37.784424+0000","last_peered":"2024-03-24T11:56:37.784424+0000","last_clean":"2024-03-24T11:56:37.784424+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:56:37.784424+0000","last_undegraded":"2024-03-24T11:56:37.784424+0000","last_fullsized":"2024-03-24T11:56:37.784424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:49.113375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:54:49.113375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.612693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":181,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.778998+0000","last_change":"2024-03-24T11:55:05.889871+0000","last_active":"2024-03-24T11:56:37.778998+0000","last_peered":"2024-03-24T11:56:37.778998+0000","last_clean":"2024-03-24T11:56:37.778998+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:56:37.778998+0000","last_undegraded":"2024-03-24T11:56:37.778998+0000","last_fullsized":"2024-03-24T11:56:37.778998+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:55:05.889771+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:55:05.889771+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.916591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":213,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.773555+0000","last_change":"2024-03-24T11:56:13.769093+0000","last_active":"2024-03-24T11:56:37.773555+0000","last_peered":"2024-03-24T11:56:37.773555+0000","last_clean":"2024-03-24T11:56:37.773555+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:56:37.773555+0000","last_undegraded":"2024-03-24T11:56:37.773555+0000","last_fullsized":"2024-03-24T11:56:37.773555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:13.768824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:13.768824+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:22.717779+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"132'9532","reported_seq":9690,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.109702+0000","last_change":"2024-03-24T11:56:18.704600+0000","last_active":"2024-03-24T11:56:41.109702+0000","last_peered":"2024-03-24T11:56:41.109702+0000","last_clean":"2024-03-24T11:56:41.109702+0000","last_became_active":"2024-03-24T11:54:16.241724+0000","last_became_peered":"2024-03-24T11:54:16.241724+0000","last_unstale":"2024-03-24T11:56:41.109702+0000","last_undegraded":"2024-03-24T11:56:41.109702+0000","last_fullsized":"2024-03-24T11:56:41.109702+0000","mapping_epoch":97,"log_start":"112'3400","ondisk_log_start":"112'3400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8281","last_scrub_stamp":"2024-03-24T11:56:18.704486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:18.704486+0000","objects_scrubbed":6804,"log_size":6132,"log_dups_size":0,"ondisk_log_size":6132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:24.848265+0000","scrub_duration":1857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360775704,"num_objects":5506,"num_object_clones":0,"num_object_copies":16518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9532,"num_write_kb":481153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777479+0000","last_change":"2024-03-24T11:55:55.660561+0000","last_active":"2024-03-24T11:56:37.777479+0000","last_peered":"2024-03-24T11:56:37.777479+0000","last_clean":"2024-03-24T11:56:37.777479+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:56:37.777479+0000","last_undegraded":"2024-03-24T11:56:37.777479+0000","last_fullsized":"2024-03-24T11:56:37.777479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:55.660368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:55.660368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:14.298173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775543+0000","last_change":"2024-03-24T11:55:03.730546+0000","last_active":"2024-03-24T11:56:37.775543+0000","last_peered":"2024-03-24T11:56:37.775543+0000","last_clean":"2024-03-24T11:56:37.775543+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:56:37.775543+0000","last_undegraded":"2024-03-24T11:56:37.775543+0000","last_fullsized":"2024-03-24T11:56:37.775543+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:03.730436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:03.730436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.783818+0000","last_change":"2024-03-24T11:56:20.726116+0000","last_active":"2024-03-24T11:56:37.783818+0000","last_peered":"2024-03-24T11:56:37.783818+0000","last_clean":"2024-03-24T11:56:37.783818+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:56:37.783818+0000","last_undegraded":"2024-03-24T11:56:37.783818+0000","last_fullsized":"2024-03-24T11:56:37.783818+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T11:56:20.725884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.588943+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779721+0000","last_change":"2024-03-24T11:56:06.009518+0000","last_active":"2024-03-24T11:56:37.779721+0000","last_peered":"2024-03-24T11:56:37.779721+0000","last_clean":"2024-03-24T11:56:37.779721+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:56:37.779721+0000","last_undegraded":"2024-03-24T11:56:37.779721+0000","last_fullsized":"2024-03-24T11:56:37.779721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:06.009394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.009394+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:12.022411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"132'9521","reported_seq":9680,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.139387+0000","last_change":"2024-03-24T11:56:14.978194+0000","last_active":"2024-03-24T11:56:41.139387+0000","last_peered":"2024-03-24T11:56:41.139387+0000","last_clean":"2024-03-24T11:56:41.139387+0000","last_became_active":"2024-03-24T11:54:16.237102+0000","last_became_peered":"2024-03-24T11:54:16.237102+0000","last_unstale":"2024-03-24T11:56:41.139387+0000","last_undegraded":"2024-03-24T11:56:41.139387+0000","last_fullsized":"2024-03-24T11:56:41.139387+0000","mapping_epoch":97,"log_start":"111'3300","ondisk_log_start":"111'3300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8081","last_scrub_stamp":"2024-03-24T11:56:14.977871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.977871+0000","objects_scrubbed":7074,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:19.945606+0000","scrub_duration":2055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364838912,"num_objects":5567,"num_object_clones":0,"num_object_copies":16701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9521,"num_write_kb":482816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775302+0000","last_change":"2024-03-24T11:55:11.673583+0000","last_active":"2024-03-24T11:56:37.775302+0000","last_peered":"2024-03-24T11:56:37.775302+0000","last_clean":"2024-03-24T11:56:37.775302+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:56:37.775302+0000","last_undegraded":"2024-03-24T11:56:37.775302+0000","last_fullsized":"2024-03-24T11:56:37.775302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:11.673492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:11.673492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":213,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779670+0000","last_change":"2024-03-24T11:55:02.895074+0000","last_active":"2024-03-24T11:56:37.779670+0000","last_peered":"2024-03-24T11:56:37.779670+0000","last_clean":"2024-03-24T11:56:37.779670+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:56:37.779670+0000","last_undegraded":"2024-03-24T11:56:37.779670+0000","last_fullsized":"2024-03-24T11:56:37.779670+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:02.894942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:02.894942+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.916558+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779843+0000","last_change":"2024-03-24T11:55:58.143201+0000","last_active":"2024-03-24T11:56:37.779843+0000","last_peered":"2024-03-24T11:56:37.779843+0000","last_clean":"2024-03-24T11:56:37.779843+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:56:37.779843+0000","last_undegraded":"2024-03-24T11:56:37.779843+0000","last_fullsized":"2024-03-24T11:56:37.779843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:58.143072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:58.143072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:02.908198+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"130'22","reported_seq":193,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774659+0000","last_change":"2024-03-24T11:56:17.455467+0000","last_active":"2024-03-24T11:56:37.774659+0000","last_peered":"2024-03-24T11:56:37.774659+0000","last_clean":"2024-03-24T11:56:37.774659+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:56:37.774659+0000","last_undegraded":"2024-03-24T11:56:37.774659+0000","last_fullsized":"2024-03-24T11:56:37.774659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'22","last_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T11:56:17.455285+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:29.860233+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"132'9496","reported_seq":9577,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.397873+0000","last_change":"2024-03-24T11:54:16.239589+0000","last_active":"2024-03-24T11:56:41.397873+0000","last_peered":"2024-03-24T11:56:41.397873+0000","last_clean":"2024-03-24T11:56:41.397873+0000","last_became_active":"2024-03-24T11:54:16.239274+0000","last_became_peered":"2024-03-24T11:54:16.239274+0000","last_unstale":"2024-03-24T11:56:41.397873+0000","last_undegraded":"2024-03-24T11:56:41.397873+0000","last_fullsized":"2024-03-24T11:56:41.397873+0000","mapping_epoch":97,"log_start":"114'4000","ondisk_log_start":"114'4000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5496,"log_dups_size":0,"ondisk_log_size":5496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:45.404287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359661568,"num_objects":5488,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9496,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779541+0000","last_change":"2024-03-24T11:56:21.012442+0000","last_active":"2024-03-24T11:56:37.779541+0000","last_peered":"2024-03-24T11:56:37.779541+0000","last_clean":"2024-03-24T11:56:37.779541+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:56:37.779541+0000","last_undegraded":"2024-03-24T11:56:37.779541+0000","last_fullsized":"2024-03-24T11:56:37.779541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:56:21.012308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:21.012308+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.154184+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":220,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775772+0000","last_change":"2024-03-24T11:56:13.679738+0000","last_active":"2024-03-24T11:56:37.775772+0000","last_peered":"2024-03-24T11:56:37.775772+0000","last_clean":"2024-03-24T11:56:37.775772+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:56:37.775772+0000","last_undegraded":"2024-03-24T11:56:37.775772+0000","last_fullsized":"2024-03-24T11:56:37.775772+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:13.679609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:13.679609+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:42.607215+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776688+0000","last_change":"2024-03-24T11:56:00.196860+0000","last_active":"2024-03-24T11:56:37.776688+0000","last_peered":"2024-03-24T11:56:37.776688+0000","last_clean":"2024-03-24T11:56:37.776688+0000","last_became_active":"2024-03-24T11:52:15.134326+0000","last_became_peered":"2024-03-24T11:52:15.134326+0000","last_unstale":"2024-03-24T11:56:37.776688+0000","last_undegraded":"2024-03-24T11:56:37.776688+0000","last_fullsized":"2024-03-24T11:56:37.776688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:00.196663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_clean_scrub_stamp":"2024-03-24T11:56:00.196663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:08.616618+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"130'15","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779077+0000","last_change":"2024-03-24T11:56:28.036136+0000","last_active":"2024-03-24T11:56:37.779077+0000","last_peered":"2024-03-24T11:56:37.779077+0000","last_clean":"2024-03-24T11:56:37.779077+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:56:37.779077+0000","last_undegraded":"2024-03-24T11:56:37.779077+0000","last_fullsized":"2024-03-24T11:56:37.779077+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'15","last_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T11:56:28.035859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.724323+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"132'9563","reported_seq":9730,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.318648+0000","last_change":"2024-03-24T11:56:06.470793+0000","last_active":"2024-03-24T11:56:41.318648+0000","last_peered":"2024-03-24T11:56:41.318648+0000","last_clean":"2024-03-24T11:56:41.318648+0000","last_became_active":"2024-03-24T11:54:16.236885+0000","last_became_peered":"2024-03-24T11:54:16.236885+0000","last_unstale":"2024-03-24T11:56:41.318648+0000","last_undegraded":"2024-03-24T11:56:41.318648+0000","last_fullsized":"2024-03-24T11:56:41.318648+0000","mapping_epoch":97,"log_start":"109'3000","ondisk_log_start":"109'3000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7612","last_scrub_stamp":"2024-03-24T11:56:06.470609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.470609+0000","objects_scrubbed":7541,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:21.968168+0000","scrub_duration":2905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366280704,"num_objects":5589,"num_object_clones":0,"num_object_copies":16767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775342+0000","last_change":"2024-03-24T11:55:20.629992+0000","last_active":"2024-03-24T11:56:37.775342+0000","last_peered":"2024-03-24T11:56:37.775342+0000","last_clean":"2024-03-24T11:56:37.775342+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:56:37.775342+0000","last_undegraded":"2024-03-24T11:56:37.775342+0000","last_fullsized":"2024-03-24T11:56:37.775342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.629800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.629800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:42.680923+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774388+0000","last_change":"2024-03-24T11:56:06.310590+0000","last_active":"2024-03-24T11:56:37.774388+0000","last_peered":"2024-03-24T11:56:37.774388+0000","last_clean":"2024-03-24T11:56:37.774388+0000","last_became_active":"2024-03-24T11:52:19.387115+0000","last_became_peered":"2024-03-24T11:52:19.387115+0000","last_unstale":"2024-03-24T11:56:37.774388+0000","last_undegraded":"2024-03-24T11:56:37.774388+0000","last_fullsized":"2024-03-24T11:56:37.774388+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:06.310357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.310357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:29.666932+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775059+0000","last_change":"2024-03-24T11:56:18.582669+0000","last_active":"2024-03-24T11:56:37.775059+0000","last_peered":"2024-03-24T11:56:37.775059+0000","last_clean":"2024-03-24T11:56:37.775059+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:56:37.775059+0000","last_undegraded":"2024-03-24T11:56:37.775059+0000","last_fullsized":"2024-03-24T11:56:37.775059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:56:18.582550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:18.582550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:31.743441+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"116'25","reported_seq":188,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.779208+0000","last_change":"2024-03-24T11:55:19.864480+0000","last_active":"2024-03-24T11:56:37.779208+0000","last_peered":"2024-03-24T11:56:37.779208+0000","last_clean":"2024-03-24T11:56:37.779208+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:56:37.779208+0000","last_undegraded":"2024-03-24T11:56:37.779208+0000","last_fullsized":"2024-03-24T11:56:37.779208+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'25","last_scrub_stamp":"2024-03-24T11:55:19.864301+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T11:55:19.864301+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.864301+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:45.201869+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"132'9622","reported_seq":9703,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.408262+0000","last_change":"2024-03-24T11:54:16.237670+0000","last_active":"2024-03-24T11:56:41.408262+0000","last_peered":"2024-03-24T11:56:41.408262+0000","last_clean":"2024-03-24T11:56:41.408262+0000","last_became_active":"2024-03-24T11:54:16.237374+0000","last_became_peered":"2024-03-24T11:54:16.237374+0000","last_unstale":"2024-03-24T11:56:41.408262+0000","last_undegraded":"2024-03-24T11:56:41.408262+0000","last_fullsized":"2024-03-24T11:56:41.408262+0000","mapping_epoch":97,"log_start":"115'4100","ondisk_log_start":"115'4100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5522,"log_dups_size":0,"ondisk_log_size":5522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:45.404213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365297664,"num_objects":5574,"num_object_clones":0,"num_object_copies":16722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9622,"num_write_kb":486272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.783704+0000","last_change":"2024-03-24T11:55:18.299756+0000","last_active":"2024-03-24T11:56:37.783704+0000","last_peered":"2024-03-24T11:56:37.783704+0000","last_clean":"2024-03-24T11:56:37.783704+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:56:37.783704+0000","last_undegraded":"2024-03-24T11:56:37.783704+0000","last_fullsized":"2024-03-24T11:56:37.783704+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.299637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775167+0000","last_change":"2024-03-24T11:56:23.615139+0000","last_active":"2024-03-24T11:56:37.775167+0000","last_peered":"2024-03-24T11:56:37.775167+0000","last_clean":"2024-03-24T11:56:37.775167+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:56:37.775167+0000","last_undegraded":"2024-03-24T11:56:37.775167+0000","last_fullsized":"2024-03-24T11:56:37.775167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:23.614720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:23.614720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:38.993316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.775563+0000","last_change":"2024-03-24T11:56:14.639416+0000","last_active":"2024-03-24T11:56:37.775563+0000","last_peered":"2024-03-24T11:56:37.775563+0000","last_clean":"2024-03-24T11:56:37.775563+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:56:37.775563+0000","last_undegraded":"2024-03-24T11:56:37.775563+0000","last_fullsized":"2024-03-24T11:56:37.775563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.639264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:16.071252+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":183,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.777327+0000","last_change":"2024-03-24T11:55:19.105549+0000","last_active":"2024-03-24T11:56:37.777327+0000","last_peered":"2024-03-24T11:56:37.777327+0000","last_clean":"2024-03-24T11:56:37.777327+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:56:37.777327+0000","last_undegraded":"2024-03-24T11:56:37.777327+0000","last_fullsized":"2024-03-24T11:56:37.777327+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:55:19.105397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.105397+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.612766+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"132'9576","reported_seq":9741,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.294738+0000","last_change":"2024-03-24T11:56:05.111342+0000","last_active":"2024-03-24T11:56:41.294738+0000","last_peered":"2024-03-24T11:56:41.294738+0000","last_clean":"2024-03-24T11:56:41.294738+0000","last_became_active":"2024-03-24T11:54:16.237483+0000","last_became_peered":"2024-03-24T11:54:16.237483+0000","last_unstale":"2024-03-24T11:56:41.294738+0000","last_undegraded":"2024-03-24T11:56:41.294738+0000","last_fullsized":"2024-03-24T11:56:41.294738+0000","mapping_epoch":97,"log_start":"109'3000","ondisk_log_start":"109'3000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7549","last_scrub_stamp":"2024-03-24T11:56:05.111031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.111031+0000","objects_scrubbed":7494,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:05.209536+0000","scrub_duration":2574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366739456,"num_objects":5596,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9576,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":137,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781271+0000","last_change":"2024-03-24T11:55:01.085505+0000","last_active":"2024-03-24T11:56:37.781271+0000","last_peered":"2024-03-24T11:56:37.781271+0000","last_clean":"2024-03-24T11:56:37.781271+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:56:37.781271+0000","last_undegraded":"2024-03-24T11:56:37.781271+0000","last_fullsized":"2024-03-24T11:56:37.781271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.085396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.085396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.784140+0000","last_change":"2024-03-24T11:56:25.642344+0000","last_active":"2024-03-24T11:56:37.784140+0000","last_peered":"2024-03-24T11:56:37.784140+0000","last_clean":"2024-03-24T11:56:37.784140+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:56:37.784140+0000","last_undegraded":"2024-03-24T11:56:37.784140+0000","last_fullsized":"2024-03-24T11:56:37.784140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.642093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.642093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:41.995645+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776382+0000","last_change":"2024-03-24T11:56:19.421031+0000","last_active":"2024-03-24T11:56:37.776382+0000","last_peered":"2024-03-24T11:56:37.776382+0000","last_clean":"2024-03-24T11:56:37.776382+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:56:37.776382+0000","last_undegraded":"2024-03-24T11:56:37.776382+0000","last_fullsized":"2024-03-24T11:56:37.776382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:19.420806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:56:19.420806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.704616+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.781682+0000","last_change":"2024-03-24T11:56:02.340688+0000","last_active":"2024-03-24T11:56:37.781682+0000","last_peered":"2024-03-24T11:56:37.781682+0000","last_clean":"2024-03-24T11:56:37.781682+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:56:37.781682+0000","last_undegraded":"2024-03-24T11:56:37.781682+0000","last_fullsized":"2024-03-24T11:56:37.781682+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:02.340507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:02.340507+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:06.085699+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"132'9593","reported_seq":9755,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:41.258702+0000","last_change":"2024-03-24T11:56:11.635289+0000","last_active":"2024-03-24T11:56:41.258702+0000","last_peered":"2024-03-24T11:56:41.258702+0000","last_clean":"2024-03-24T11:56:41.258702+0000","last_became_active":"2024-03-24T11:54:16.239338+0000","last_became_peered":"2024-03-24T11:54:16.239338+0000","last_unstale":"2024-03-24T11:56:41.258702+0000","last_undegraded":"2024-03-24T11:56:41.258702+0000","last_fullsized":"2024-03-24T11:56:41.258702+0000","mapping_epoch":97,"log_start":"115'4500","ondisk_log_start":"115'4500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7969","last_scrub_stamp":"2024-03-24T11:56:11.635036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:11.635036+0000","objects_scrubbed":7267,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:18.008121+0000","scrub_duration":2413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365232128,"num_objects":5573,"num_object_clones":0,"num_object_copies":16719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9593,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.774971+0000","last_change":"2024-03-24T11:55:17.618704+0000","last_active":"2024-03-24T11:56:37.774971+0000","last_peered":"2024-03-24T11:56:37.774971+0000","last_clean":"2024-03-24T11:56:37.774971+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:56:37.774971+0000","last_undegraded":"2024-03-24T11:56:37.774971+0000","last_fullsized":"2024-03-24T11:56:37.774971+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:17.618606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T11:55:17.618606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:42.294065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776271+0000","last_change":"2024-03-24T11:56:21.558305+0000","last_active":"2024-03-24T11:56:37.776271+0000","last_peered":"2024-03-24T11:56:37.776271+0000","last_clean":"2024-03-24T11:56:37.776271+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:56:37.776271+0000","last_undegraded":"2024-03-24T11:56:37.776271+0000","last_fullsized":"2024-03-24T11:56:37.776271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:21.558180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:21.558180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.692594+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":178,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.776637+0000","last_change":"2024-03-24T11:55:19.645322+0000","last_active":"2024-03-24T11:56:37.776637+0000","last_peered":"2024-03-24T11:56:37.776637+0000","last_clean":"2024-03-24T11:56:37.776637+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:56:37.776637+0000","last_undegraded":"2024-03-24T11:56:37.776637+0000","last_fullsized":"2024-03-24T11:56:37.776637+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:55:19.645176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.645176+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:45.387352+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":211,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-24T11:56:37.780698+0000","last_change":"2024-03-24T11:55:02.415044+0000","last_active":"2024-03-24T11:56:37.780698+0000","last_peered":"2024-03-24T11:56:37.780698+0000","last_clean":"2024-03-24T11:56:37.780698+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:56:37.780698+0000","last_undegraded":"2024-03-24T11:56:37.780698+0000","last_fullsized":"2024-03-24T11:56:37.780698+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:55:02.414931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:02.414931+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:41.726917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5829623832,"num_objects":88954,"num_object_clones":0,"num_object_copies":266862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152956,"num_write_kb":7741057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17492422656,"data_stored":17492410440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88945,"ondisk_log_size":88945,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":260,"num_read_kb":255,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005113,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1859264,"kb_used_data":1787832,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91885888,"statfs":{"total":95995035648,"available":94091149312,"internally_reserved":0,"allocated":1830739968,"data_stored":1830384156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332160,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1872768,"kb_used_data":1801336,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91872384,"statfs":{"total":95995035648,"available":94077321216,"internally_reserved":0,"allocated":1844568064,"data_stored":1844212094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2240200,"kb_used_data":2150720,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":91504952,"statfs":{"total":95995035648,"available":93701070848,"internally_reserved":0,"allocated":2202337280,"data_stored":2201972153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2955396,"kb_used_data":2847868,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":90789756,"statfs":{"total":95995035648,"available":92968710144,"internally_reserved":0,"allocated":2916216832,"data_stored":2915855815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2247872,"kb_used_data":2148664,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91497280,"statfs":{"total":95995035648,"available":93693214720,"internally_reserved":0,"allocated":2200231936,"data_stored":2199875397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3368148,"kb_used_data":3220300,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":90377004,"statfs":{"total":95995035648,"available":92546052096,"internally_reserved":0,"allocated":3297587200,"data_stored":3297210611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851592,"kb_used_data":1776000,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91893560,"statfs":{"total":95995035648,"available":94099005440,"internally_reserved":0,"allocated":1818624000,"data_stored":1818258884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659204,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1519424,"kb_used_data":1452152,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92225728,"statfs":{"total":95995035648,"available":94439145472,"internally_reserved":0,"allocated":1487003648,"data_stored":1486646836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2912358400,"data_stored":2912354328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2185826304,"data_stored":2185822232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2186608640,"data_stored":2186608640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3268870144,"data_stored":3268870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1813118976,"data_stored":1813118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1474953216,"data_stored":1474953216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1829703680,"data_stored":1829699608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1820983296,"data_stored":1820983296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:56:45.236 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [0, 1, 7] 2024-03-24T11:56:45.237 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.0', '0', '1', '7'] 2024-03-24T11:56:45.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 2.0 0 1 7 2024-03-24T11:56:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:45.254 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:45.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:45.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:56:45.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:45.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:45.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:45.541 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.34742.0:157266 7.a e134/97 7:5019477d:::benchmark_data_smithi064_87013_object36309:head v 134'13601, mlcod=134'13599)", 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:45.540713+0000", 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.00095691199999999998, 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00098922700000000008, 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:45.542 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540713+0000", 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540713+0000", 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540713+0000", 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.58e-07 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540719+0000", 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4200000000000004e-06 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540730+0000", 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0390999999999999e-05 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540735+0000", 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7369999999999997e-06 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.544 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540853+0000", 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011862 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540886+0000", 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3303000000000003e-05 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.34742.0:157268 7.a e134/97)", 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:45.540749+0000", 2024-03-24T11:56:45.545 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.00092026100000000002, 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0010108210000000001, 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "queued for pg", 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540749+0000", 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540749+0000", 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.546 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540750+0000", 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.75e-07 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540754+0000", 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1590000000000003e-06 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540759+0000", 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4319999999999998e-06 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:56:45.547 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:45.540764+0000", 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7740000000000004e-06 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 2 2024-03-24T11:56:45.548 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:56:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:45.549 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:45.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:45.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:45.677 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:45.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:45.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:45.840 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:45.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:45.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:45.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:46.051 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:46.061 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:46.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:46.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:46.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:46.292 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:46.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:46.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T11:56:46.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:46.436 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:46.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:46.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:46.570 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:46.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:46.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:46.687 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:46.713 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:56:46.722 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:46.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:46.760 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:56:46.805 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:56:46.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:46.849 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:56:46.863 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:46.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:46.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:46.888 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:56:46.929 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:56:46.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:46.968 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:56:47.005 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:47.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:47.007 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:56:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:56:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:56:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:56:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:56:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:56:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:56:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:56:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:56:47.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:46 smithi064 ceph-mon[30223]: pgmap v171: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 688 B/s rd, 0 B/s wr, 608 op/s 2024-03-24T11:56:47.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:46 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:56:47.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:46 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:56:47.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:46 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4127430195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:47.020 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:46 smithi064 ceph-mon[42243]: pgmap v171: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 688 B/s rd, 0 B/s wr, 608 op/s 2024-03-24T11:56:47.020 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:46 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:56:47.020 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:46 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:56:47.020 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:46 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4127430195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:47.061 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:56:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:56:47.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:46 smithi193 ceph-mon[33593]: pgmap v171: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 688 B/s rd, 0 B/s wr, 608 op/s 2024-03-24T11:56:47.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:46 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:56:47.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:46 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:56:47.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:46 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4127430195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:47.107 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:47.141 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:47.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:47.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:47.290 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:47.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:47.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:47.430 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:47.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:47.532 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:47.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:56:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:56:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:47.561 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:47.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:47.663 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:47.693 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:47.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:47.795 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:47.827 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:47.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:47.928 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:47.957 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:47.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:48.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:47 smithi064 ceph-mon[30223]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:56:48.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:47 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:48.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:47 smithi064 ceph-mon[42243]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:56:48.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:47 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:48.060 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:48.090 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:48.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:48.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:47 smithi193 ceph-mon[33593]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:56:48.148 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:47 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:48.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:48.236 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:48.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:48.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:48.370 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:48.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:48.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:48.521 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:48.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:48.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:48.653 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:48.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:48.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:48.794 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:48.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:48.853 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:56:48.854 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:56:48.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:48.939 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:48.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:49.041 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:49.071 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:49.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:48 smithi193 ceph-mon[33593]: pgmap v173: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:49.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:49.217 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:49.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:49.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:48 smithi064 ceph-mon[30223]: pgmap v173: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:48 smithi064 ceph-mon[42243]: pgmap v173: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:49.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:49.359 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:49.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:49.460 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:49.489 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:49.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:49.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:49.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:49.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:49.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T11:56:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:49.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:49.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:49.909 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:49.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:49.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:50.011 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:50.051 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:50.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:50.085 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:50.151 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:50.197 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:50.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:50.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:50.339 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:50.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:50.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:50.481 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:50.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:50.583 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:50.621 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:50.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:50.723 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:50.758 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:50.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:50.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:50.888 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:50.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:50.989 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:51.022 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:51.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:51.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:50 smithi193 ceph-mon[33593]: pgmap v174: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 690 op/s 2024-03-24T11:56:51.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:51.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:50 smithi064 ceph-mon[30223]: pgmap v174: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 690 op/s 2024-03-24T11:56:51.137 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:50 smithi064 ceph-mon[42243]: pgmap v174: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 690 op/s 2024-03-24T11:56:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:51.173 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:51.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:51.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:51.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:51.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:51.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:51.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:56:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:51.455 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:51.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:51.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:51.584 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:51.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:51.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:51.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:51.740 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:51.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:51.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:51.904 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:51.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:51.979 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:52.005 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:52.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:52.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:52.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:52.168 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:52.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:52.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:52.311 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:52.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:52.352 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:56:52.353 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:56:52.413 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:52.442 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:52.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:52.544 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:52.574 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:52.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:52.675 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:52.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:52.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:52.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:52.851 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:52.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:52.878 INFO:teuthology.orchestra.run.smithi064.stderr:set 2.0 pg_upmap mapping to [0,1,7] 2024-03-24T11:56:52.884 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T11:56:52.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:52.991 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:52.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:52 smithi193 ceph-mon[33593]: pgmap v175: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-03-24T11:56:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:52 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3433297376' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [0, 1, 7]}]: dispatch 2024-03-24T11:56:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:52 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1681393177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:56:53.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:53.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:53.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:53.185 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T11:56:53.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:53.265 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:53.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:52 smithi064 ceph-mon[30223]: pgmap v175: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-03-24T11:56:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3433297376' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [0, 1, 7]}]: dispatch 2024-03-24T11:56:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1681393177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:56:53.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:52 smithi064 ceph-mon[42243]: pgmap v175: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-03-24T11:56:53.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:52 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3433297376' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [0, 1, 7]}]: dispatch 2024-03-24T11:56:53.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:52 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1681393177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:56:53.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:53.409 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:53.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:53.510 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:53.533 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:53.551 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:53.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:53.652 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:53.681 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:53.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:53.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:53.827 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:53.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:53.929 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:53.968 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:53.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:54.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:53 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:54.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3433297376' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [0, 1, 7]}]': finished 2024-03-24T11:56:54.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1681393177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:56:54.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:53 smithi193 ceph-mon[33593]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:56:54.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:54.100 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:54.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:54.148 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:53 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:54.148 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3433297376' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [0, 1, 7]}]': finished 2024-03-24T11:56:54.148 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1681393177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:56:54.148 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:53 smithi064 ceph-mon[30223]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:56:54.149 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:53 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:56:54.149 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3433297376' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [0, 1, 7]}]': finished 2024-03-24T11:56:54.149 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1681393177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:56:54.149 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:53 smithi064 ceph-mon[42243]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:56:54.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:54.242 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:54.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:54.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:56:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:54.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:56:54.488 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:54.518 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:54.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:54.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:54.662 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:54.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:54.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:54.792 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:54.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:54.894 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:54.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:54.922 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:54.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:54.962 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:56:54.962 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:56:54.962 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[30223]: pgmap v177: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[30223]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[42243]: pgmap v177: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T11:56:54.963 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:54 smithi064 ceph-mon[42243]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:56:55.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:55.064 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:55.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:55.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:55.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:54 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:56:55.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:54 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:56:55.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:54 smithi193 ceph-mon[33593]: pgmap v177: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:55.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:54 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T11:56:55.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:54 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T11:56:55.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:54 smithi193 ceph-mon[33593]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:56:55.202 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:55.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:55.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:55.343 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:55.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:55.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:55.487 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:55.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:55.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:55.620 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:55.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:55.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:55.755 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:55.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:55.776 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:56:55.822 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:56:55.822 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:56:55.823 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:56:55.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:55.866 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:56:55.885 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:55.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:55.912 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:56:55.913 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T11:56:55.913 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T11:56:55.913 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T11:56:55.913 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T11:56:55.913 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T11:56:55.913 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2807993391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:55.914 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:55 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2807993391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:55.953 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:56:55.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:55.991 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:56:56.030 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:56.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:56.031 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:56:56.070 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:56:56.074 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:56:56.074 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:56:56.074 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:56:56.074 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:56:56.074 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:56:56.074 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:56:56.075 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:56:56.075 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:56:56.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:55 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T11:56:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:55 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T11:56:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:55 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T11:56:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:55 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T11:56:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:55 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T11:56:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:55 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T11:56:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:55 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2807993391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:56:56.132 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:56.160 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:56.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:56.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:56.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:56.289 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:56.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:56.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:56.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:56.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:56.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:56.568 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:56.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:56.670 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:56.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T11:56:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:56.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:56.801 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:56.834 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:56.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:56.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:56.975 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:56.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:57.028 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:56:57.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[30223]: pgmap v179: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 680 op/s 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[30223]: 7.6 scrub starts 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[30223]: 7.a scrub starts 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[30223]: 2.2 scrub starts 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[30223]: 2.2 scrub ok 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[42243]: pgmap v179: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 680 op/s 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[42243]: 7.6 scrub starts 2024-03-24T11:56:57.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T11:56:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T11:56:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[42243]: 7.a scrub starts 2024-03-24T11:56:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[42243]: 2.2 scrub starts 2024-03-24T11:56:57.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:56 smithi064 ceph-mon[42243]: 2.2 scrub ok 2024-03-24T11:56:57.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:57.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:57.107 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:57.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:57.160 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:56:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:56:57.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:56 smithi193 ceph-mon[33593]: pgmap v179: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 680 op/s 2024-03-24T11:56:57.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:56 smithi193 ceph-mon[33593]: 7.6 scrub starts 2024-03-24T11:56:57.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:56 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T11:56:57.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:56 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T11:56:57.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:56 smithi193 ceph-mon[33593]: 7.a scrub starts 2024-03-24T11:56:57.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:56 smithi193 ceph-mon[33593]: 2.2 scrub starts 2024-03-24T11:56:57.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:56 smithi193 ceph-mon[33593]: 2.2 scrub ok 2024-03-24T11:56:57.208 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:57.247 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:57.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:57.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:57.384 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:57.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:57.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:57.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:56:57.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:57.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:57.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:57.645 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:57.689 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:57.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:57.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:57.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:56:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:56:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:57.817 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:57.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:57.883 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:56:57.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:57.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:58.079 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:58.109 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:58.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:58.199 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:57 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:58.199 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:57 smithi064 ceph-mon[30223]: 7.6 scrub ok 2024-03-24T11:56:58.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:57 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:58.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:57 smithi064 ceph-mon[42243]: 7.6 scrub ok 2024-03-24T11:56:58.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:56:58.238 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:58.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:56:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:57 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:57 smithi193 ceph-mon[33593]: 7.6 scrub ok 2024-03-24T11:56:58.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:58.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:58.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:58.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:58.515 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:58.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:58.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:58.655 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:58.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:58.757 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:58.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:56:58.786 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:58.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:58.888 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:58.918 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:58.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:59.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:56:59.081 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:59.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:56:59.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:58 smithi193 ceph-mon[33593]: pgmap v180: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:59.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:58 smithi193 ceph-mon[33593]: 7.a scrub ok 2024-03-24T11:56:59.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:56:59.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:59.238 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:59.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:56:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:58 smithi064 ceph-mon[30223]: pgmap v180: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:58 smithi064 ceph-mon[30223]: 7.a scrub ok 2024-03-24T11:56:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:58 smithi064 ceph-mon[42243]: pgmap v180: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:56:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:58 smithi064 ceph-mon[42243]: 7.a scrub ok 2024-03-24T11:56:59.306 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:56:59.307 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T11:56:59.307 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T11:56:59.307 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T11:56:59.307 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:56:59.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:56:59.383 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:56:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:59.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:56:59.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:56:59.515 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:59.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:56:59.617 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:56:59.648 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:59.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:56:59.745 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:56:59.746 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:56:59.746 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-24T11:56:59.746 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T11:56:59.746 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T11:56:59.746 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-24T11:56:59.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:59.828 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:59.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:56:59.929 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:56:59.966 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:56:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:56:59.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:00.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:59 smithi193 ceph-mon[33593]: 4.1 scrub starts 2024-03-24T11:57:00.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:56:59 smithi193 ceph-mon[33593]: 4.1 scrub ok 2024-03-24T11:57:00.113 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:00.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:00.215 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:00.245 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:00.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:59 smithi064 ceph-mon[30223]: 4.1 scrub starts 2024-03-24T11:57:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:56:59 smithi064 ceph-mon[30223]: 4.1 scrub ok 2024-03-24T11:57:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:59 smithi064 ceph-mon[42243]: 4.1 scrub starts 2024-03-24T11:57:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:56:59 smithi064 ceph-mon[42243]: 4.1 scrub ok 2024-03-24T11:57:00.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:00.390 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:00.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:00.492 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:00.522 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:00.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:00.613 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:57:00.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:00.659 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T11:57:00.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:00.666 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:00.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:00.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:00.802 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:00.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:00.903 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:00.948 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:00.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:01.049 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[30223]: pgmap v181: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[42243]: pgmap v181: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T11:57:01.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:00 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T11:57:01.078 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:01.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:01.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:01.186 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:00 smithi193 ceph-mon[33593]: pgmap v181: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:57:01.186 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:00 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T11:57:01.187 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:00 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T11:57:01.187 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:00 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T11:57:01.187 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:00 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T11:57:01.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:01.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:01.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:01.366 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:01.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:01.442 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:57:01.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:01.504 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:01.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:01.606 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:01.634 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:01.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:01.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:01.777 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:01.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:01.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:01.912 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:01.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:02.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:02.054 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:02.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:02.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:01 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T11:57:02.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:01 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T11:57:02.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:01 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T11:57:02.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:01 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T11:57:02.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:01 smithi193 ceph-mon[33593]: 2.18 scrub starts 2024-03-24T11:57:02.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:01 smithi193 ceph-mon[33593]: 2.18 scrub ok 2024-03-24T11:57:02.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:02.185 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:02.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:02.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T11:57:02.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T11:57:02.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T11:57:02.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T11:57:02.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[30223]: 2.18 scrub starts 2024-03-24T11:57:02.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[30223]: 2.18 scrub ok 2024-03-24T11:57:02.285 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T11:57:02.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T11:57:02.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T11:57:02.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T11:57:02.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[42243]: 2.18 scrub starts 2024-03-24T11:57:02.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:01 smithi064 ceph-mon[42243]: 2.18 scrub ok 2024-03-24T11:57:02.287 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:02.317 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:02.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:02.419 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:02.449 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:02.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:02.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:02.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:02.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:02.757 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:02.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:02.823 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:02.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:02.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:02.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:03.028 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:03.057 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:03.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:03.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: pgmap v182: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 491 B/s rd, 0 B/s wr, 854 op/s 2024-03-24T11:57:03.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T11:57:03.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T11:57:03.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T11:57:03.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: 6.17 scrub starts 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: 6.17 scrub ok 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: 6.d scrub starts 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[30223]: 6.d scrub ok 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: pgmap v182: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 491 B/s rd, 0 B/s wr, 854 op/s 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: 6.17 scrub starts 2024-03-24T11:57:03.111 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: 6.17 scrub ok 2024-03-24T11:57:03.112 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: 6.d scrub starts 2024-03-24T11:57:03.112 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:02 smithi064 ceph-mon[42243]: 6.d scrub ok 2024-03-24T11:57:03.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:03.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:03.310 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:03.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: pgmap v182: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 491 B/s rd, 0 B/s wr, 854 op/s 2024-03-24T11:57:03.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T11:57:03.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T11:57:03.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T11:57:03.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T11:57:03.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: 6.17 scrub starts 2024-03-24T11:57:03.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: 6.17 scrub ok 2024-03-24T11:57:03.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: 6.d scrub starts 2024-03-24T11:57:03.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:02 smithi193 ceph-mon[33593]: 6.d scrub ok 2024-03-24T11:57:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:03.341 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:03.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:03.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:03.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:03.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:03.496 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:57:03.497 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "" 2024-03-24T11:57:03.497 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:57:03.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:03.625 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:03.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:03.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:03.757 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:03.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:03.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:03.890 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:03.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:03.967 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T11:57:03.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:04.030 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:04.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:04.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:04.159 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:04.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:04.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T11:57:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T11:57:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T11:57:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T11:57:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1283440408' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T11:57:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T11:57:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T11:57:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T11:57:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:03 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1283440408' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:04.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:04.304 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:04.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:03 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T11:57:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:03 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T11:57:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:03 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T11:57:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:03 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T11:57:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:03 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1283440408' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:04.405 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:04.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:04.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:04.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:04.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:04.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:04.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:04.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:04.710 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:04.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:04.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:04.854 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:04.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:04.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:04.980 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:04.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:04.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:05.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:05.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:05.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:05.266 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:05.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: pgmap v183: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1283440408' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:57:05.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[30223]: 7.9 scrub starts 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: pgmap v183: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1283440408' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:57:05.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:04 smithi064 ceph-mon[42243]: 7.9 scrub starts 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:04 smithi193 ceph-mon[33593]: pgmap v183: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1283440408' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:57:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: 7.9 scrub starts 2024-03-24T11:57:05.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:05.397 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:05.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:05.499 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:05.527 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:05.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:05.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:05.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:05.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:05.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:05.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:05.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:05.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:06.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:05 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T11:57:06.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:05 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T11:57:06.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:05 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T11:57:06.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:05 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T11:57:06.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T11:57:06.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:05 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T11:57:06.074 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:06.108 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:06.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:06.209 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:06.241 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:06.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:06.330 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:57:06.343 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:06.370 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:57:06.374 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:06.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28186 7.7 7:e54252ff:::benchmark_data_smithi064_87013_object28185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.185466+0000", 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 139.21957047399999, 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16022612, 2024-03-24T11:57:06.407 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28186 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185466+0000", 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185466+0000", 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185468+0000", 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.886e-06 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185507+0000", 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9397999999999998e-05 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185517+0000", 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.005e-05 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185524+0000", 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.328e-06 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185583+0000", 2024-03-24T11:57:06.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8666999999999999e-05 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185742+0000", 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015933700000000001 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185936+0000", 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019364800000000001 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188116+0000", 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021797510000000002 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.339991+0000", 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.151875807 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345426+0000", 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054349259999999996 2024-03-24T11:57:06.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345654+0000", 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022796399999999999 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345692+0000", 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7357999999999997e-05 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.413 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28187 7.7 7:ebde3806:::benchmark_data_smithi064_87013_object28186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.185544+0000", 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 139.219492479, 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16043975899999999, 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28187 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185544+0000", 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185544+0000", 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185544+0000", 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5899999999999995e-07 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185630+0000", 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5143999999999997e-05 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185637+0000", 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9820000000000002e-06 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185641+0000", 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.882e-06 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187084+0000", 2024-03-24T11:57:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014425 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187210+0000", 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012592300000000001 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187345+0000", 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000134647 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188284+0000", 2024-03-24T11:57:06.417 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000939368 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.340027+0000", 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15174342199999999 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345915+0000", 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0058873720000000001 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345958+0000", 2024-03-24T11:57:06.418 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3513000000000002e-05 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345983+0000", 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5147000000000002e-05 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28192 7.c 7:3adf894d:::benchmark_data_smithi064_87013_object28191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.188462+0000", 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 139.21657391700001, 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15559229499999999, 2024-03-24T11:57:06.419 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28192 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.420 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188462+0000", 2024-03-24T11:57:06.421 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.421 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.421 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188462+0000", 2024-03-24T11:57:06.421 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.422 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.422 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188464+0000", 2024-03-24T11:57:06.422 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6050000000000001e-06 2024-03-24T11:57:06.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.422 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188526+0000", 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2663000000000006e-05 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188536+0000", 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5349999999999993e-06 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188542+0000", 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9179999999999999e-06 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.423 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188599+0000", 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6616000000000001e-05 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188689+0000", 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0540999999999999e-05 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188776+0000", 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6911999999999997e-05 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.424 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.189737+0000", 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00096098199999999998 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.190043+0000", 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00030576399999999998 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.343860+0000", 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15381721500000001 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.344001+0000", 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014119999999999999 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.344054+0000", 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3344000000000003e-05 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28593 7.c 7:30603fbe:::benchmark_data_smithi064_87013_object28592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.547265+0000", 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 138.85777144900001, 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15797111, 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28593 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547265+0000", 2024-03-24T11:57:06.427 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547265+0000", 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547266+0000", 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.463e-06 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547316+0000", 2024-03-24T11:57:06.428 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9428999999999997e-05 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547326+0000", 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0365e-05 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547333+0000", 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0029999999999997e-06 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547374+0000", 2024-03-24T11:57:06.429 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1217999999999997e-05 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547462+0000", 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7540999999999994e-05 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547546+0000", 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4585000000000004e-05 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548738+0000", 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0011919260000000001 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548948+0000", 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000209495 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705156+0000", 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15620814699999999 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.431 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705194+0000", 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8552e-05 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705236+0000", 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1386e-05 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28597 7.7 7:e8c53c18:::benchmark_data_smithi064_87013_object28596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.547932+0000", 2024-03-24T11:57:06.432 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 138.85710402000001, 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15730358899999999, 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28597 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547932+0000", 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.434 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.434 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547932+0000", 2024-03-24T11:57:06.434 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.434 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.434 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.434 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547933+0000", 2024-03-24T11:57:06.434 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.251e-06 2024-03-24T11:57:06.435 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:57:06.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.435 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547980+0000", 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.71e-05 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547991+0000", 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.059e-05 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547999+0000", 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7309999999999993e-06 2024-03-24T11:57:06.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548053+0000", 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4187999999999997e-05 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548136+0000", 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3467000000000006e-05 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548231+0000", 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4534e-05 2024-03-24T11:57:06.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.549937+0000", 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017055709999999999 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552361+0000", 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024243770000000001 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.438 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705177+0000", 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.152815912 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705208+0000", 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1489999999999998e-05 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.439 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705236+0000", 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7378000000000001e-05 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28602 7.c 7:38c9ee92:::benchmark_data_smithi064_87013_object28601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.550423+0000", 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 138.854613392, 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.161645607, 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.440 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28602 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550423+0000", 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550423+0000", 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.441 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550424+0000", 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4309999999999999e-06 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550479+0000", 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5044999999999999e-05 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550488+0000", 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7180000000000002e-06 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.442 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550495+0000", 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4749999999999996e-06 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550540+0000", 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4774000000000002e-05 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550626+0000", 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5971999999999996e-05 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.443 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550717+0000", 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0962000000000001e-05 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.551765+0000", 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0010477710000000001 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552014+0000", 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024906499999999997 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.444 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.711966+0000", 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15995161299999999 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712024+0000", 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8112999999999997e-05 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712068+0000", 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4668e-05 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28607 7.c 7:3d17d51d:::benchmark_data_smithi064_87013_object28606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.552587+0000", 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 138.852449017, 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15961876999999999, 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28607 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552587+0000", 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552587+0000", 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552589+0000", 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.618e-06 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552646+0000", 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7494999999999998e-05 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552658+0000", 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1987999999999999e-05 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552666+0000", 2024-03-24T11:57:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3370000000000002e-06 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552710+0000", 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4261999999999997e-05 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552807+0000", 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7011999999999999e-05 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552891+0000", 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4592000000000006e-05 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.554174+0000", 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001282748 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.554186+0000", 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1783999999999999e-05 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712113+0000", 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15792745499999999 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712164+0000", 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0955999999999998e-05 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712206+0000", 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1523000000000002e-05 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:30151 7.7 7:e93669ad:::benchmark_data_smithi064_87013_object30150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:48.734184+0000", 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 137.67085207299999, 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.12013056799999999, 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 30151 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.734184+0000", 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.734184+0000", 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.734185+0000", 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.057e-06 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.734261+0000", 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5518000000000004e-05 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.763349+0000", 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029088395999999999 2024-03-24T11:57:06.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.763364+0000", 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.465e-05 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.763690+0000", 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00032602899999999998 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.763812+0000", 2024-03-24T11:57:06.455 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012279899999999999 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.763974+0000", 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000161607 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766466+0000", 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024914730000000001 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.456 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.766535+0000", 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9750000000000001e-05 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.854205+0000", 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.087669340999999998 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.854264+0000", 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9632000000000001e-05 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:48.854315+0000", 2024-03-24T11:57:06.457 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0315999999999997e-05 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:65830 7.c 7:35fc51e6:::benchmark_data_smithi064_87013_object65829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:12.717425+0000", 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 113.687610873, 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14593989900000001, 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.458 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 65830 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717425+0000", 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717425+0000", 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717427+0000", 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.796e-06 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717482+0000", 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5118999999999999e-05 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717499+0000", 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6694999999999999e-05 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717507+0000", 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.49e-06 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717553+0000", 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5330999999999997e-05 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717647+0000", 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4835999999999998e-05 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717743+0000", 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5464000000000006e-05 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.719456+0000", 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017131799999999999 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.719864+0000", 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00040831899999999998 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.462 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.863247+0000", 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14338300300000001 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.863312+0000", 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4553000000000003e-05 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.863365+0000", 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3112999999999998e-05 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.463 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67059 7.7 7:e00040fd:::benchmark_data_smithi064_87013_object67058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:13.875611+0000", 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.529424976, 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.18307747699999999, 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67059 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.464 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875611+0000", 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875611+0000", 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875612+0000", 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0809999999999999e-06 2024-03-24T11:57:06.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875685+0000", 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2611e-05 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875694+0000", 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3519999999999996e-06 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875700+0000", 2024-03-24T11:57:06.466 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0619999999999997e-06 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875751+0000", 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0868999999999997e-05 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875873+0000", 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000122045 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876004+0000", 2024-03-24T11:57:06.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000131027 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878138+0000", 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021343310000000002 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.881941+0000", 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0038020810000000001 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.058578+0000", 2024-03-24T11:57:06.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.176637233 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.058641+0000", 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2897000000000004e-05 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.058689+0000", 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7888e-05 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.469 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67060 7.c 7:37ae5022:::benchmark_data_smithi064_87013_object67059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:13.876645+0000", 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.52839151800001, 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17569818300000001, 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67060 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876645+0000", 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876645+0000", 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876646+0000", 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1310000000000001e-06 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876726+0000", 2024-03-24T11:57:06.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0137999999999997e-05 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876738+0000", 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2257e-05 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876746+0000", 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7700000000000001e-06 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876803+0000", 2024-03-24T11:57:06.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7612999999999999e-05 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876893+0000", 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0004000000000001e-05 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876978+0000", 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4778999999999996e-05 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878831+0000", 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018530280000000001 2024-03-24T11:57:06.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.879123+0000", 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00029211500000000002 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.052232+0000", 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.173108807 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.052302+0000", 2024-03-24T11:57:06.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0028000000000001e-05 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.052343+0000", 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0513000000000003e-05 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67063 7.7 7:e32fc8d7:::benchmark_data_smithi064_87013_object67062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:13.878827+0000", 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.52620942, 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.182956279, 2024-03-24T11:57:06.475 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67063 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878827+0000", 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878827+0000", 2024-03-24T11:57:06.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878828+0000", 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.793e-06 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878863+0000", 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4944999999999998e-05 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878874+0000", 2024-03-24T11:57:06.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0152e-05 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878882+0000", 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9619999999999993e-06 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878920+0000", 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8940999999999999e-05 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.879015+0000", 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4102000000000001e-05 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.879142+0000", 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012751400000000001 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.880753+0000", 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001610956 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.024206+0000", 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.143453308 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.061630+0000", 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037423374000000002 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.061742+0000", 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011199700000000001 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.061783+0000", 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1235000000000001e-05 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67336 7.d 7:b31b3432:::benchmark_data_smithi064_87013_object67335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.241068+0000", 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.163967927, 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.23462398400000001, 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.481 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67336 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241068+0000", 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241068+0000", 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241070+0000", 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.702e-06 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241167+0000", 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6700000000000006e-05 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241181+0000", 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4052e-05 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241200+0000", 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9684e-05 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241253+0000", 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2281000000000003e-05 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241375+0000", 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012239799999999999 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.484 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241535+0000", 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016008099999999999 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244010+0000", 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024753790000000002 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244856+0000", 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00084533399999999999 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.485 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475606+0000", 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.230749913 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475658+0000", 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2725000000000001e-05 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475692+0000", 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3735000000000001e-05 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.486 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.487 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:06.488 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67344 7.7 7:e9b6c53c:::benchmark_data_smithi064_87013_object67343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.244558+0000", 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.16047824100001, 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.128471002, 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.489 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67344 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244558+0000", 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244558+0000", 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244562+0000", 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1590000000000003e-06 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244608+0000", 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5840000000000002e-05 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244617+0000", 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.363e-06 2024-03-24T11:57:06.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244623+0000", 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5459999999999999e-06 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244682+0000", 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8757e-05 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244828+0000", 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000146114 2024-03-24T11:57:06.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244923+0000", 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5341999999999996e-05 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248157+0000", 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032344140000000001 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.249552+0000", 2024-03-24T11:57:06.493 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013948490000000001 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.372957+0000", 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.12340505 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.372995+0000", 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.731e-05 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.373029+0000", 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4258000000000003e-05 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67345 7.7 7:ed1efbc8:::benchmark_data_smithi064_87013_object67344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.244668+0000", 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.160368074, 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13090933799999999, 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.495 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67345 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244668+0000", 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244668+0000", 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.496 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244669+0000", 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3799999999999996e-07 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244711+0000", 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1737999999999999e-05 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244718+0000", 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9670000000000004e-06 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.497 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244726+0000", 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3240000000000003e-06 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.246631+0000", 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0019046449999999999 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.246739+0000", 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000108255 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.498 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.246827+0000", 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7949999999999996e-05 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248266+0000", 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001439552 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.249776+0000", 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0015096510000000001 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.499 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.375474+0000", 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.125697846 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.375532+0000", 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8628999999999997e-05 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.375577+0000", 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4943e-05 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.500 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67346 7.d 7:b983acce:::benchmark_data_smithi064_87013_object67345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.244737+0000", 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.16029872, 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.23104503000000001, 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67346 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.501 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244737+0000", 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244737+0000", 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244738+0000", 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9100000000000003e-07 2024-03-24T11:57:06.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244768+0000", 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.987e-05 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244775+0000", 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0790000000000003e-06 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244783+0000", 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9240000000000007e-06 2024-03-24T11:57:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244881+0000", 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8303999999999999e-05 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244971+0000", 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0192999999999995e-05 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.245064+0000", 2024-03-24T11:57:06.504 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2862999999999995e-05 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.246920+0000", 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018555080000000001 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248033+0000", 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001113492 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475728+0000", 2024-03-24T11:57:06.505 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.22769488399999999 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475759+0000", 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0743e-05 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475782+0000", 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3479e-05 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.506 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67349 7.d 7:bb7ecb5e:::benchmark_data_smithi064_87013_object67348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.248333+0000", 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.156702753, 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.22997953400000001, 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67349 2024-03-24T11:57:06.507 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248333+0000", 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248333+0000", 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248335+0000", 2024-03-24T11:57:06.508 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.691e-06 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248394+0000", 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8768999999999999e-05 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248404+0000", 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0180000000000001e-05 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248412+0000", 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8839999999999994e-06 2024-03-24T11:57:06.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248458+0000", 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6019000000000001e-05 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248536+0000", 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8510999999999994e-05 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248624+0000", 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7881999999999996e-05 2024-03-24T11:57:06.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.250421+0000", 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001797079 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.251047+0000", 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00062609000000000004 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.478197+0000", 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.227149829 2024-03-24T11:57:06.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.478258+0000", 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0225e-05 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.478313+0000", 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5374999999999998e-05 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.512 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67354 7.7 7:edc7c1d6:::benchmark_data_smithi064_87013_object67353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.252428+0000", 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 112.152608492, 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.21488665200000001, 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67354 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252428+0000", 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.513 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252428+0000", 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252428+0000", 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7199999999999997e-07 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252486+0000", 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7927000000000003e-05 2024-03-24T11:57:06.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252495+0000", 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.952e-06 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252500+0000", 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7779999999999998e-06 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252541+0000", 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1078000000000002e-05 2024-03-24T11:57:06.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252625+0000", 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4281000000000002e-05 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252715+0000", 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9786999999999999e-05 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.254391+0000", 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00167537 2024-03-24T11:57:06.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.254646+0000", 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025550399999999999 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.467237+0000", 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.21259097299999999 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.467280+0000", 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3071999999999997e-05 2024-03-24T11:57:06.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.467314+0000", 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4057999999999998e-05 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:128882 7.d 7:b28c9fde:::benchmark_data_smithi064_87013_object7925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:13.800816+0000", 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 52.604220298000001, 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.119697076, 2024-03-24T11:57:06.518 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 128882 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.800816+0000", 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.519 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.800816+0000", 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.800818+0000", 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9980000000000002e-06 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.800823+0000", 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1699999999999996e-06 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.520 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.800835+0000", 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1867e-05 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.800843+0000", 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4829999999999999e-06 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.800892+0000", 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8560999999999998e-05 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.521 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.800969+0000", 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7411999999999997e-05 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.801039+0000", 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0060000000000003e-05 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.804288+0000", 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032486860000000002 2024-03-24T11:57:06.522 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.920282+0000", 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.115993623 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.920431+0000", 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014940900000000001 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.920474+0000", 2024-03-24T11:57:06.523 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3101e-05 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:13.920513+0000", 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8705999999999998e-05 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:129026 7.c 7:3c24f0a8:::benchmark_data_smithi064_87013_object8069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:56:14.027065+0000", 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 52.377971174999999, 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.121132375, 2024-03-24T11:57:06.524 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 129026 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.027065+0000", 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:57:06.525 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.027065+0000", 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.027067+0000", 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0200000000000001e-06 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.027073+0000", 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6400000000000002e-06 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.526 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.027084+0000", 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1685000000000001e-05 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.027091+0000", 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4740000000000002e-06 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.037092+0000", 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010001075999999999 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.527 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.037211+0000", 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000118952 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.037300+0000", 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9634000000000003e-05 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.041085+0000", 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0037843880000000001 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.528 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.041323+0000", 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000237982 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.148043+0000", 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.106719751 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.148139+0000", 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6249000000000006e-05 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:57:06.529 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:57:06.530 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:56:14.148197+0000", 2024-03-24T11:57:06.530 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8523999999999997e-05 2024-03-24T11:57:06.530 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.530 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.530 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.530 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:57:06.530 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:57:06.530 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:57:06.531 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:57:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:06.532 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:06.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:06.536 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:57:06.576 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:57:06.620 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:57:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:57:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:57:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:57:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:57:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:57:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:57:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:57:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:57:06.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:06.673 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:06.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:06.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:06.811 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:06.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:06.912 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:06.940 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:06.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:06.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:57:07.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:07.083 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:07.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:07.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: pgmap v185: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 4.2 deep-scrub starts 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 4.2 deep-scrub ok 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 7.9 scrub ok 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2586316823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T11:57:07.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:06 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T11:57:07.185 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:07.213 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:07.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:07.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:07.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T11:57:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: pgmap v185: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T11:57:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 4.2 deep-scrub starts 2024-03-24T11:57:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 4.2 deep-scrub ok 2024-03-24T11:57:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 7.9 scrub ok 2024-03-24T11:57:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T11:57:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T11:57:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T11:57:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T11:57:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2586316823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T11:57:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T11:57:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: pgmap v185: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T11:57:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 4.2 deep-scrub starts 2024-03-24T11:57:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 4.2 deep-scrub ok 2024-03-24T11:57:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 7.9 scrub ok 2024-03-24T11:57:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T11:57:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T11:57:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T11:57:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T11:57:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2586316823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:07.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T11:57:07.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:06 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T11:57:07.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:07.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:07.356 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:07.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:07.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:07.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:07.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:07.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:07.626 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:57:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:07.642 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:07.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:07.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:07.779 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:07.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:07.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:57:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:07.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:07.910 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:07.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:08.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:08.048 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:07 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T11:57:08.048 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:07 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T11:57:08.048 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:07 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T11:57:08.048 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:07 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T11:57:08.048 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:07 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:08.050 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:08.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:08.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:08.190 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:08.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:08.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T11:57:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T11:57:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T11:57:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T11:57:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T11:57:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T11:57:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T11:57:08.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T11:57:08.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:07 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:08.333 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:08.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:08.436 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:08.465 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:08.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:08.566 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:08.590 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:08.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:08.691 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:08.720 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:08.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:08.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:08.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:08.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:09.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:09.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:09.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:09.147 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:09.149 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:09.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:09.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:09.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[42243]: pgmap v186: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:09.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T11:57:09.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T11:57:09.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[42243]: 6.7 scrub starts 2024-03-24T11:57:09.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[42243]: 6.7 scrub ok 2024-03-24T11:57:09.300 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[30223]: pgmap v186: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:09.300 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T11:57:09.300 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T11:57:09.300 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[30223]: 6.7 scrub starts 2024-03-24T11:57:09.300 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:08 smithi064 ceph-mon[30223]: 6.7 scrub ok 2024-03-24T11:57:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:09.302 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:09.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:09.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:08 smithi193 ceph-mon[33593]: pgmap v186: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:08 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T11:57:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:08 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T11:57:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:08 smithi193 ceph-mon[33593]: 6.7 scrub starts 2024-03-24T11:57:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:08 smithi193 ceph-mon[33593]: 6.7 scrub ok 2024-03-24T11:57:09.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:09.443 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:09.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:09.544 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:09.580 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:09.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:09.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:09.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:09.826 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:09.858 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:09.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:09.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:09.998 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:09.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:10.100 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:10.135 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:10.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:10.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:10.273 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:10.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:10.315 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:57:10.315 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:57:10.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:10.416 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:10.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:10.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:10.579 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:10.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:10.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:10.731 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:10.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:10.826 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:57:10.827 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:57:10.827 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-03-24T11:57:10.827 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T11:57:10.827 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T11:57:10.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-03-24T11:57:10.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:10.861 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:10.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:10.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:10.998 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:10.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:11.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:57:11.100 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[30223]: pgmap v187: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[30223]: 7.4 scrub starts 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[30223]: 2.12 scrub starts 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[30223]: 2.12 scrub ok 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[42243]: pgmap v187: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[42243]: 7.4 scrub starts 2024-03-24T11:57:11.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[42243]: 2.12 scrub starts 2024-03-24T11:57:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[42243]: 2.12 scrub ok 2024-03-24T11:57:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:11.128 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:11.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:11.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:11.278 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:10 smithi193 ceph-mon[33593]: pgmap v187: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:11.278 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:10 smithi193 ceph-mon[33593]: 7.4 scrub starts 2024-03-24T11:57:11.279 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:10 smithi193 ceph-mon[33593]: 2.12 scrub starts 2024-03-24T11:57:11.279 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:10 smithi193 ceph-mon[33593]: 2.12 scrub ok 2024-03-24T11:57:11.279 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:11.279 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:11.286 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:11.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:11.311 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T11:57:11.373 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:11.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:11.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:11.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:11.606 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:57:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:11.607 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:11.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:11.708 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:11.738 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:11.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:11.839 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:11.882 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:11.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:11.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:11.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T11:57:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:12.026 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:12.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:12.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:12.163 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:12.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:12.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:12 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1477038882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:57:12.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:12 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1477038882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:57:12.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:12.307 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:12.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:12 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1477038882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:57:12.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:12.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:12.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:12.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:12.606 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:12.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:12.707 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:12.741 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:12.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:12.843 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:12.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:12.875 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:12.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:12.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:13.020 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:13.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:13.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:13.047 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:13.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:13.164 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:13.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:13.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[30223]: pgmap v188: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[30223]: 7.4 scrub ok 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[30223]: 4.d deep-scrub starts 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[30223]: 4.d deep-scrub ok 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1477038882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[30223]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[42243]: pgmap v188: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[42243]: 7.4 scrub ok 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[42243]: 4.d deep-scrub starts 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[42243]: 4.d deep-scrub ok 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1477038882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:57:13.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:13 smithi064 ceph-mon[42243]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:57:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:13.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:13 smithi193 ceph-mon[33593]: pgmap v188: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:13 smithi193 ceph-mon[33593]: 7.4 scrub ok 2024-03-24T11:57:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:13 smithi193 ceph-mon[33593]: 4.d deep-scrub starts 2024-03-24T11:57:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:13 smithi193 ceph-mon[33593]: 4.d deep-scrub ok 2024-03-24T11:57:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:13 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:13 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1477038882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:57:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:13 smithi193 ceph-mon[33593]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:57:13.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:13.449 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:13.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:13.550 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:13.580 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:13.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:13.682 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:13.720 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:13.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:13.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:13.839 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:57:13.839 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:57:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:13.876 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:13.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:13.977 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:14.009 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:14.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:14.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:14.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:14.254 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:14.281 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:14.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:14.383 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:14.408 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:14.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:14.510 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:14.540 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:14.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:14.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:14.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T11:57:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:14.689 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:14.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:14.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:14.830 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:14.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:14.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:14.976 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:14.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:15.022 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:57:15.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:57:15.065 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:57:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T11:57:15.065 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:57:15.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:15.133 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:15.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:15.235 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:15.264 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:15.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:15 smithi064 ceph-mon[30223]: pgmap v190: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:57:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:15 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:57:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:15 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:57:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:15 smithi064 ceph-mon[30223]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:57:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:15 smithi064 ceph-mon[42243]: pgmap v190: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:57:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:15 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:57:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:15 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:57:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:15 smithi064 ceph-mon[42243]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:57:15.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:15 smithi193 ceph-mon[33593]: pgmap v190: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:57:15.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:15 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:57:15.306 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:15 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:57:15.306 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:15 smithi193 ceph-mon[33593]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:57:15.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:15.413 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:15.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:15.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:15.548 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:15.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:15.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:15.695 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:15.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:15.798 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:15.825 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:15.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:15.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:15.970 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:15.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:16.072 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:16.099 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:16.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:16.200 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:16.229 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:16.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:16.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:16.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:16.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:16.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:16.501 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:16.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:16.531 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:16.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:16.651 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:16.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:16.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:16.796 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:16.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:16.897 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:16.932 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:16.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:17.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:17.078 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:17.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:17.179 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:17.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:57:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:57:17.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:17 smithi193 ceph-mon[33593]: pgmap v192: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s 2024-03-24T11:57:17.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:17 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:57:17.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:17 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:17 smithi064 ceph-mon[30223]: pgmap v192: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:17 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:17 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:17 smithi064 ceph-mon[42243]: pgmap v192: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s 2024-03-24T11:57:17.212 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:17 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:57:17.212 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:17 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:57:17.214 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:17.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:17.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:17.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:17.358 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:17.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:17.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:17.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:17.499 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:57:17.500 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:57:17.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:57:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:57:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:17.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:17.630 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:17.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:17.732 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:17.760 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:17.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:17.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:17.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:17.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:18.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:18.032 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:57:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:18.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:18.073 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:57:18.121 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:57:18.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:18.166 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:57:18.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:18.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:18.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:18.206 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:57:18.243 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:57:18.285 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:57:18.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:18.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:18 smithi064 ceph-mon[30223]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:57:18.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:18 smithi064 ceph-mon[30223]: pgmap v194: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-24T11:57:18.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2312887626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:18.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:18.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:18 smithi064 ceph-mon[42243]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:57:18.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:18 smithi064 ceph-mon[42243]: pgmap v194: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-24T11:57:18.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2312887626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:18.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:18.324 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:57:18.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:57:18.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:57:18.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:57:18.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:57:18.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:57:18.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:57:18.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:57:18.328 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:57:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:18 smithi193 ceph-mon[33593]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:57:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:18 smithi193 ceph-mon[33593]: pgmap v194: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-24T11:57:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2312887626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:18.336 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:18.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:18.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T11:57:18.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:18.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:18.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:18.663 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:18.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:18.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:18.812 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:18.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:18.913 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:18.941 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:18.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:19.042 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:19.070 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:19.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:19.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:19.231 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:19.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:19.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:19.359 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:19.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:19.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:19.497 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:19.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:19.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:19.628 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:19.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:19.645 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:19.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:19.779 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:19.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:19.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:19.923 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:19.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:19.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:20.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:20.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:20.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:20.198 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:20.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:20.287 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T11:57:20.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:20.327 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:20.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:20.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:20.465 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:20.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:20.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:20 smithi064 ceph-mon[30223]: pgmap v195: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 712 B/s rd, 0 B/s wr, 739 op/s 2024-03-24T11:57:20.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:20 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2330297249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:20.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:20 smithi064 ceph-mon[30223]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:20.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:20 smithi064 ceph-mon[42243]: pgmap v195: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 712 B/s rd, 0 B/s wr, 739 op/s 2024-03-24T11:57:20.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2330297249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:20.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:20 smithi064 ceph-mon[42243]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:20.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:20 smithi193 ceph-mon[33593]: pgmap v195: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 712 B/s rd, 0 B/s wr, 739 op/s 2024-03-24T11:57:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:20 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2330297249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:20 smithi193 ceph-mon[33593]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:20.606 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:20.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:20.708 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:20.742 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:20.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:20.744 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:57:20.744 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:57:20.744 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-03-24T11:57:20.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:57:20.843 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:20.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:20.977 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:21.007 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:21.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:21.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:21.157 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:21.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:21.259 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:21.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:21.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:21 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:21 smithi193 ceph-mon[33593]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:21 smithi193 ceph-mon[33593]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:57:21.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:21.422 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:21.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:21.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:21.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:21 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:21 smithi064 ceph-mon[30223]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:21 smithi064 ceph-mon[30223]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:57:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:21 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:21 smithi064 ceph-mon[42243]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:21 smithi064 ceph-mon[42243]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:57:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:21.563 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:21.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:21.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:21.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:57:21.697 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:21.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:21.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:21.856 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:21.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:21.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:22.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:22.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:22.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:22.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:22.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:22.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:22.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:22.289 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:22.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:22.393 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:22.394 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:22.420 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:22.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:22.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:22.575 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:22.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:22.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:22.731 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:22.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:22.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:22.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:22.862 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:22.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:22.963 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:22.991 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:22.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:23.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:22 smithi064 ceph-mon[30223]: pgmap v197: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 712 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:23.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:22 smithi064 ceph-mon[42243]: pgmap v197: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 712 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:23.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:22 smithi193 ceph-mon[33593]: pgmap v197: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 712 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:23.093 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:23.122 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:23.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:23.224 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:23.253 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:23.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:23.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:57:23.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:23.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:23.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:23.509 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:23.537 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:23.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:23.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:23.690 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:23.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:23.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:23.820 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:23.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:23.922 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:23.949 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:23.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:23.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:24.051 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:24.080 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:24.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:24.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:24.220 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:24.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:24.249 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:57:24.249 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T11:57:24.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:196891 7.6 7:6c462e83:::benchmark_data_smithi064_87013_object75934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T11:57:24.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.983537+0000", 2024-03-24T11:57:24.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.012725339, 2024-03-24T11:57:24.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012756793000000001, 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 196891 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983537+0000", 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983537+0000", 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983538+0000", 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.519e-06 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983544+0000", 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8769999999999998e-06 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983558+0000", 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983730+0000", 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171857 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.991302+0000", 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075724210000000002 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.991427+0000", 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012474 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.991537+0000", 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110351 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:196892 7.4 7:2cb4e22f:::benchmark_data_smithi064_87013_object75935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.983755+0000", 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.012506859, 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012624656999999999, 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 196892 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983755+0000", 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983755+0000", 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983756+0000", 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983761+0000", 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6759999999999998e-06 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983770+0000", 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1330000000000005e-06 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983778+0000", 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8399999999999995e-06 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.984010+0000", 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023144499999999999 2024-03-24T11:57:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.984158+0000", 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148727 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.984289+0000", 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013094699999999999 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.987553+0000", 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00326348 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.988223+0000", 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067054899999999999 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996041+0000", 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078180660000000003 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:196893 7.6 7:64f3386c:::benchmark_data_smithi064_87013_object75936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.983794+0000", 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.012468526000000001, 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012681994, 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 196893 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983794+0000", 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983794+0000", 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983795+0000", 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2500000000000004e-07 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983798+0000", 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.484e-06 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983805+0000", 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9779999999999999e-06 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983811+0000", 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8139999999999997e-06 2024-03-24T11:57:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.994606+0000", 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010795331 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.994707+0000", 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100457 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.994808+0000", 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100958 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:196898 7.e e142/97 7:7425578a:::benchmark_data_smithi064_87013_object75941:head v 142'12342, mlcod=142'12340)", 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.988036+0000", 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0082259189999999999, 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085174499999999993, 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.988036+0000", 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.988036+0000", 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.988038+0000", 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2470000000000001e-06 2024-03-24T11:57:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.988045+0000", 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6569999999999997e-06 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.988055+0000", 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3670000000000003e-06 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.988061+0000", 2024-03-24T11:57:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6120000000000004e-06 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.990500+0000", 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002438927 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.990518+0000", 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8155000000000002e-05 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:196902 7.f 7:fe5ad716:::benchmark_data_smithi064_87013_object75945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.989087+0000", 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0071754619999999996, 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075297790000000003, 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 196902 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989087+0000", 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989087+0000", 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989089+0000", 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989094+0000", 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0499999999999999e-06 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989106+0000", 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2768999999999999e-05 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989114+0000", 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3760000000000002e-06 2024-03-24T11:57:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989192+0000", 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8312999999999994e-05 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989298+0000", 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105394 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.989400+0000", 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010218 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.993450+0000", 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040505539999999996 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:196905 7.f 7.74e94b9f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.995418+0000", 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00084407099999999995, 2024-03-24T11:57:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001279227, 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 196905 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995418+0000", 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995418+0000", 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995420+0000", 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9379999999999999e-06 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995426+0000", 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5799999999999999e-06 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995437+0000", 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1164e-05 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995445+0000", 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3010000000000007e-06 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.34742.0:196892 7.4 e142/97 ondisk, result = 0)", 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.995902+0000", 2024-03-24T11:57:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00036010099999999999, 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084858600000000002, 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995902+0000", 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995902+0000", 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995903+0000", 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4300000000000002e-07 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995907+0000", 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2350000000000001e-06 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995919+0000", 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2165e-05 2024-03-24T11:57:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.995926+0000", 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7569999999999998e-06 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996030+0000", 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103507 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996036+0000", 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3929999999999996e-06 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:196907 7.9 7:9e4e59f3:::benchmark_data_smithi064_87013_object75950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.996282+0000", 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4294967295.99998, 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053344899999999997, 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:57:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 196907 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996282+0000", 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996282+0000", 2024-03-24T11:57:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996284+0000", 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1289999999999999e-06 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996288+0000", 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3259999999999997e-06 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996301+0000", 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2874000000000001e-05 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996309+0000", 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7060000000000004e-06 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996357+0000", 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8751000000000001e-05 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996466+0000", 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109166 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.996573+0000", 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106588 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:196890 7.6 7:6b88340a:::benchmark_data_smithi064_87013_object75933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:23.980839+0000", 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.015423286, 2024-03-24T11:57:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016061341, 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 196890 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.980839+0000", 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.980839+0000", 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.980841+0000", 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.980846+0000", 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6470000000000003e-06 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.980858+0000", 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.980874+0000", 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6917000000000001e-05 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.983846+0000", 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029716260000000002 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.984069+0000", 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223396 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.984362+0000", 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029208499999999998 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:23.994575+0000", 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010213144 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:57:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 9 2024-03-24T11:57:24.288 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:57:24.321 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:24.346 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:24.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:24.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:24.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:24.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:24.584 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:24.616 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:24.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:24.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:24.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:24.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:24.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:24.887 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:24.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:24.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:24.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:25.018 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:25.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:25.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:24 smithi064 ceph-mon[30223]: pgmap v198: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:57:25.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:24 smithi064 ceph-mon[42243]: pgmap v198: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:57:25.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:24 smithi193 ceph-mon[33593]: pgmap v198: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:57:25.120 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:25.152 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:25.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:25.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:25.295 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:25.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:25.397 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:25.422 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:25.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:25.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:25.573 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:25.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:25.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:25.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:25.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:25.851 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:25.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:25.905 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:25.953 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:25.981 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:25.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:25 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:57:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:25 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:57:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:25 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:26.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:25 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:26.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:25 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:57:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:25 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:57:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:25 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:26.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:25 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:26.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:25 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:57:26.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:25 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:57:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:25 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:26.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:25 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:26.113 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:26.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:26.215 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:26.243 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:26.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:26.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:26.387 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:26.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:26.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:26.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:26.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T11:57:26.638 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:26.666 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:26.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:26.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:26.796 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:26.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:26.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:26.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:26 smithi064 ceph-mon[30223]: pgmap v199: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 626 B/s rd, 0 B/s wr, 638 op/s 2024-03-24T11:57:26.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:26 smithi064 ceph-mon[30223]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:57:26.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:26 smithi064 ceph-mon[42243]: pgmap v199: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 626 B/s rd, 0 B/s wr, 638 op/s 2024-03-24T11:57:26.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:26 smithi064 ceph-mon[42243]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:57:26.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:26.946 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:26.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:26.993 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:26 smithi193 ceph-mon[33593]: pgmap v199: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 626 B/s rd, 0 B/s wr, 638 op/s 2024-03-24T11:57:26.994 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:26 smithi193 ceph-mon[33593]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:57:27.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:27.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:27.095 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:27.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:27.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:27.225 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:27.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:27.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:27.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:57:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:57:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:27.374 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:27.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:27.475 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:27.505 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:27.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:27.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:57:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:57:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:27.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:27.651 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:27.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:27.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:27.782 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:27.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:27.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:27 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:27.860 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:27 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:27.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:27.931 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:27.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:28.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:28.071 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:28.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:28.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:27 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:28.092 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:28.173 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:28.201 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:28.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:28.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:28.330 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:28.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:28.379 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:57:28.384 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:57:28.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:28.472 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:28.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:28.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:28.610 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:28.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:28.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:28.761 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:28.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:28.862 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:28 smithi064 ceph-mon[30223]: pgmap v201: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:57:28.862 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:28 smithi064 ceph-mon[42243]: pgmap v201: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:57:28.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:28.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:29.027 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:29.056 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:29.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:28 smithi193 ceph-mon[33593]: pgmap v201: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:57:29.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:29.200 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:29.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:29.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:29.332 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:29.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:29.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:29.474 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:29.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:29.500 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:29.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:29.631 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:29.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:29.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:29.774 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:29.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:29.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:29.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:29.905 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:29.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:30.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:30.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:29 smithi064 ceph-mon[30223]: from='client.24910 v1:172.21.15.64:0/2775031248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:57:30.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:29 smithi064 ceph-mon[42243]: from='client.24910 v1:172.21.15.64:0/2775031248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:57:30.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:30.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:30.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:29 smithi193 ceph-mon[33593]: from='client.24910 v1:172.21.15.64:0/2775031248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:57:30.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:30.195 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:30.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:30.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:30.343 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:30.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:30.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:30.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:30.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:30.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:30.610 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:30.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:30.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:30.712 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:30.741 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:30.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:30.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:30.869 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:30.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:30.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:31.000 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:31.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:31.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:30 smithi064 ceph-mon[30223]: pgmap v202: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:57:31.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:30 smithi064 ceph-mon[42243]: pgmap v202: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:57:31.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:30 smithi193 ceph-mon[33593]: pgmap v202: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:57:31.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:31.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:31.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:31.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:31.259 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:31.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:31.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:31.403 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:31.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:31.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:31.548 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:31.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:31.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:31.694 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:31.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:31.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:31.831 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:31.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:31.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:31.973 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:31.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:32.074 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:32.103 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:32.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:32.172 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-03-24T11:57:27.213201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2785745270,"num_objects":42697,"num_object_clones":0,"num_object_copies":128091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42697,"num_whiteouts":0,"num_read":1232,"num_read_kb":1078,"num_write":200026,"num_write_kb":7743518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89695,"ondisk_log_size":89695,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9097920,"kb_used_data":8246296,"kb_used_omap":12,"kb_used_meta":850483,"kb_avail":740863296,"statfs":{"total":767960285184,"available":758644015104,"internally_reserved":0,"allocated":8444207104,"data_stored":8441231538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":870895152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,1,2,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-674824192,"num_objects":-10297,"num_object_clones":0,"num_object_copies":-30891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10297,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10297,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002322"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722345+0000","last_change":"2024-03-24T11:55:59.687133+0000","last_active":"2024-03-24T11:57:25.722345+0000","last_peered":"2024-03-24T11:57:25.722345+0000","last_clean":"2024-03-24T11:57:25.722345+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:57:25.722345+0000","last_undegraded":"2024-03-24T11:57:25.722345+0000","last_fullsized":"2024-03-24T11:57:25.722345+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:59.686983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:59.686983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.726560+0000","last_change":"2024-03-24T11:56:26.035534+0000","last_active":"2024-03-24T11:57:25.726560+0000","last_peered":"2024-03-24T11:57:25.726560+0000","last_clean":"2024-03-24T11:57:25.726560+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:57:25.726560+0000","last_undegraded":"2024-03-24T11:57:25.726560+0000","last_fullsized":"2024-03-24T11:57:25.726560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:26.035422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:56:26.035422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.390126+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":180,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.727201+0000","last_change":"2024-03-24T11:56:59.281520+0000","last_active":"2024-03-24T11:57:25.727201+0000","last_peered":"2024-03-24T11:57:25.727201+0000","last_clean":"2024-03-24T11:57:25.727201+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:57:25.727201+0000","last_undegraded":"2024-03-24T11:57:25.727201+0000","last_fullsized":"2024-03-24T11:57:25.727201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:59.278906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T11:56:59.278906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:59.502027+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"117'22","reported_seq":209,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722736+0000","last_change":"2024-03-24T11:55:21.286888+0000","last_active":"2024-03-24T11:57:25.722736+0000","last_peered":"2024-03-24T11:57:25.722736+0000","last_clean":"2024-03-24T11:57:25.722736+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:57:25.722736+0000","last_undegraded":"2024-03-24T11:57:25.722736+0000","last_fullsized":"2024-03-24T11:57:25.722736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'22","last_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T11:55:21.286727+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805394+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":175,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723647+0000","last_change":"2024-03-24T11:57:00.830211+0000","last_active":"2024-03-24T11:57:25.723647+0000","last_peered":"2024-03-24T11:57:25.723647+0000","last_clean":"2024-03-24T11:57:25.723647+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:57:25.723647+0000","last_undegraded":"2024-03-24T11:57:25.723647+0000","last_fullsized":"2024-03-24T11:57:25.723647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:00.830069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:00.830069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:04.858253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":172,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724400+0000","last_change":"2024-03-24T11:56:03.305225+0000","last_active":"2024-03-24T11:57:25.724400+0000","last_peered":"2024-03-24T11:57:25.724400+0000","last_clean":"2024-03-24T11:57:25.724400+0000","last_became_active":"2024-03-24T11:52:19.378282+0000","last_became_peered":"2024-03-24T11:52:19.378282+0000","last_unstale":"2024-03-24T11:57:25.724400+0000","last_undegraded":"2024-03-24T11:57:25.724400+0000","last_fullsized":"2024-03-24T11:57:25.724400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:03.305072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:03.305072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480428+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":181,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722771+0000","last_change":"2024-03-24T11:55:20.239169+0000","last_active":"2024-03-24T11:57:25.722771+0000","last_peered":"2024-03-24T11:57:25.722771+0000","last_clean":"2024-03-24T11:57:25.722771+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:57:25.722771+0000","last_undegraded":"2024-03-24T11:57:25.722771+0000","last_fullsized":"2024-03-24T11:57:25.722771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.239058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.239058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805309+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":191,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.726786+0000","last_change":"2024-03-24T11:56:08.033498+0000","last_active":"2024-03-24T11:57:25.726786+0000","last_peered":"2024-03-24T11:57:25.726786+0000","last_clean":"2024-03-24T11:57:25.726786+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:57:25.726786+0000","last_undegraded":"2024-03-24T11:57:25.726786+0000","last_fullsized":"2024-03-24T11:57:25.726786+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:08.033370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.033370+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.723689+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":175,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.712632+0000","last_change":"2024-03-24T11:56:25.437491+0000","last_active":"2024-03-24T11:57:25.712632+0000","last_peered":"2024-03-24T11:57:25.712632+0000","last_clean":"2024-03-24T11:57:25.712632+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:57:25.712632+0000","last_undegraded":"2024-03-24T11:57:25.712632+0000","last_fullsized":"2024-03-24T11:57:25.712632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.437410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.437410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.581445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":172,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728674+0000","last_change":"2024-03-24T11:56:00.086492+0000","last_active":"2024-03-24T11:57:25.728674+0000","last_peered":"2024-03-24T11:57:25.728674+0000","last_clean":"2024-03-24T11:57:25.728674+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:57:25.728674+0000","last_undegraded":"2024-03-24T11:57:25.728674+0000","last_fullsized":"2024-03-24T11:57:25.728674+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:00.086396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:00.086396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":188,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729340+0000","last_change":"2024-03-24T11:56:23.689326+0000","last_active":"2024-03-24T11:57:25.729340+0000","last_peered":"2024-03-24T11:57:25.729340+0000","last_clean":"2024-03-24T11:57:25.729340+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:57:25.729340+0000","last_undegraded":"2024-03-24T11:57:25.729340+0000","last_fullsized":"2024-03-24T11:57:25.729340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:23.689192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:23.689192+0000","last_clean_scrub_stamp":"2024-03-24T11:56:23.689192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":195,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724227+0000","last_change":"2024-03-24T11:56:24.027679+0000","last_active":"2024-03-24T11:57:25.724227+0000","last_peered":"2024-03-24T11:57:25.724227+0000","last_clean":"2024-03-24T11:57:25.724227+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:57:25.724227+0000","last_undegraded":"2024-03-24T11:57:25.724227+0000","last_fullsized":"2024-03-24T11:57:25.724227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T11:56:24.027577+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:46.704448+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":160,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723702+0000","last_change":"2024-03-24T11:54:58.045843+0000","last_active":"2024-03-24T11:57:25.723702+0000","last_peered":"2024-03-24T11:57:25.723702+0000","last_clean":"2024-03-24T11:57:25.723702+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:57:25.723702+0000","last_undegraded":"2024-03-24T11:57:25.723702+0000","last_fullsized":"2024-03-24T11:57:25.723702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:58.045730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:58.045730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.712976+0000","last_change":"2024-03-24T11:56:03.549369+0000","last_active":"2024-03-24T11:57:25.712976+0000","last_peered":"2024-03-24T11:57:25.712976+0000","last_clean":"2024-03-24T11:57:25.712976+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:57:25.712976+0000","last_undegraded":"2024-03-24T11:57:25.712976+0000","last_fullsized":"2024-03-24T11:57:25.712976+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:03.549293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:03.549293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.690911+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":188,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729128+0000","last_change":"2024-03-24T11:56:54.664307+0000","last_active":"2024-03-24T11:57:25.729128+0000","last_peered":"2024-03-24T11:57:25.729128+0000","last_clean":"2024-03-24T11:57:25.729128+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:57:25.729128+0000","last_undegraded":"2024-03-24T11:57:25.729128+0000","last_fullsized":"2024-03-24T11:57:25.729128+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:54.664176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:54.664176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:05.226417+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":199,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723746+0000","last_change":"2024-03-24T11:56:15.991404+0000","last_active":"2024-03-24T11:57:25.723746+0000","last_peered":"2024-03-24T11:57:25.723746+0000","last_clean":"2024-03-24T11:57:25.723746+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:57:25.723746+0000","last_undegraded":"2024-03-24T11:57:25.723746+0000","last_fullsized":"2024-03-24T11:57:25.723746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:15.991241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:15.991241+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:42.423985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":177,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.727438+0000","last_change":"2024-03-24T11:56:27.778229+0000","last_active":"2024-03-24T11:57:25.727438+0000","last_peered":"2024-03-24T11:57:25.727438+0000","last_clean":"2024-03-24T11:57:25.727438+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:57:25.727438+0000","last_undegraded":"2024-03-24T11:57:25.727438+0000","last_fullsized":"2024-03-24T11:57:25.727438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:27.777985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:27.777985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:54.716158+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724990+0000","last_change":"2024-03-24T11:56:11.992907+0000","last_active":"2024-03-24T11:57:25.724990+0000","last_peered":"2024-03-24T11:57:25.724990+0000","last_clean":"2024-03-24T11:57:25.724990+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:57:25.724990+0000","last_undegraded":"2024-03-24T11:57:25.724990+0000","last_fullsized":"2024-03-24T11:57:25.724990+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:11.992773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:11.992773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722580+0000","last_change":"2024-03-24T11:54:59.374964+0000","last_active":"2024-03-24T11:57:25.722580+0000","last_peered":"2024-03-24T11:57:25.722580+0000","last_clean":"2024-03-24T11:57:25.722580+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:57:25.722580+0000","last_undegraded":"2024-03-24T11:57:25.722580+0000","last_fullsized":"2024-03-24T11:57:25.722580+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:54:59.374820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:54:59.374820+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805258+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":223,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725772+0000","last_change":"2024-03-24T11:55:19.242716+0000","last_active":"2024-03-24T11:57:25.725772+0000","last_peered":"2024-03-24T11:57:25.725772+0000","last_clean":"2024-03-24T11:57:25.725772+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:57:25.725772+0000","last_undegraded":"2024-03-24T11:57:25.725772+0000","last_fullsized":"2024-03-24T11:57:25.725772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:55:19.242577+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.242577+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805423+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":175,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724802+0000","last_change":"2024-03-24T11:56:25.010309+0000","last_active":"2024-03-24T11:57:25.724802+0000","last_peered":"2024-03-24T11:57:25.724802+0000","last_clean":"2024-03-24T11:57:25.724802+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:57:25.724802+0000","last_undegraded":"2024-03-24T11:57:25.724802+0000","last_fullsized":"2024-03-24T11:57:25.724802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.010190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.010190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:54.549909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":180,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723325+0000","last_change":"2024-03-24T11:57:01.452495+0000","last_active":"2024-03-24T11:57:25.723325+0000","last_peered":"2024-03-24T11:57:25.723325+0000","last_clean":"2024-03-24T11:57:25.723325+0000","last_became_active":"2024-03-24T11:52:19.384249+0000","last_became_peered":"2024-03-24T11:52:19.384249+0000","last_unstale":"2024-03-24T11:57:25.723325+0000","last_undegraded":"2024-03-24T11:57:25.723325+0000","last_fullsized":"2024-03-24T11:57:25.723325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:01.452078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:01.452078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:22.574897+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723741+0000","last_change":"2024-03-24T11:56:25.722636+0000","last_active":"2024-03-24T11:57:25.723741+0000","last_peered":"2024-03-24T11:57:25.723741+0000","last_clean":"2024-03-24T11:57:25.723741+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:57:25.723741+0000","last_undegraded":"2024-03-24T11:57:25.723741+0000","last_fullsized":"2024-03-24T11:57:25.723741+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.722560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.722560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:28.956369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"115'22","reported_seq":217,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.730299+0000","last_change":"2024-03-24T11:57:03.604742+0000","last_active":"2024-03-24T11:57:25.730299+0000","last_peered":"2024-03-24T11:57:25.730299+0000","last_clean":"2024-03-24T11:57:25.730299+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:57:25.730299+0000","last_undegraded":"2024-03-24T11:57:25.730299+0000","last_fullsized":"2024-03-24T11:57:25.730299+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'22","last_scrub_stamp":"2024-03-24T11:57:03.604590+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T11:57:03.604590+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:15.675401+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":176,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722969+0000","last_change":"2024-03-24T11:56:25.369507+0000","last_active":"2024-03-24T11:57:25.722969+0000","last_peered":"2024-03-24T11:57:25.722969+0000","last_clean":"2024-03-24T11:57:25.722969+0000","last_became_active":"2024-03-24T11:52:21.390764+0000","last_became_peered":"2024-03-24T11:52:21.390764+0000","last_unstale":"2024-03-24T11:57:25.722969+0000","last_undegraded":"2024-03-24T11:57:25.722969+0000","last_fullsized":"2024-03-24T11:57:25.722969+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.369074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.369074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.196833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":172,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.726608+0000","last_change":"2024-03-24T11:55:06.054268+0000","last_active":"2024-03-24T11:57:25.726608+0000","last_peered":"2024-03-24T11:57:25.726608+0000","last_clean":"2024-03-24T11:57:25.726608+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:57:25.726608+0000","last_undegraded":"2024-03-24T11:57:25.726608+0000","last_fullsized":"2024-03-24T11:57:25.726608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:06.054160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:21.941323+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.054160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475370+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":182,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.721984+0000","last_change":"2024-03-24T11:56:14.756794+0000","last_active":"2024-03-24T11:57:25.721984+0000","last_peered":"2024-03-24T11:57:25.721984+0000","last_clean":"2024-03-24T11:57:25.721984+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:57:25.721984+0000","last_undegraded":"2024-03-24T11:57:25.721984+0000","last_fullsized":"2024-03-24T11:57:25.721984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:56:14.756549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.756549+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:44.749261+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":202,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725458+0000","last_change":"2024-03-24T11:56:19.749277+0000","last_active":"2024-03-24T11:57:25.725458+0000","last_peered":"2024-03-24T11:57:25.725458+0000","last_clean":"2024-03-24T11:57:25.725458+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:57:25.725458+0000","last_undegraded":"2024-03-24T11:57:25.725458+0000","last_fullsized":"2024-03-24T11:57:25.725458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:19.749138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:19.749138+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:33.204161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":177,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729390+0000","last_change":"2024-03-24T11:56:22.692993+0000","last_active":"2024-03-24T11:57:25.729390+0000","last_peered":"2024-03-24T11:57:25.729390+0000","last_clean":"2024-03-24T11:57:25.729390+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:57:25.729390+0000","last_undegraded":"2024-03-24T11:57:25.729390+0000","last_fullsized":"2024-03-24T11:57:25.729390+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:56:22.692822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:22.692822+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:44.367501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":245,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725402+0000","last_change":"2024-03-24T11:56:06.931131+0000","last_active":"2024-03-24T11:57:25.725402+0000","last_peered":"2024-03-24T11:57:25.725402+0000","last_clean":"2024-03-24T11:57:25.725402+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:57:25.725402+0000","last_undegraded":"2024-03-24T11:57:25.725402+0000","last_fullsized":"2024-03-24T11:57:25.725402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:06.931035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.931035+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.717875+0000","last_change":"2024-03-24T11:57:01.435324+0000","last_active":"2024-03-24T11:57:25.717875+0000","last_peered":"2024-03-24T11:57:25.717875+0000","last_clean":"2024-03-24T11:57:25.717875+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:57:25.717875+0000","last_undegraded":"2024-03-24T11:57:25.717875+0000","last_fullsized":"2024-03-24T11:57:25.717875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:01.435247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:57:01.435247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:05.919024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":196,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.726503+0000","last_change":"2024-03-24T11:56:15.211513+0000","last_active":"2024-03-24T11:57:25.726503+0000","last_peered":"2024-03-24T11:57:25.726503+0000","last_clean":"2024-03-24T11:57:25.726503+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:57:25.726503+0000","last_undegraded":"2024-03-24T11:57:25.726503+0000","last_fullsized":"2024-03-24T11:57:25.726503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T11:56:15.211165+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:53:46.014065+0000","last_clean_scrub_stamp":"2024-03-24T11:56:15.211165+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.992778+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724201+0000","last_change":"2024-03-24T11:56:24.748650+0000","last_active":"2024-03-24T11:57:25.724201+0000","last_peered":"2024-03-24T11:57:25.724201+0000","last_clean":"2024-03-24T11:57:25.724201+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:57:25.724201+0000","last_undegraded":"2024-03-24T11:57:25.724201+0000","last_fullsized":"2024-03-24T11:57:25.724201+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:24.748528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:24.748528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.615827+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":172,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728912+0000","last_change":"2024-03-24T11:56:01.069407+0000","last_active":"2024-03-24T11:57:25.728912+0000","last_peered":"2024-03-24T11:57:25.728912+0000","last_clean":"2024-03-24T11:57:25.728912+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:57:25.728912+0000","last_undegraded":"2024-03-24T11:57:25.728912+0000","last_fullsized":"2024-03-24T11:57:25.728912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:01.069280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.632287+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.069280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725297+0000","last_change":"2024-03-24T11:56:04.915981+0000","last_active":"2024-03-24T11:57:25.725297+0000","last_peered":"2024-03-24T11:57:25.725297+0000","last_clean":"2024-03-24T11:57:25.725297+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:57:25.725297+0000","last_undegraded":"2024-03-24T11:57:25.725297+0000","last_fullsized":"2024-03-24T11:57:25.725297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:04.915872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:04.915872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.722003+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":196,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722630+0000","last_change":"2024-03-24T11:56:05.799020+0000","last_active":"2024-03-24T11:57:25.722630+0000","last_peered":"2024-03-24T11:57:25.722630+0000","last_clean":"2024-03-24T11:57:25.722630+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:57:25.722630+0000","last_undegraded":"2024-03-24T11:57:25.722630+0000","last_fullsized":"2024-03-24T11:57:25.722630+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:05.798510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.798510+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:34.206728+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":168,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728546+0000","last_change":"2024-03-24T11:56:14.695577+0000","last_active":"2024-03-24T11:57:25.728546+0000","last_peered":"2024-03-24T11:57:25.728546+0000","last_clean":"2024-03-24T11:57:25.728546+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:57:25.728546+0000","last_undegraded":"2024-03-24T11:57:25.728546+0000","last_fullsized":"2024-03-24T11:57:25.728546+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:14.695456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.695456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.844357+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724232+0000","last_change":"2024-03-24T11:57:02.648430+0000","last_active":"2024-03-24T11:57:25.724232+0000","last_peered":"2024-03-24T11:57:25.724232+0000","last_clean":"2024-03-24T11:57:25.724232+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:57:25.724232+0000","last_undegraded":"2024-03-24T11:57:25.724232+0000","last_fullsized":"2024-03-24T11:57:25.724232+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:02.648304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:02.648304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:25.103363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.713516+0000","last_change":"2024-03-24T11:55:18.619637+0000","last_active":"2024-03-24T11:57:25.713516+0000","last_peered":"2024-03-24T11:57:25.713516+0000","last_clean":"2024-03-24T11:57:25.713516+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:57:25.713516+0000","last_undegraded":"2024-03-24T11:57:25.713516+0000","last_fullsized":"2024-03-24T11:57:25.713516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.619533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.619533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.690882+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":183,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.731769+0000","last_change":"2024-03-24T11:55:00.363697+0000","last_active":"2024-03-24T11:57:25.731769+0000","last_peered":"2024-03-24T11:57:25.731769+0000","last_clean":"2024-03-24T11:57:25.731769+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:57:25.731769+0000","last_undegraded":"2024-03-24T11:57:25.731769+0000","last_fullsized":"2024-03-24T11:57:25.731769+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T11:55:00.363559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:00.363559+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805229+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.721664+0000","last_change":"2024-03-24T11:56:25.759157+0000","last_active":"2024-03-24T11:57:25.721664+0000","last_peered":"2024-03-24T11:57:25.721664+0000","last_clean":"2024-03-24T11:57:25.721664+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:57:25.721664+0000","last_undegraded":"2024-03-24T11:57:25.721664+0000","last_fullsized":"2024-03-24T11:57:25.721664+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.759010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.759010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:37.953711+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":172,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724665+0000","last_change":"2024-03-24T11:55:56.132313+0000","last_active":"2024-03-24T11:57:25.724665+0000","last_peered":"2024-03-24T11:57:25.724665+0000","last_clean":"2024-03-24T11:57:25.724665+0000","last_became_active":"2024-03-24T11:52:19.377484+0000","last_became_peered":"2024-03-24T11:52:19.377484+0000","last_unstale":"2024-03-24T11:57:25.724665+0000","last_undegraded":"2024-03-24T11:57:25.724665+0000","last_fullsized":"2024-03-24T11:57:25.724665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:56.132189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:55:56.132189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480277+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":180,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723084+0000","last_change":"2024-03-24T11:55:15.993168+0000","last_active":"2024-03-24T11:57:25.723084+0000","last_peered":"2024-03-24T11:57:25.723084+0000","last_clean":"2024-03-24T11:57:25.723084+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:57:25.723084+0000","last_undegraded":"2024-03-24T11:57:25.723084+0000","last_fullsized":"2024-03-24T11:57:25.723084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:15.992998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:15.992998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":194,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724843+0000","last_change":"2024-03-24T11:56:01.946316+0000","last_active":"2024-03-24T11:57:25.724843+0000","last_peered":"2024-03-24T11:57:25.724843+0000","last_clean":"2024-03-24T11:57:25.724843+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:57:25.724843+0000","last_undegraded":"2024-03-24T11:57:25.724843+0000","last_fullsized":"2024-03-24T11:57:25.724843+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:01.946162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.946162+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.723605+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":168,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723194+0000","last_change":"2024-03-24T11:55:19.108172+0000","last_active":"2024-03-24T11:57:25.723194+0000","last_peered":"2024-03-24T11:57:25.723194+0000","last_clean":"2024-03-24T11:57:25.723194+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:57:25.723194+0000","last_undegraded":"2024-03-24T11:57:25.723194+0000","last_fullsized":"2024-03-24T11:57:25.723194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.108015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475396+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":173,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723911+0000","last_change":"2024-03-24T11:55:22.292901+0000","last_active":"2024-03-24T11:57:25.723911+0000","last_peered":"2024-03-24T11:57:25.723911+0000","last_clean":"2024-03-24T11:57:25.723911+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:57:25.723911+0000","last_undegraded":"2024-03-24T11:57:25.723911+0000","last_fullsized":"2024-03-24T11:57:25.723911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:22.292785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.292785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805284+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724304+0000","last_change":"2024-03-24T11:56:54.488109+0000","last_active":"2024-03-24T11:57:25.724304+0000","last_peered":"2024-03-24T11:57:25.724304+0000","last_clean":"2024-03-24T11:57:25.724304+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:57:25.724304+0000","last_undegraded":"2024-03-24T11:57:25.724304+0000","last_fullsized":"2024-03-24T11:57:25.724304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:54.487937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:54.487937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:01.528067+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":200,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724414+0000","last_change":"2024-03-24T11:56:11.308420+0000","last_active":"2024-03-24T11:57:25.724414+0000","last_peered":"2024-03-24T11:57:25.724414+0000","last_clean":"2024-03-24T11:57:25.724414+0000","last_became_active":"2024-03-24T11:52:17.355956+0000","last_became_peered":"2024-03-24T11:52:17.355956+0000","last_unstale":"2024-03-24T11:57:25.724414+0000","last_undegraded":"2024-03-24T11:57:25.724414+0000","last_fullsized":"2024-03-24T11:57:25.724414+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:11.308278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:11.308278+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:28.504792+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":175,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724245+0000","last_change":"2024-03-24T11:57:01.827065+0000","last_active":"2024-03-24T11:57:25.724245+0000","last_peered":"2024-03-24T11:57:25.724245+0000","last_clean":"2024-03-24T11:57:25.724245+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:57:25.724245+0000","last_undegraded":"2024-03-24T11:57:25.724245+0000","last_fullsized":"2024-03-24T11:57:25.724245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:01.826994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:01.826994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:29.289930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":252,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722172+0000","last_change":"2024-03-24T11:57:05.768573+0000","last_active":"2024-03-24T11:57:25.722172+0000","last_peered":"2024-03-24T11:57:25.722172+0000","last_clean":"2024-03-24T11:57:25.722172+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:57:25.722172+0000","last_undegraded":"2024-03-24T11:57:25.722172+0000","last_fullsized":"2024-03-24T11:57:25.722172+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:57:05.768444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:05.768444+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:21.074962+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":180,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724652+0000","last_change":"2024-03-24T11:56:24.140811+0000","last_active":"2024-03-24T11:57:25.724652+0000","last_peered":"2024-03-24T11:57:25.724652+0000","last_clean":"2024-03-24T11:57:25.724652+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:57:25.724652+0000","last_undegraded":"2024-03-24T11:57:25.724652+0000","last_fullsized":"2024-03-24T11:57:25.724652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:24.140700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:24.140700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:51.459650+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":199,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.713411+0000","last_change":"2024-03-24T11:57:06.521085+0000","last_active":"2024-03-24T11:57:25.713411+0000","last_peered":"2024-03-24T11:57:25.713411+0000","last_clean":"2024-03-24T11:57:25.713411+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:57:25.713411+0000","last_undegraded":"2024-03-24T11:57:25.713411+0000","last_fullsized":"2024-03-24T11:57:25.713411+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:57:06.520933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:57:06.520933+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:25.935698+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":168,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724944+0000","last_change":"2024-03-24T11:56:14.264963+0000","last_active":"2024-03-24T11:57:25.724944+0000","last_peered":"2024-03-24T11:57:25.724944+0000","last_clean":"2024-03-24T11:57:25.724944+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:57:25.724944+0000","last_undegraded":"2024-03-24T11:57:25.724944+0000","last_fullsized":"2024-03-24T11:57:25.724944+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:14.264656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.264656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:38.580881+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":250,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.713275+0000","last_change":"2024-03-24T11:56:24.399112+0000","last_active":"2024-03-24T11:57:25.713275+0000","last_peered":"2024-03-24T11:57:25.713275+0000","last_clean":"2024-03-24T11:57:25.713275+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:57:25.713275+0000","last_undegraded":"2024-03-24T11:57:25.713275+0000","last_fullsized":"2024-03-24T11:57:25.713275+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:24.398998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:24.398998+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:49.258784+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.712921+0000","last_change":"2024-03-24T11:57:10.539877+0000","last_active":"2024-03-24T11:57:25.712921+0000","last_peered":"2024-03-24T11:57:25.712921+0000","last_clean":"2024-03-24T11:57:25.712921+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:57:25.712921+0000","last_undegraded":"2024-03-24T11:57:25.712921+0000","last_fullsized":"2024-03-24T11:57:25.712921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:10.539754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T11:57:10.539754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:26.279773+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":197,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722050+0000","last_change":"2024-03-24T11:55:06.420550+0000","last_active":"2024-03-24T11:57:25.722050+0000","last_peered":"2024-03-24T11:57:25.722050+0000","last_clean":"2024-03-24T11:57:25.722050+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:57:25.722050+0000","last_undegraded":"2024-03-24T11:57:25.722050+0000","last_fullsized":"2024-03-24T11:57:25.722050+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:55:06.420406+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:55:06.420406+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805196+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":176,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729188+0000","last_change":"2024-03-24T11:56:59.714972+0000","last_active":"2024-03-24T11:57:25.729188+0000","last_peered":"2024-03-24T11:57:25.729188+0000","last_clean":"2024-03-24T11:57:25.729188+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:57:25.729188+0000","last_undegraded":"2024-03-24T11:57:25.729188+0000","last_fullsized":"2024-03-24T11:57:25.729188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:59.714893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:59.714893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:24.638598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.713286+0000","last_change":"2024-03-24T11:56:05.489055+0000","last_active":"2024-03-24T11:57:25.713286+0000","last_peered":"2024-03-24T11:57:25.713286+0000","last_clean":"2024-03-24T11:57:25.713286+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:57:25.713286+0000","last_undegraded":"2024-03-24T11:57:25.713286+0000","last_fullsized":"2024-03-24T11:57:25.713286+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:05.488932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.488932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.690939+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724261+0000","last_change":"2024-03-24T11:56:26.760375+0000","last_active":"2024-03-24T11:57:25.724261+0000","last_peered":"2024-03-24T11:57:25.724261+0000","last_clean":"2024-03-24T11:57:25.724261+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:57:25.724261+0000","last_undegraded":"2024-03-24T11:57:25.724261+0000","last_fullsized":"2024-03-24T11:57:25.724261+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T11:56:26.760297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:48.927348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":194,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.727528+0000","last_change":"2024-03-24T11:57:03.760132+0000","last_active":"2024-03-24T11:57:25.727528+0000","last_peered":"2024-03-24T11:57:25.727528+0000","last_clean":"2024-03-24T11:57:25.727528+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:57:25.727528+0000","last_undegraded":"2024-03-24T11:57:25.727528+0000","last_fullsized":"2024-03-24T11:57:25.727528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:57:03.756872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:57:03.756872+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:08.840988+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729877+0000","last_change":"2024-03-24T11:56:01.567568+0000","last_active":"2024-03-24T11:57:25.729877+0000","last_peered":"2024-03-24T11:57:25.729877+0000","last_clean":"2024-03-24T11:57:25.729877+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:57:25.729877+0000","last_undegraded":"2024-03-24T11:57:25.729877+0000","last_fullsized":"2024-03-24T11:57:25.729877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:01.567421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:01.567421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.945947+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":173,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.727614+0000","last_change":"2024-03-24T11:56:23.736925+0000","last_active":"2024-03-24T11:57:25.727614+0000","last_peered":"2024-03-24T11:57:25.727614+0000","last_clean":"2024-03-24T11:57:25.727614+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:57:25.727614+0000","last_undegraded":"2024-03-24T11:57:25.727614+0000","last_fullsized":"2024-03-24T11:57:25.727614+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:23.736791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:23.736791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:46.576335+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":189,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.733420+0000","last_change":"2024-03-24T11:57:06.629404+0000","last_active":"2024-03-24T11:57:25.733420+0000","last_peered":"2024-03-24T11:57:25.733420+0000","last_clean":"2024-03-24T11:57:25.733420+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:57:25.733420+0000","last_undegraded":"2024-03-24T11:57:25.733420+0000","last_fullsized":"2024-03-24T11:57:25.733420+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:06.629264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:57:06.629264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:31.183700+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"115'13","reported_seq":196,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725267+0000","last_change":"2024-03-24T11:55:16.094065+0000","last_active":"2024-03-24T11:57:25.725267+0000","last_peered":"2024-03-24T11:57:25.725267+0000","last_clean":"2024-03-24T11:57:25.725267+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:57:25.725267+0000","last_undegraded":"2024-03-24T11:57:25.725267+0000","last_fullsized":"2024-03-24T11:57:25.725267+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'13","last_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T11:55:16.093668+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475426+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"143'16340","reported_seq":16661,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.728529+0000","last_change":"2024-03-24T11:56:57.940634+0000","last_active":"2024-03-24T11:57:26.728529+0000","last_peered":"2024-03-24T11:57:26.728529+0000","last_clean":"2024-03-24T11:57:26.728529+0000","last_became_active":"2024-03-24T11:54:16.232098+0000","last_became_peered":"2024-03-24T11:54:16.232098+0000","last_unstale":"2024-03-24T11:57:26.728529+0000","last_undegraded":"2024-03-24T11:57:26.728529+0000","last_fullsized":"2024-03-24T11:57:26.728529+0000","mapping_epoch":97,"log_start":"131'11602","ondisk_log_start":"131'11602","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'14465","last_scrub_stamp":"2024-03-24T11:56:57.940467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:57.940467+0000","objects_scrubbed":4590,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:12.061249+0000","scrub_duration":2163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173277184,"num_objects":2644,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12724,"num_write_kb":491776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724006+0000","last_change":"2024-03-24T11:56:02.765493+0000","last_active":"2024-03-24T11:57:25.724006+0000","last_peered":"2024-03-24T11:57:25.724006+0000","last_clean":"2024-03-24T11:57:25.724006+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:57:25.724006+0000","last_undegraded":"2024-03-24T11:57:25.724006+0000","last_fullsized":"2024-03-24T11:57:25.724006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:02.765358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:02.765358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.107895+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":199,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724044+0000","last_change":"2024-03-24T11:57:05.795776+0000","last_active":"2024-03-24T11:57:25.724044+0000","last_peered":"2024-03-24T11:57:25.724044+0000","last_clean":"2024-03-24T11:57:25.724044+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:57:25.724044+0000","last_undegraded":"2024-03-24T11:57:25.724044+0000","last_fullsized":"2024-03-24T11:57:25.724044+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:57:05.795646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:57:05.795646+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:25.705074+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":224,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725487+0000","last_change":"2024-03-24T11:56:14.948071+0000","last_active":"2024-03-24T11:57:25.725487+0000","last_peered":"2024-03-24T11:57:25.725487+0000","last_clean":"2024-03-24T11:57:25.725487+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:57:25.725487+0000","last_undegraded":"2024-03-24T11:57:25.725487+0000","last_fullsized":"2024-03-24T11:57:25.725487+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:56:14.947940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.947940+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":181,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.733330+0000","last_change":"2024-03-24T11:57:04.570839+0000","last_active":"2024-03-24T11:57:25.733330+0000","last_peered":"2024-03-24T11:57:25.733330+0000","last_clean":"2024-03-24T11:57:25.733330+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:57:25.733330+0000","last_undegraded":"2024-03-24T11:57:25.733330+0000","last_fullsized":"2024-03-24T11:57:25.733330+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:04.570340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:04.570340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:24.522306+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"143'12527","reported_seq":12713,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.178931+0000","last_change":"2024-03-24T11:56:08.680053+0000","last_active":"2024-03-24T11:57:26.178931+0000","last_peered":"2024-03-24T11:57:26.178931+0000","last_clean":"2024-03-24T11:57:26.178931+0000","last_became_active":"2024-03-24T11:54:16.238055+0000","last_became_peered":"2024-03-24T11:54:16.238055+0000","last_unstale":"2024-03-24T11:57:26.178931+0000","last_undegraded":"2024-03-24T11:57:26.178931+0000","last_fullsized":"2024-03-24T11:57:26.178931+0000","mapping_epoch":97,"log_start":"127'7200","ondisk_log_start":"127'7200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7741","last_scrub_stamp":"2024-03-24T11:56:08.679818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.679818+0000","objects_scrubbed":7448,"log_size":5327,"log_dups_size":0,"ondisk_log_size":5327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.903434+0000","scrub_duration":1667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171245568,"num_objects":2613,"num_object_clones":0,"num_object_copies":7839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":168,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723301+0000","last_change":"2024-03-24T11:56:27.229810+0000","last_active":"2024-03-24T11:57:25.723301+0000","last_peered":"2024-03-24T11:57:25.723301+0000","last_clean":"2024-03-24T11:57:25.723301+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:57:25.723301+0000","last_undegraded":"2024-03-24T11:57:25.723301+0000","last_fullsized":"2024-03-24T11:57:25.723301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:27.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:27.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:42.352370+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":188,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723004+0000","last_change":"2024-03-24T11:56:23.429848+0000","last_active":"2024-03-24T11:57:25.723004+0000","last_peered":"2024-03-24T11:57:25.723004+0000","last_clean":"2024-03-24T11:57:25.723004+0000","last_became_active":"2024-03-24T11:52:15.138792+0000","last_became_peered":"2024-03-24T11:52:15.138792+0000","last_unstale":"2024-03-24T11:57:25.723004+0000","last_undegraded":"2024-03-24T11:57:25.723004+0000","last_fullsized":"2024-03-24T11:57:25.723004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:23.429712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:56:23.429712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.052978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":204,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724765+0000","last_change":"2024-03-24T11:57:05.447578+0000","last_active":"2024-03-24T11:57:25.724765+0000","last_peered":"2024-03-24T11:57:25.724765+0000","last_clean":"2024-03-24T11:57:25.724765+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:57:25.724765+0000","last_undegraded":"2024-03-24T11:57:25.724765+0000","last_fullsized":"2024-03-24T11:57:25.724765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T11:57:05.447425+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:07.856095+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":243,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724876+0000","last_change":"2024-03-24T11:56:25.188171+0000","last_active":"2024-03-24T11:57:25.724876+0000","last_peered":"2024-03-24T11:57:25.724876+0000","last_clean":"2024-03-24T11:57:25.724876+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:57:25.724876+0000","last_undegraded":"2024-03-24T11:57:25.724876+0000","last_fullsized":"2024-03-24T11:57:25.724876+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:25.188045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.188045+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.612880+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"143'12693","reported_seq":12879,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.193081+0000","last_change":"2024-03-24T11:56:10.140756+0000","last_active":"2024-03-24T11:57:26.193081+0000","last_peered":"2024-03-24T11:57:26.193081+0000","last_clean":"2024-03-24T11:57:26.193081+0000","last_became_active":"2024-03-24T11:54:16.240514+0000","last_became_peered":"2024-03-24T11:54:16.240514+0000","last_unstale":"2024-03-24T11:57:26.193081+0000","last_undegraded":"2024-03-24T11:57:26.193081+0000","last_fullsized":"2024-03-24T11:57:26.193081+0000","mapping_epoch":97,"log_start":"124'6500","ondisk_log_start":"124'6500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'8025","last_scrub_stamp":"2024-03-24T11:56:10.140421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.140421+0000","objects_scrubbed":7499,"log_size":6193,"log_dups_size":0,"ondisk_log_size":6193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.721947+0000","scrub_duration":2191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181600256,"num_objects":2771,"num_object_clones":0,"num_object_copies":8313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12693,"num_write_kb":494848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.726322+0000","last_change":"2024-03-24T11:55:12.389096+0000","last_active":"2024-03-24T11:57:25.726322+0000","last_peered":"2024-03-24T11:57:25.726322+0000","last_clean":"2024-03-24T11:57:25.726322+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:57:25.726322+0000","last_undegraded":"2024-03-24T11:57:25.726322+0000","last_fullsized":"2024-03-24T11:57:25.726322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:12.388835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:12.388835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.712986+0000","last_change":"2024-03-24T11:56:56.505316+0000","last_active":"2024-03-24T11:57:25.712986+0000","last_peered":"2024-03-24T11:57:25.712986+0000","last_clean":"2024-03-24T11:57:25.712986+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:57:25.712986+0000","last_undegraded":"2024-03-24T11:57:25.712986+0000","last_fullsized":"2024-03-24T11:57:25.712986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:56.505190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:56.505190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:19.816653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":238,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.713504+0000","last_change":"2024-03-24T11:56:19.300291+0000","last_active":"2024-03-24T11:57:25.713504+0000","last_peered":"2024-03-24T11:57:25.713504+0000","last_clean":"2024-03-24T11:57:25.713504+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:57:25.713504+0000","last_undegraded":"2024-03-24T11:57:25.713504+0000","last_fullsized":"2024-03-24T11:57:25.713504+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-03-24T11:56:19.300145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:19.300145+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.340159+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":172,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725914+0000","last_change":"2024-03-24T11:56:06.187599+0000","last_active":"2024-03-24T11:57:25.725914+0000","last_peered":"2024-03-24T11:57:25.725914+0000","last_clean":"2024-03-24T11:57:25.725914+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:57:25.725914+0000","last_undegraded":"2024-03-24T11:57:25.725914+0000","last_fullsized":"2024-03-24T11:57:25.725914+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:06.187195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.187195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:29.724378+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"143'12461","reported_seq":12610,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.542112+0000","last_change":"2024-03-24T11:57:11.597662+0000","last_active":"2024-03-24T11:57:26.542112+0000","last_peered":"2024-03-24T11:57:26.542112+0000","last_clean":"2024-03-24T11:57:26.542112+0000","last_became_active":"2024-03-24T11:54:16.238056+0000","last_became_peered":"2024-03-24T11:54:16.238056+0000","last_unstale":"2024-03-24T11:57:26.542112+0000","last_undegraded":"2024-03-24T11:57:26.542112+0000","last_fullsized":"2024-03-24T11:57:26.542112+0000","mapping_epoch":97,"log_start":"127'7000","ondisk_log_start":"127'7000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'11466","last_scrub_stamp":"2024-03-24T11:57:11.596798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:11.596798+0000","objects_scrubbed":3667,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:32.195437+0000","scrub_duration":1373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":168,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729376+0000","last_change":"2024-03-24T11:55:57.171256+0000","last_active":"2024-03-24T11:57:25.729376+0000","last_peered":"2024-03-24T11:57:25.729376+0000","last_clean":"2024-03-24T11:57:25.729376+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:57:25.729376+0000","last_undegraded":"2024-03-24T11:57:25.729376+0000","last_fullsized":"2024-03-24T11:57:25.729376+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:57.171146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:57.171146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":494,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.722402+0000","last_change":"2024-03-24T11:55:22.048179+0000","last_active":"2024-03-24T11:57:25.722402+0000","last_peered":"2024-03-24T11:57:25.722402+0000","last_clean":"2024-03-24T11:57:25.722402+0000","last_became_active":"2024-03-24T11:52:15.144954+0000","last_became_peered":"2024-03-24T11:52:15.144954+0000","last_unstale":"2024-03-24T11:57:25.722402+0000","last_undegraded":"2024-03-24T11:57:25.722402+0000","last_fullsized":"2024-03-24T11:57:25.722402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:55:22.048036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:22.048036+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480249+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":304,"num_read_kb":299,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":189,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728171+0000","last_change":"2024-03-24T11:56:10.706814+0000","last_active":"2024-03-24T11:57:25.728171+0000","last_peered":"2024-03-24T11:57:25.728171+0000","last_clean":"2024-03-24T11:57:25.728171+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:57:25.728171+0000","last_undegraded":"2024-03-24T11:57:25.728171+0000","last_fullsized":"2024-03-24T11:57:25.728171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.706685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.706685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.852507+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":203,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.721748+0000","last_change":"2024-03-24T11:56:04.340339+0000","last_active":"2024-03-24T11:57:25.721748+0000","last_peered":"2024-03-24T11:57:25.721748+0000","last_clean":"2024-03-24T11:57:25.721748+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:57:25.721748+0000","last_undegraded":"2024-03-24T11:57:25.721748+0000","last_fullsized":"2024-03-24T11:57:25.721748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-03-24T11:56:04.340193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:04.340193+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480370+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":181,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.730861+0000","last_change":"2024-03-24T11:56:20.561269+0000","last_active":"2024-03-24T11:57:25.730861+0000","last_peered":"2024-03-24T11:57:25.730861+0000","last_clean":"2024-03-24T11:57:25.730861+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:57:25.730861+0000","last_undegraded":"2024-03-24T11:57:25.730861+0000","last_fullsized":"2024-03-24T11:57:25.730861+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:20.561139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T11:56:20.561139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:33.363698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"143'12095","reported_seq":12252,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.546948+0000","last_change":"2024-03-24T11:56:57.061849+0000","last_active":"2024-03-24T11:57:26.546948+0000","last_peered":"2024-03-24T11:57:26.546948+0000","last_clean":"2024-03-24T11:57:26.546948+0000","last_became_active":"2024-03-24T11:54:16.238515+0000","last_became_peered":"2024-03-24T11:54:16.238515+0000","last_unstale":"2024-03-24T11:57:26.546948+0000","last_undegraded":"2024-03-24T11:57:26.546948+0000","last_fullsized":"2024-03-24T11:57:26.546948+0000","mapping_epoch":97,"log_start":"125'6600","ondisk_log_start":"125'6600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'10259","last_scrub_stamp":"2024-03-24T11:56:57.060611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:57.060611+0000","objects_scrubbed":4510,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:18.676827+0000","scrub_duration":1755,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171507712,"num_objects":2617,"num_object_clones":0,"num_object_copies":7851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":470784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":175,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.714169+0000","last_change":"2024-03-24T11:57:08.529656+0000","last_active":"2024-03-24T11:57:25.714169+0000","last_peered":"2024-03-24T11:57:25.714169+0000","last_clean":"2024-03-24T11:57:25.714169+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:57:25.714169+0000","last_undegraded":"2024-03-24T11:57:25.714169+0000","last_fullsized":"2024-03-24T11:57:25.714169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:08.529532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:57:08.529532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:31.584820+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":190,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724988+0000","last_change":"2024-03-24T11:55:56.666658+0000","last_active":"2024-03-24T11:57:25.724988+0000","last_peered":"2024-03-24T11:57:25.724988+0000","last_clean":"2024-03-24T11:57:25.724988+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:57:25.724988+0000","last_undegraded":"2024-03-24T11:57:25.724988+0000","last_fullsized":"2024-03-24T11:57:25.724988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T11:55:56.665991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:56.665991+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.725538+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.713743+0000","last_change":"2024-03-24T11:55:20.647830+0000","last_active":"2024-03-24T11:57:25.713743+0000","last_peered":"2024-03-24T11:57:25.713743+0000","last_clean":"2024-03-24T11:57:25.713743+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:57:25.713743+0000","last_undegraded":"2024-03-24T11:57:25.713743+0000","last_fullsized":"2024-03-24T11:57:25.713743+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.647693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.647693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.690783+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":435,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728943+0000","last_change":"2024-03-24T11:56:05.931697+0000","last_active":"2024-03-24T11:57:25.728943+0000","last_peered":"2024-03-24T11:57:25.728943+0000","last_clean":"2024-03-24T11:57:25.728943+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:57:25.728943+0000","last_undegraded":"2024-03-24T11:57:25.728943+0000","last_fullsized":"2024-03-24T11:57:25.728943+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:56:05.931549+0000","last_deep_scrub":"71'119","last_deep_scrub_stamp":"2024-03-24T11:52:19.498276+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.931549+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":268,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724618+0000","last_change":"2024-03-24T11:57:07.673650+0000","last_active":"2024-03-24T11:57:25.724618+0000","last_peered":"2024-03-24T11:57:25.724618+0000","last_clean":"2024-03-24T11:57:25.724618+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:57:25.724618+0000","last_undegraded":"2024-03-24T11:57:25.724618+0000","last_fullsized":"2024-03-24T11:57:25.724618+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:57:07.673476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:57:07.673476+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:08.287519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":180,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729569+0000","last_change":"2024-03-24T11:57:02.710574+0000","last_active":"2024-03-24T11:57:25.729569+0000","last_peered":"2024-03-24T11:57:25.729569+0000","last_clean":"2024-03-24T11:57:25.729569+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:57:25.729569+0000","last_undegraded":"2024-03-24T11:57:25.729569+0000","last_fullsized":"2024-03-24T11:57:25.729569+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:02.710441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:02.710441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:26.374754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"143'12532","reported_seq":12715,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.174223+0000","last_change":"2024-03-24T11:56:12.814952+0000","last_active":"2024-03-24T11:57:26.174223+0000","last_peered":"2024-03-24T11:57:26.174223+0000","last_clean":"2024-03-24T11:57:26.174223+0000","last_became_active":"2024-03-24T11:54:16.236927+0000","last_became_peered":"2024-03-24T11:54:16.236927+0000","last_unstale":"2024-03-24T11:57:26.174223+0000","last_undegraded":"2024-03-24T11:57:26.174223+0000","last_fullsized":"2024-03-24T11:57:26.174223+0000","mapping_epoch":97,"log_start":"127'7600","ondisk_log_start":"127'7600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8029","last_scrub_stamp":"2024-03-24T11:56:12.814717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:12.814717+0000","objects_scrubbed":7259,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.844676+0000","scrub_duration":2088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176291840,"num_objects":2690,"num_object_clones":0,"num_object_copies":8070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12532,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":168,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723099+0000","last_change":"2024-03-24T11:55:14.967693+0000","last_active":"2024-03-24T11:57:25.723099+0000","last_peered":"2024-03-24T11:57:25.723099+0000","last_clean":"2024-03-24T11:57:25.723099+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:57:25.723099+0000","last_undegraded":"2024-03-24T11:57:25.723099+0000","last_fullsized":"2024-03-24T11:57:25.723099+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:14.967137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.967137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480186+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724946+0000","last_change":"2024-03-24T11:56:26.209516+0000","last_active":"2024-03-24T11:57:25.724946+0000","last_peered":"2024-03-24T11:57:25.724946+0000","last_clean":"2024-03-24T11:57:25.724946+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:57:25.724946+0000","last_undegraded":"2024-03-24T11:57:25.724946+0000","last_fullsized":"2024-03-24T11:57:25.724946+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:26.209386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:26.209386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.139031+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723620+0000","last_change":"2024-03-24T11:56:53.906725+0000","last_active":"2024-03-24T11:57:25.723620+0000","last_peered":"2024-03-24T11:57:25.723620+0000","last_clean":"2024-03-24T11:57:25.723620+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T11:57:25.723620+0000","last_undegraded":"2024-03-24T11:57:25.723620+0000","last_fullsized":"2024-03-24T11:57:25.723620+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:19.706115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:19.706115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:45.318106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":194,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725056+0000","last_change":"2024-03-24T11:55:14.029361+0000","last_active":"2024-03-24T11:57:25.725056+0000","last_peered":"2024-03-24T11:57:25.725056+0000","last_clean":"2024-03-24T11:57:25.725056+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:57:25.725056+0000","last_undegraded":"2024-03-24T11:57:25.725056+0000","last_fullsized":"2024-03-24T11:57:25.725056+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:55:14.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:14.029279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.714115+0000","last_change":"2024-03-24T11:56:08.456477+0000","last_active":"2024-03-24T11:57:25.714115+0000","last_peered":"2024-03-24T11:57:25.714115+0000","last_clean":"2024-03-24T11:57:25.714115+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:57:25.714115+0000","last_undegraded":"2024-03-24T11:57:25.714115+0000","last_fullsized":"2024-03-24T11:57:25.714115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:08.456346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:08.456346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.633752+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"143'12701","reported_seq":12878,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.172611+0000","last_change":"2024-03-24T11:56:22.846505+0000","last_active":"2024-03-24T11:57:26.172611+0000","last_peered":"2024-03-24T11:57:26.172611+0000","last_clean":"2024-03-24T11:57:26.172611+0000","last_became_active":"2024-03-24T11:54:16.236857+0000","last_became_peered":"2024-03-24T11:54:16.236857+0000","last_unstale":"2024-03-24T11:57:26.172611+0000","last_undegraded":"2024-03-24T11:57:26.172611+0000","last_fullsized":"2024-03-24T11:57:26.172611+0000","mapping_epoch":97,"log_start":"125'7000","ondisk_log_start":"125'7000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8794","last_scrub_stamp":"2024-03-24T11:56:22.846262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:22.846262+0000","objects_scrubbed":6680,"log_size":5701,"log_dups_size":0,"ondisk_log_size":5701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.664107+0000","scrub_duration":1492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179240960,"num_objects":2735,"num_object_clones":0,"num_object_copies":8205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12701,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":176,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724455+0000","last_change":"2024-03-24T11:56:10.262911+0000","last_active":"2024-03-24T11:57:25.724455+0000","last_peered":"2024-03-24T11:57:25.724455+0000","last_clean":"2024-03-24T11:57:25.724455+0000","last_became_active":"2024-03-24T11:52:21.851069+0000","last_became_peered":"2024-03-24T11:52:21.851069+0000","last_unstale":"2024-03-24T11:57:25.724455+0000","last_undegraded":"2024-03-24T11:57:25.724455+0000","last_fullsized":"2024-03-24T11:57:25.724455+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.262776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.262776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480399+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":188,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729433+0000","last_change":"2024-03-24T11:56:55.718318+0000","last_active":"2024-03-24T11:57:25.729433+0000","last_peered":"2024-03-24T11:57:25.729433+0000","last_clean":"2024-03-24T11:57:25.729433+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:57:25.729433+0000","last_undegraded":"2024-03-24T11:57:25.729433+0000","last_fullsized":"2024-03-24T11:57:25.729433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:55.718180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:55.718180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:04.474315+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":212,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724267+0000","last_change":"2024-03-24T11:56:05.227501+0000","last_active":"2024-03-24T11:57:25.724267+0000","last_peered":"2024-03-24T11:57:25.724267+0000","last_clean":"2024-03-24T11:57:25.724267+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:57:25.724267+0000","last_undegraded":"2024-03-24T11:57:25.724267+0000","last_fullsized":"2024-03-24T11:57:25.724267+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:56:05.227347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.227347+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475546+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723824+0000","last_change":"2024-03-24T11:55:15.058768+0000","last_active":"2024-03-24T11:57:25.723824+0000","last_peered":"2024-03-24T11:57:25.723824+0000","last_clean":"2024-03-24T11:57:25.723824+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:57:25.723824+0000","last_undegraded":"2024-03-24T11:57:25.723824+0000","last_fullsized":"2024-03-24T11:57:25.723824+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:15.058688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T11:55:15.058688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.721910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"143'12347","reported_seq":12528,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.756236+0000","last_change":"2024-03-24T11:56:13.177828+0000","last_active":"2024-03-24T11:57:26.756236+0000","last_peered":"2024-03-24T11:57:26.756236+0000","last_clean":"2024-03-24T11:57:26.756236+0000","last_became_active":"2024-03-24T11:54:16.238176+0000","last_became_peered":"2024-03-24T11:54:16.238176+0000","last_unstale":"2024-03-24T11:57:26.756236+0000","last_undegraded":"2024-03-24T11:57:26.756236+0000","last_fullsized":"2024-03-24T11:57:26.756236+0000","mapping_epoch":97,"log_start":"127'7600","ondisk_log_start":"127'7600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'7875","last_scrub_stamp":"2024-03-24T11:56:13.177489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:13.177489+0000","objects_scrubbed":7037,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805452+0000","scrub_duration":2432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163643392,"num_objects":2497,"num_object_clones":0,"num_object_copies":7491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12347,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":175,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724048+0000","last_change":"2024-03-24T11:56:22.106831+0000","last_active":"2024-03-24T11:57:25.724048+0000","last_peered":"2024-03-24T11:57:25.724048+0000","last_clean":"2024-03-24T11:57:25.724048+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:57:25.724048+0000","last_undegraded":"2024-03-24T11:57:25.724048+0000","last_fullsized":"2024-03-24T11:57:25.724048+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:22.106699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:22.106699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:43.372371+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":189,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.721446+0000","last_change":"2024-03-24T11:56:27.426637+0000","last_active":"2024-03-24T11:57:25.721446+0000","last_peered":"2024-03-24T11:57:25.721446+0000","last_clean":"2024-03-24T11:57:25.721446+0000","last_became_active":"2024-03-24T11:52:15.142974+0000","last_became_peered":"2024-03-24T11:52:15.142974+0000","last_unstale":"2024-03-24T11:57:25.721446+0000","last_undegraded":"2024-03-24T11:57:25.721446+0000","last_fullsized":"2024-03-24T11:57:25.721446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:56:27.426497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:27.426497+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:40.245207+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":209,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.731477+0000","last_change":"2024-03-24T11:54:57.356969+0000","last_active":"2024-03-24T11:57:25.731477+0000","last_peered":"2024-03-24T11:57:25.731477+0000","last_clean":"2024-03-24T11:57:25.731477+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:57:25.731477+0000","last_undegraded":"2024-03-24T11:57:25.731477+0000","last_fullsized":"2024-03-24T11:57:25.731477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:54:57.356797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:54:57.356797+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805339+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725594+0000","last_change":"2024-03-24T11:56:10.062814+0000","last_active":"2024-03-24T11:57:25.725594+0000","last_peered":"2024-03-24T11:57:25.725594+0000","last_clean":"2024-03-24T11:57:25.725594+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:57:25.725594+0000","last_undegraded":"2024-03-24T11:57:25.725594+0000","last_fullsized":"2024-03-24T11:57:25.725594+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:10.062643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:10.062643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:31.233410+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"143'12271","reported_seq":12456,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.760034+0000","last_change":"2024-03-24T11:56:06.946412+0000","last_active":"2024-03-24T11:57:26.760034+0000","last_peered":"2024-03-24T11:57:26.760034+0000","last_clean":"2024-03-24T11:57:26.760034+0000","last_became_active":"2024-03-24T11:54:16.235994+0000","last_became_peered":"2024-03-24T11:54:16.235994+0000","last_unstale":"2024-03-24T11:57:26.760034+0000","last_undegraded":"2024-03-24T11:57:26.760034+0000","last_fullsized":"2024-03-24T11:57:26.760034+0000","mapping_epoch":97,"log_start":"127'7500","ondisk_log_start":"127'7500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7491","last_scrub_stamp":"2024-03-24T11:56:06.946102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.946102+0000","objects_scrubbed":7377,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:28.786025+0000","scrub_duration":3152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168099840,"num_objects":2565,"num_object_clones":0,"num_object_copies":7695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12271,"num_write_kb":474752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":176,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729036+0000","last_change":"2024-03-24T11:57:00.692112+0000","last_active":"2024-03-24T11:57:25.729036+0000","last_peered":"2024-03-24T11:57:25.729036+0000","last_clean":"2024-03-24T11:57:25.729036+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:57:25.729036+0000","last_undegraded":"2024-03-24T11:57:25.729036+0000","last_fullsized":"2024-03-24T11:57:25.729036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:00.691997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:00.691997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:20.529490+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725368+0000","last_change":"2024-03-24T11:56:53.520514+0000","last_active":"2024-03-24T11:57:25.725368+0000","last_peered":"2024-03-24T11:57:25.725368+0000","last_clean":"2024-03-24T11:57:25.725368+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:57:25.725368+0000","last_undegraded":"2024-03-24T11:57:25.725368+0000","last_fullsized":"2024-03-24T11:57:25.725368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:53.520294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:56:53.520294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:16.317361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":211,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725481+0000","last_change":"2024-03-24T11:56:58.841337+0000","last_active":"2024-03-24T11:57:25.725481+0000","last_peered":"2024-03-24T11:57:25.725481+0000","last_clean":"2024-03-24T11:57:25.725481+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:57:25.725481+0000","last_undegraded":"2024-03-24T11:57:25.725481+0000","last_fullsized":"2024-03-24T11:57:25.725481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:56:58.841126+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:56:58.841126+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:18.745710+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":244,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.734824+0000","last_change":"2024-03-24T11:56:13.769093+0000","last_active":"2024-03-24T11:57:25.734824+0000","last_peered":"2024-03-24T11:57:25.734824+0000","last_clean":"2024-03-24T11:57:25.734824+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:57:25.734824+0000","last_undegraded":"2024-03-24T11:57:25.734824+0000","last_fullsized":"2024-03-24T11:57:25.734824+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:13.768824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:13.768824+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805506+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"143'12399","reported_seq":12579,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.192601+0000","last_change":"2024-03-24T11:56:18.704600+0000","last_active":"2024-03-24T11:57:26.192601+0000","last_peered":"2024-03-24T11:57:26.192601+0000","last_clean":"2024-03-24T11:57:26.192601+0000","last_became_active":"2024-03-24T11:54:16.241724+0000","last_became_peered":"2024-03-24T11:54:16.241724+0000","last_unstale":"2024-03-24T11:57:26.192601+0000","last_undegraded":"2024-03-24T11:57:26.192601+0000","last_fullsized":"2024-03-24T11:57:26.192601+0000","mapping_epoch":97,"log_start":"123'6200","ondisk_log_start":"123'6200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8281","last_scrub_stamp":"2024-03-24T11:56:18.704486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:18.704486+0000","objects_scrubbed":6804,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.772004+0000","scrub_duration":1857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172883992,"num_objects":2639,"num_object_clones":0,"num_object_copies":7917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2639,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12399,"num_write_kb":481153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.721942+0000","last_change":"2024-03-24T11:55:55.660561+0000","last_active":"2024-03-24T11:57:25.721942+0000","last_peered":"2024-03-24T11:57:25.721942+0000","last_clean":"2024-03-24T11:57:25.721942+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:57:25.721942+0000","last_undegraded":"2024-03-24T11:57:25.721942+0000","last_fullsized":"2024-03-24T11:57:25.721942+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:55.660368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:55.660368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.805477+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.714087+0000","last_change":"2024-03-24T11:55:03.730546+0000","last_active":"2024-03-24T11:57:25.714087+0000","last_peered":"2024-03-24T11:57:25.714087+0000","last_clean":"2024-03-24T11:57:25.714087+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:57:25.714087+0000","last_undegraded":"2024-03-24T11:57:25.714087+0000","last_fullsized":"2024-03-24T11:57:25.714087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:03.730436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:55:03.730436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.690749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":188,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728672+0000","last_change":"2024-03-24T11:56:20.726116+0000","last_active":"2024-03-24T11:57:25.728672+0000","last_peered":"2024-03-24T11:57:25.728672+0000","last_clean":"2024-03-24T11:57:25.728672+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:57:25.728672+0000","last_undegraded":"2024-03-24T11:57:25.728672+0000","last_fullsized":"2024-03-24T11:57:25.728672+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T11:56:20.725884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":194,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724893+0000","last_change":"2024-03-24T11:56:06.009518+0000","last_active":"2024-03-24T11:57:25.724893+0000","last_peered":"2024-03-24T11:57:25.724893+0000","last_clean":"2024-03-24T11:57:25.724893+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:57:25.724893+0000","last_undegraded":"2024-03-24T11:57:25.724893+0000","last_fullsized":"2024-03-24T11:57:25.724893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:56:06.009394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.009394+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.723649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"143'12445","reported_seq":12626,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.160677+0000","last_change":"2024-03-24T11:56:14.978194+0000","last_active":"2024-03-24T11:57:26.160677+0000","last_peered":"2024-03-24T11:57:26.160677+0000","last_clean":"2024-03-24T11:57:26.160677+0000","last_became_active":"2024-03-24T11:54:16.237102+0000","last_became_peered":"2024-03-24T11:54:16.237102+0000","last_unstale":"2024-03-24T11:57:26.160677+0000","last_undegraded":"2024-03-24T11:57:26.160677+0000","last_fullsized":"2024-03-24T11:57:26.160677+0000","mapping_epoch":97,"log_start":"123'6300","ondisk_log_start":"123'6300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8081","last_scrub_stamp":"2024-03-24T11:56:14.977871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.977871+0000","objects_scrubbed":7074,"log_size":6145,"log_dups_size":0,"ondisk_log_size":6145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.721977+0000","scrub_duration":2055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173211648,"num_objects":2643,"num_object_clones":0,"num_object_copies":7929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":482816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":175,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.713129+0000","last_change":"2024-03-24T11:57:02.448529+0000","last_active":"2024-03-24T11:57:25.713129+0000","last_peered":"2024-03-24T11:57:25.713129+0000","last_clean":"2024-03-24T11:57:25.713129+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:57:25.713129+0000","last_undegraded":"2024-03-24T11:57:25.713129+0000","last_fullsized":"2024-03-24T11:57:25.713129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:02.448364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:02.448364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:10.915906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":253,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724647+0000","last_change":"2024-03-24T11:56:54.862433+0000","last_active":"2024-03-24T11:57:25.724647+0000","last_peered":"2024-03-24T11:57:25.724647+0000","last_clean":"2024-03-24T11:57:25.724647+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:57:25.724647+0000","last_undegraded":"2024-03-24T11:57:25.724647+0000","last_fullsized":"2024-03-24T11:57:25.724647+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:54.862306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:54.862306+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:12.321272+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.734618+0000","last_change":"2024-03-24T11:55:58.143201+0000","last_active":"2024-03-24T11:57:25.734618+0000","last_peered":"2024-03-24T11:57:25.734618+0000","last_clean":"2024-03-24T11:57:25.734618+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:57:25.734618+0000","last_undegraded":"2024-03-24T11:57:25.734618+0000","last_fullsized":"2024-03-24T11:57:25.734618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:58.143072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:55:58.143072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480306+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"130'22","reported_seq":216,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.721320+0000","last_change":"2024-03-24T11:56:17.455467+0000","last_active":"2024-03-24T11:57:25.721320+0000","last_peered":"2024-03-24T11:57:25.721320+0000","last_clean":"2024-03-24T11:57:25.721320+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:57:25.721320+0000","last_undegraded":"2024-03-24T11:57:25.721320+0000","last_fullsized":"2024-03-24T11:57:25.721320+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'22","last_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T11:56:17.455285+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:29.860233+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"143'12346","reported_seq":12449,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.567477+0000","last_change":"2024-03-24T11:54:16.239589+0000","last_active":"2024-03-24T11:57:26.567477+0000","last_peered":"2024-03-24T11:57:26.567477+0000","last_clean":"2024-03-24T11:57:26.567477+0000","last_became_active":"2024-03-24T11:54:16.239274+0000","last_became_peered":"2024-03-24T11:54:16.239274+0000","last_unstale":"2024-03-24T11:57:26.567477+0000","last_undegraded":"2024-03-24T11:57:26.567477+0000","last_fullsized":"2024-03-24T11:57:26.567477+0000","mapping_epoch":97,"log_start":"125'6800","ondisk_log_start":"125'6800","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:54:15.215608+0000","objects_scrubbed":0,"log_size":5546,"log_dups_size":0,"ondisk_log_size":5546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172883968,"num_objects":2638,"num_object_clones":0,"num_object_copies":7914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12346,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":176,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724002+0000","last_change":"2024-03-24T11:56:21.012442+0000","last_active":"2024-03-24T11:57:25.724002+0000","last_peered":"2024-03-24T11:57:25.724002+0000","last_clean":"2024-03-24T11:57:25.724002+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:57:25.724002+0000","last_undegraded":"2024-03-24T11:57:25.724002+0000","last_fullsized":"2024-03-24T11:57:25.724002+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:56:21.012308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:21.012308+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.154184+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":252,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.730643+0000","last_change":"2024-03-24T11:56:13.679738+0000","last_active":"2024-03-24T11:57:25.730643+0000","last_peered":"2024-03-24T11:57:25.730643+0000","last_clean":"2024-03-24T11:57:25.730643+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:57:25.730643+0000","last_undegraded":"2024-03-24T11:57:25.730643+0000","last_fullsized":"2024-03-24T11:57:25.730643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:56:13.679609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:13.679609+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:42.607215+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":180,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724280+0000","last_change":"2024-03-24T11:56:00.196860+0000","last_active":"2024-03-24T11:57:25.724280+0000","last_peered":"2024-03-24T11:57:25.724280+0000","last_clean":"2024-03-24T11:57:25.724280+0000","last_became_active":"2024-03-24T11:52:15.134326+0000","last_became_peered":"2024-03-24T11:52:15.134326+0000","last_unstale":"2024-03-24T11:57:25.724280+0000","last_undegraded":"2024-03-24T11:57:25.724280+0000","last_fullsized":"2024-03-24T11:57:25.724280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:00.196663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_clean_scrub_stamp":"2024-03-24T11:56:00.196663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.480334+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"130'15","reported_seq":203,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724004+0000","last_change":"2024-03-24T11:56:28.036136+0000","last_active":"2024-03-24T11:57:25.724004+0000","last_peered":"2024-03-24T11:57:25.724004+0000","last_clean":"2024-03-24T11:57:25.724004+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:57:25.724004+0000","last_undegraded":"2024-03-24T11:57:25.724004+0000","last_fullsized":"2024-03-24T11:57:25.724004+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'15","last_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T11:56:28.035859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.724323+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"143'12490","reported_seq":12679,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.363373+0000","last_change":"2024-03-24T11:56:06.470793+0000","last_active":"2024-03-24T11:57:26.363373+0000","last_peered":"2024-03-24T11:57:26.363373+0000","last_clean":"2024-03-24T11:57:26.363373+0000","last_became_active":"2024-03-24T11:54:16.236885+0000","last_became_peered":"2024-03-24T11:54:16.236885+0000","last_unstale":"2024-03-24T11:57:26.363373+0000","last_undegraded":"2024-03-24T11:57:26.363373+0000","last_fullsized":"2024-03-24T11:57:26.363373+0000","mapping_epoch":97,"log_start":"122'5900","ondisk_log_start":"122'5900","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7612","last_scrub_stamp":"2024-03-24T11:56:06.470609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.470609+0000","objects_scrubbed":7541,"log_size":6590,"log_dups_size":0,"ondisk_log_size":6590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.879748+0000","scrub_duration":2905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174456832,"num_objects":2662,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12490,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.730654+0000","last_change":"2024-03-24T11:55:20.629992+0000","last_active":"2024-03-24T11:57:25.730654+0000","last_peered":"2024-03-24T11:57:25.730654+0000","last_clean":"2024-03-24T11:57:25.730654+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:57:25.730654+0000","last_undegraded":"2024-03-24T11:57:25.730654+0000","last_fullsized":"2024-03-24T11:57:25.730654+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:20.629800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:20.629800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:25.852442+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":172,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723187+0000","last_change":"2024-03-24T11:56:06.310590+0000","last_active":"2024-03-24T11:57:25.723187+0000","last_peered":"2024-03-24T11:57:25.723187+0000","last_clean":"2024-03-24T11:57:25.723187+0000","last_became_active":"2024-03-24T11:52:19.387115+0000","last_became_peered":"2024-03-24T11:52:19.387115+0000","last_unstale":"2024-03-24T11:57:25.723187+0000","last_undegraded":"2024-03-24T11:57:25.723187+0000","last_fullsized":"2024-03-24T11:57:25.723187+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:06.310357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:06.310357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:29.666932+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728190+0000","last_change":"2024-03-24T11:56:18.582669+0000","last_active":"2024-03-24T11:57:25.728190+0000","last_peered":"2024-03-24T11:57:25.728190+0000","last_clean":"2024-03-24T11:57:25.728190+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:57:25.728190+0000","last_undegraded":"2024-03-24T11:57:25.728190+0000","last_fullsized":"2024-03-24T11:57:25.728190+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:56:18.582550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:18.582550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:31.743441+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"116'25","reported_seq":219,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724455+0000","last_change":"2024-03-24T11:57:11.800831+0000","last_active":"2024-03-24T11:57:25.724455+0000","last_peered":"2024-03-24T11:57:25.724455+0000","last_clean":"2024-03-24T11:57:25.724455+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:57:25.724455+0000","last_undegraded":"2024-03-24T11:57:25.724455+0000","last_fullsized":"2024-03-24T11:57:25.724455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'25","last_scrub_stamp":"2024-03-24T11:57:11.800708+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T11:57:11.800708+0000","last_clean_scrub_stamp":"2024-03-24T11:57:11.800708+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:19.849716+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"143'12503","reported_seq":12653,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.519764+0000","last_change":"2024-03-24T11:57:05.732819+0000","last_active":"2024-03-24T11:57:26.519764+0000","last_peered":"2024-03-24T11:57:26.519764+0000","last_clean":"2024-03-24T11:57:26.519764+0000","last_became_active":"2024-03-24T11:54:16.237374+0000","last_became_peered":"2024-03-24T11:54:16.237374+0000","last_unstale":"2024-03-24T11:57:26.519764+0000","last_undegraded":"2024-03-24T11:57:26.519764+0000","last_fullsized":"2024-03-24T11:57:26.519764+0000","mapping_epoch":97,"log_start":"126'7000","ondisk_log_start":"126'7000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'11160","last_scrub_stamp":"2024-03-24T11:57:05.732516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:05.732516+0000","objects_scrubbed":4072,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:32.359886+0000","scrub_duration":1534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176488448,"num_objects":2693,"num_object_clones":0,"num_object_copies":8079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12503,"num_write_kb":486272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":168,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728479+0000","last_change":"2024-03-24T11:55:18.299756+0000","last_active":"2024-03-24T11:57:25.728479+0000","last_peered":"2024-03-24T11:57:25.728479+0000","last_clean":"2024-03-24T11:57:25.728479+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:57:25.728479+0000","last_undegraded":"2024-03-24T11:57:25.728479+0000","last_fullsized":"2024-03-24T11:57:25.728479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T11:55:18.299637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":181,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.729736+0000","last_change":"2024-03-24T11:56:23.615139+0000","last_active":"2024-03-24T11:57:25.729736+0000","last_peered":"2024-03-24T11:57:25.729736+0000","last_clean":"2024-03-24T11:57:25.729736+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:57:25.729736+0000","last_undegraded":"2024-03-24T11:57:25.729736+0000","last_fullsized":"2024-03-24T11:57:25.729736+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:23.614720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:23.614720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:38.993316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":189,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.730113+0000","last_change":"2024-03-24T11:56:14.639416+0000","last_active":"2024-03-24T11:57:25.730113+0000","last_peered":"2024-03-24T11:57:25.730113+0000","last_clean":"2024-03-24T11:57:25.730113+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:57:25.730113+0000","last_undegraded":"2024-03-24T11:57:25.730113+0000","last_fullsized":"2024-03-24T11:57:25.730113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T11:56:14.639264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.855218+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":214,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723649+0000","last_change":"2024-03-24T11:57:06.624543+0000","last_active":"2024-03-24T11:57:25.723649+0000","last_peered":"2024-03-24T11:57:25.723649+0000","last_clean":"2024-03-24T11:57:25.723649+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:57:25.723649+0000","last_undegraded":"2024-03-24T11:57:25.723649+0000","last_fullsized":"2024-03-24T11:57:25.723649+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:57:06.624420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:57:06.624420+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:25.607974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"143'12441","reported_seq":12628,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.398323+0000","last_change":"2024-03-24T11:56:05.111342+0000","last_active":"2024-03-24T11:57:26.398323+0000","last_peered":"2024-03-24T11:57:26.398323+0000","last_clean":"2024-03-24T11:57:26.398323+0000","last_became_active":"2024-03-24T11:54:16.237483+0000","last_became_peered":"2024-03-24T11:54:16.237483+0000","last_unstale":"2024-03-24T11:57:26.398323+0000","last_undegraded":"2024-03-24T11:57:26.398323+0000","last_fullsized":"2024-03-24T11:57:26.398323+0000","mapping_epoch":97,"log_start":"122'5900","ondisk_log_start":"122'5900","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'7549","last_scrub_stamp":"2024-03-24T11:56:05.111031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:05.111031+0000","objects_scrubbed":7494,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.864430+0000","scrub_duration":2574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12441,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":160,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.725668+0000","last_change":"2024-03-24T11:55:01.085505+0000","last_active":"2024-03-24T11:57:25.725668+0000","last_peered":"2024-03-24T11:57:25.725668+0000","last_clean":"2024-03-24T11:57:25.725668+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:57:25.725668+0000","last_undegraded":"2024-03-24T11:57:25.725668+0000","last_fullsized":"2024-03-24T11:57:25.725668+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:01.085396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:55:01.085396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475344+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":181,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.733127+0000","last_change":"2024-03-24T11:56:25.642344+0000","last_active":"2024-03-24T11:57:25.733127+0000","last_peered":"2024-03-24T11:57:25.733127+0000","last_clean":"2024-03-24T11:57:25.733127+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:57:25.733127+0000","last_undegraded":"2024-03-24T11:57:25.733127+0000","last_fullsized":"2024-03-24T11:57:25.733127+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:25.642093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:56:25.642093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:41.995645+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":188,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.723236+0000","last_change":"2024-03-24T11:56:19.421031+0000","last_active":"2024-03-24T11:57:25.723236+0000","last_peered":"2024-03-24T11:57:25.723236+0000","last_clean":"2024-03-24T11:57:25.723236+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:57:25.723236+0000","last_undegraded":"2024-03-24T11:57:25.723236+0000","last_fullsized":"2024-03-24T11:57:25.723236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:19.420806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:56:19.420806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.704616+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":192,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.724762+0000","last_change":"2024-03-24T11:56:02.340688+0000","last_active":"2024-03-24T11:57:25.724762+0000","last_peered":"2024-03-24T11:57:25.724762+0000","last_clean":"2024-03-24T11:57:25.724762+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:57:25.724762+0000","last_undegraded":"2024-03-24T11:57:25.724762+0000","last_fullsized":"2024-03-24T11:57:25.724762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:56:02.340507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:56:02.340507+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:27.475514+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"143'12436","reported_seq":12620,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:26.287554+0000","last_change":"2024-03-24T11:56:11.635289+0000","last_active":"2024-03-24T11:57:26.287554+0000","last_peered":"2024-03-24T11:57:26.287554+0000","last_clean":"2024-03-24T11:57:26.287554+0000","last_became_active":"2024-03-24T11:54:16.239338+0000","last_became_peered":"2024-03-24T11:54:16.239338+0000","last_unstale":"2024-03-24T11:57:26.287554+0000","last_undegraded":"2024-03-24T11:57:26.287554+0000","last_fullsized":"2024-03-24T11:57:26.287554+0000","mapping_epoch":97,"log_start":"127'7300","ondisk_log_start":"127'7300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7969","last_scrub_stamp":"2024-03-24T11:56:11.635036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:15.215608+0000","last_clean_scrub_stamp":"2024-03-24T11:56:11.635036+0000","objects_scrubbed":7267,"log_size":5136,"log_dups_size":0,"ondisk_log_size":5136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:26.481082+0000","scrub_duration":2413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178913280,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12436,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.712886+0000","last_change":"2024-03-24T11:55:17.618704+0000","last_active":"2024-03-24T11:57:25.712886+0000","last_peered":"2024-03-24T11:57:25.712886+0000","last_clean":"2024-03-24T11:57:25.712886+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:57:25.712886+0000","last_undegraded":"2024-03-24T11:57:25.712886+0000","last_fullsized":"2024-03-24T11:57:25.712886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:55:17.618606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T11:55:17.618606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.690853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":189,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.730665+0000","last_change":"2024-03-24T11:56:21.558305+0000","last_active":"2024-03-24T11:57:25.730665+0000","last_peered":"2024-03-24T11:57:25.730665+0000","last_clean":"2024-03-24T11:57:25.730665+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:57:25.730665+0000","last_undegraded":"2024-03-24T11:57:25.730665+0000","last_fullsized":"2024-03-24T11:57:25.730665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:21.558180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:56:21.558180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:35.692594+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":200,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.717897+0000","last_change":"2024-03-24T11:55:19.645322+0000","last_active":"2024-03-24T11:57:25.717897+0000","last_peered":"2024-03-24T11:57:25.717897+0000","last_clean":"2024-03-24T11:57:25.717897+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:57:25.717897+0000","last_undegraded":"2024-03-24T11:57:25.717897+0000","last_fullsized":"2024-03-24T11:57:25.717897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:55:19.645176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:55:19.645176+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:30.690826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":251,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-24T11:57:25.728850+0000","last_change":"2024-03-24T11:57:01.681899+0000","last_active":"2024-03-24T11:57:25.728850+0000","last_peered":"2024-03-24T11:57:25.728850+0000","last_clean":"2024-03-24T11:57:25.728850+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:57:25.728850+0000","last_undegraded":"2024-03-24T11:57:25.728850+0000","last_fullsized":"2024-03-24T11:57:25.728850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:57:01.681784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:01.681784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:22.548424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2785148952,"num_objects":42499,"num_object_clones":0,"num_object_copies":127497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199411,"num_write_kb":7741057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8360177664,"data_stored":8360165448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89025,"ondisk_log_size":89025,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":304,"num_read_kb":299,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":461,"ondisk_log_size":461,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005122,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964808,"kb_used_data":865400,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92780344,"statfs":{"total":95995035648,"available":95007072256,"internally_reserved":0,"allocated":886169600,"data_stored":885817950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332169,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":967752,"kb_used_data":868280,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92777400,"statfs":{"total":95995035648,"available":95004057600,"internally_reserved":0,"allocated":889118720,"data_stored":888766912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1150464,"kb_used_data":1047096,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92594688,"statfs":{"total":95995035648,"available":94816960512,"internally_reserved":0,"allocated":1072226304,"data_stored":1071807399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1486284,"kb_used_data":1364668,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92258868,"statfs":{"total":95995035648,"available":94473080832,"internally_reserved":0,"allocated":1397420032,"data_stored":1397063177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1129096,"kb_used_data":1029752,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92616056,"statfs":{"total":95995035648,"available":94838841344,"internally_reserved":0,"allocated":1054466048,"data_stored":1054113671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1665300,"kb_used_data":1507660,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92079852,"statfs":{"total":95995035648,"available":94289768448,"internally_reserved":0,"allocated":1543843840,"data_stored":1543471413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920850,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963536,"kb_used_data":860032,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92781616,"statfs":{"total":95995035648,"available":95008374784,"internally_reserved":0,"allocated":880672768,"data_stored":880311814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659213,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":770680,"kb_used_data":703408,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92974472,"statfs":{"total":95995035648,"available":95205859328,"internally_reserved":0,"allocated":720289792,"data_stored":719879202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1392316416,"data_stored":1392312344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1051660288,"data_stored":1051656216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1038811136,"data_stored":1038811136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1541734400,"data_stored":1541734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":869793792,"data_stored":869793792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":710606848,"data_stored":710606848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":878317568,"data_stored":878313496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":876937216,"data_stored":876937216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:57:32.178 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T11:57:32.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_0 pg_num 26 2024-03-24T11:57:32.205 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:32.297 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:32.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:32.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:32.426 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:32.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:32.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:32.578 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:32.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:32.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:32.722 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:57:32.722 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:57:32.722 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:57:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:57:32.724 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:57:32.736 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:32.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:32.838 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:32.872 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:32.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:32.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:32 smithi064 ceph-mon[42243]: pgmap v203: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T11:57:32.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:32 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1000338245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:32.972 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:32 smithi064 ceph-mon[30223]: pgmap v203: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T11:57:32.972 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:32 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1000338245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:32.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:33.032 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:33.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:33.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:32 smithi193 ceph-mon[33593]: pgmap v203: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T11:57:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:32 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1000338245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:33.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:33.166 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:33.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:33.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:33.298 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:33.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:33.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:33.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:33.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:33.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:33.584 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:33.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:33.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:33.726 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:33.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:33.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:33.869 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:33.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:33.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:34.008 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:34.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:34.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:34.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:34.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:34.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:34.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:34.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:34.314 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:34.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:34.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:34.447 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:34.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:34.550 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:34.580 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:34.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:34.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:34.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:34.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:34.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:34.883 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:34.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:34.984 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:35.012 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:34 smithi064 ceph-mon[30223]: pgmap v204: 148 pgs: 148 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:35.012 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:34 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:35.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:34 smithi064 ceph-mon[42243]: pgmap v204: 148 pgs: 148 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:35.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:34 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:35.018 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:35.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:35.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:34 smithi193 ceph-mon[33593]: pgmap v204: 148 pgs: 148 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:35.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:34 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:35.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:35.167 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:35.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:35.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:35.324 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:35.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:35.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:35.436 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:35.443 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:57:35.443 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:57:35.450 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:35.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:35.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:35.599 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:35.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:35.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:35.736 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:35.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:35.791 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T11:57:35.836 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:35.867 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:35.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:35.969 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:36.000 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:36.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:36.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2646667915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:57:36.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:35 smithi064 ceph-mon[30223]: from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:57:36.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2646667915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:57:36.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:35 smithi064 ceph-mon[42243]: from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:57:36.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2646667915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:57:36.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:35 smithi193 ceph-mon[33593]: from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:57:36.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:36.145 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:36.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:36.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:36.289 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:36.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:36.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:36.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:36.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:36.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:57:36.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:36.564 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:36.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:36.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:36.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:36.693 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:36.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:36.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:36.855 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:36.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:36.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:36.988 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:36.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:37.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:57:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:57:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:36 smithi193 ceph-mon[33593]: pgmap v205: 148 pgs: 148 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:36 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:36 smithi193 ceph-mon[33593]: from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:57:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:36 smithi193 ceph-mon[33593]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:57:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:36 smithi193 ceph-mon[33593]: 5.7 scrub starts 2024-03-24T11:57:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:36 smithi193 ceph-mon[33593]: 5.7 scrub ok 2024-03-24T11:57:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:36 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:37.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:36 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:37.091 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:37.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[30223]: pgmap v205: 148 pgs: 148 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:37.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:37.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[30223]: from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:57:37.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[30223]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:57:37.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[30223]: 5.7 scrub starts 2024-03-24T11:57:37.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[30223]: 5.7 scrub ok 2024-03-24T11:57:37.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:37.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:37.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[42243]: pgmap v205: 148 pgs: 148 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:37.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:57:37.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[42243]: from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:57:37.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[42243]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:57:37.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[42243]: 5.7 scrub starts 2024-03-24T11:57:37.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[42243]: 5.7 scrub ok 2024-03-24T11:57:37.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:37.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:36 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:37.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T11:57:37.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:37.120 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:37.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:37.221 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:37.249 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:37.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:37.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:37.386 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:57:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:57:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:37.413 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:37.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:37.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:37.542 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:37.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:37.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:37.671 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:37.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:37.773 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:37.800 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:37.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:37.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:37.930 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:37.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:38.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:37 smithi064 ceph-mon[30223]: 7.1 scrub starts 2024-03-24T11:57:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:37 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:38.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:37 smithi064 ceph-mon[30223]: 7.1 scrub ok 2024-03-24T11:57:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:37 smithi064 ceph-mon[42243]: 7.1 scrub starts 2024-03-24T11:57:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:37 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:37 smithi064 ceph-mon[42243]: 7.1 scrub ok 2024-03-24T11:57:38.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:37 smithi193 ceph-mon[33593]: 7.1 scrub starts 2024-03-24T11:57:38.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:37 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:38.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:37 smithi193 ceph-mon[33593]: 7.1 scrub ok 2024-03-24T11:57:38.094 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:38.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:38.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:38.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:38.225 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:38.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:38.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:38.373 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:38.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:38.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:38.527 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:38.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:38.628 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:38.654 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:38.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:38.756 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:38.787 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:38.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:38.889 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:38.918 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:38.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[30223]: pgmap v207: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[30223]: 7.2 scrub starts 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[42243]: pgmap v207: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T11:57:38.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[42243]: 7.2 scrub starts 2024-03-24T11:57:38.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:38.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:38.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:38 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:39.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:39.044 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:39.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:38 smithi193 ceph-mon[33593]: pgmap v207: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-24T11:57:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:38 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T11:57:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:38 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T11:57:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:38 smithi193 ceph-mon[33593]: 7.2 scrub starts 2024-03-24T11:57:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:38 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:38 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:38 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:39.147 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:39.173 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:39.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:39.200 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:39.200 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:39.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:39.327 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:39.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:39.429 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:39.455 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:39.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:39.557 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:39.585 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:39.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:39.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:57:39.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:39.718 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:39.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:39.821 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:39.849 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:39.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:39.910 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 7 pg_num to 26 2024-03-24T11:57:39.934 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:57:39.934 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:57:39.934 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: 7.2 scrub ok 2024-03-24T11:57:39.934 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1177365672' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[30223]: from='client.34901 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: 7.2 scrub ok 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T11:57:39.935 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T11:57:39.936 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T11:57:39.936 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:57:39.936 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1177365672' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:57:39.936 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:39 smithi064 ceph-mon[42243]: from='client.34901 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:57:39.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:40.026 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:40.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:40.127 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:40.160 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:40.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:40.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:40.306 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:40.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:40.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:40.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:57:40.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:57:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: 7.2 scrub ok 2024-03-24T11:57:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T11:57:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T11:57:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T11:57:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T11:57:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:57:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1177365672' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:57:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:39 smithi193 ceph-mon[33593]: from='client.34901 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:57:40.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:40.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:40.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:40.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:40.569 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:40.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:40.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:40.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:40.713 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:40.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:40.814 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:40.844 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:40.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:40.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:40.975 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:40.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:41.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[30223]: pgmap v209: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[30223]: 7.8 scrub starts 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[30223]: from='client.34901 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[30223]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[42243]: pgmap v209: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[42243]: 7.8 scrub starts 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[42243]: from='client.34901 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[42243]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:57:41.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:41.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T11:57:41.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:40 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T11:57:41.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:41.118 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:41.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:41.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:41.220 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:41.248 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:41.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:41.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:40 smithi193 ceph-mon[33593]: pgmap v209: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:57:41.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:40 smithi193 ceph-mon[33593]: 7.8 scrub starts 2024-03-24T11:57:41.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:40 smithi193 ceph-mon[33593]: from='client.34901 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-24T11:57:41.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:40 smithi193 ceph-mon[33593]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:57:41.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:41.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:40 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T11:57:41.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:40 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T11:57:41.350 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:41.377 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:41.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:41.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:41.517 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:41.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:41.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:41.657 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:41.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:41.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:41.759 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:41.790 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:41.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:41.891 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:41.944 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:41.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:42.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:42.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:41 smithi064 ceph-mon[30223]: 7.8 scrub ok 2024-03-24T11:57:42.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:41 smithi064 ceph-mon[30223]: 4.14 scrub starts 2024-03-24T11:57:42.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:41 smithi064 ceph-mon[30223]: 4.14 scrub ok 2024-03-24T11:57:42.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:57:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:41 smithi064 ceph-mon[42243]: 7.8 scrub ok 2024-03-24T11:57:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:41 smithi064 ceph-mon[42243]: 4.14 scrub starts 2024-03-24T11:57:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:41 smithi064 ceph-mon[42243]: 4.14 scrub ok 2024-03-24T11:57:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:57:42.126 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:42.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:42.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:42.255 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:42.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:41 smithi193 ceph-mon[33593]: 7.8 scrub ok 2024-03-24T11:57:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:41 smithi193 ceph-mon[33593]: 4.14 scrub starts 2024-03-24T11:57:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:41 smithi193 ceph-mon[33593]: 4.14 scrub ok 2024-03-24T11:57:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:57:42.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:42.381 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:42.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:42.482 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:42.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:42.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:42.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:42.633 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:42.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:42.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:42.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:42.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:42.887 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:42.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:42.988 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:43.018 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:43.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:43.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:43.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:43.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:43.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[30223]: pgmap v211: 148 pgs: 148 active+clean; 1.9 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:57:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T11:57:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T11:57:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T11:57:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T11:57:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T11:57:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[30223]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:57:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[42243]: pgmap v211: 148 pgs: 148 active+clean; 1.9 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:57:43.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T11:57:43.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T11:57:43.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T11:57:43.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T11:57:43.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T11:57:43.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:42 smithi064 ceph-mon[42243]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:57:43.305 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:43.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:42 smithi193 ceph-mon[33593]: pgmap v211: 148 pgs: 148 active+clean; 1.9 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:57:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:42 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T11:57:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:42 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T11:57:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:42 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T11:57:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:42 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T11:57:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T11:57:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:42 smithi193 ceph-mon[33593]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:57:43.406 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:43.436 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:43.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:43.537 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:57:43.537 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:57:43.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:43.566 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:43.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:43.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:43.721 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:43.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:43.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:43.858 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:43.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:43.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:43.985 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:43.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:44.087 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:44.111 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:44.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:44.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T11:57:44.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T11:57:44.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T11:57:44.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T11:57:44.202 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[42243]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:57:44.202 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T11:57:44.202 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T11:57:44.202 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:57:44.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T11:57:44.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T11:57:44.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T11:57:44.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T11:57:44.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[30223]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:57:44.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T11:57:44.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T11:57:44.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:57:44.212 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:44.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T11:57:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:44.236 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:44.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:43 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T11:57:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:43 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T11:57:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:43 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T11:57:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:43 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T11:57:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:43 smithi193 ceph-mon[33593]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:57:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:43 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T11:57:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:43 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T11:57:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:57:44.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:44.363 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:44.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:44.464 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:44.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:44.489 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:44.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:44.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:44.615 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:44.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:44.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:44.740 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:44.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:44.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:44.866 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:44.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:44.968 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:44.991 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:44.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:45.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:45.119 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:45.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:45.146 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:57:45.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:57:45.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:57:45.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:57:45.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:57:45.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:57:45.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:57:45.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:57:45.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:57:45.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:57:45.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:57:45.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:57:45.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:57:45.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:57:45.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:57:45.150 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:57:45.150 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: pgmap v214: 158 pgs: 10 unknown, 148 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T11:57:45.150 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.9 scrub starts 2024-03-24T11:57:45.150 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.7 scrub starts 2024-03-24T11:57:45.150 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.2 scrub starts 2024-03-24T11:57:45.150 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.1 scrub starts 2024-03-24T11:57:45.150 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.0 scrub starts 2024-03-24T11:57:45.150 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.9 scrub ok 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.8 scrub starts 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.7 scrub ok 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.1 scrub ok 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.5 scrub starts 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.0 scrub ok 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.2 scrub ok 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.8 scrub ok 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: 7.5 scrub ok 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/725435292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:45.151 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[42243]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: pgmap v214: 158 pgs: 10 unknown, 148 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.9 scrub starts 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.7 scrub starts 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.2 scrub starts 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.1 scrub starts 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.0 scrub starts 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.9 scrub ok 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.8 scrub starts 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:57:45.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.7 scrub ok 2024-03-24T11:57:45.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.1 scrub ok 2024-03-24T11:57:45.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.5 scrub starts 2024-03-24T11:57:45.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.0 scrub ok 2024-03-24T11:57:45.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.2 scrub ok 2024-03-24T11:57:45.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.8 scrub ok 2024-03-24T11:57:45.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: 7.5 scrub ok 2024-03-24T11:57:45.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/725435292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:45.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:44 smithi064 ceph-mon[30223]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:57:45.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:45.245 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:45.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: pgmap v214: 158 pgs: 10 unknown, 148 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.9 scrub starts 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.7 scrub starts 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.2 scrub starts 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.1 scrub starts 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.0 scrub starts 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.9 scrub ok 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.8 scrub starts 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:57:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.7 scrub ok 2024-03-24T11:57:45.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.1 scrub ok 2024-03-24T11:57:45.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.5 scrub starts 2024-03-24T11:57:45.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.0 scrub ok 2024-03-24T11:57:45.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.2 scrub ok 2024-03-24T11:57:45.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.8 scrub ok 2024-03-24T11:57:45.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: 7.5 scrub ok 2024-03-24T11:57:45.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/725435292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:57:45.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:44 smithi193 ceph-mon[33593]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:57:45.347 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:45.371 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:45.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:45.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:45.497 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:45.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:45.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:45.622 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:45.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:45.723 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:45.747 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:45.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:45.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:45.874 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:45.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:45.898 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:57:45.898 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:57:45.898 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-24T11:57:45.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-24T11:57:45.977 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:46.048 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:46.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:46.149 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:46.178 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:46.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:46.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:46.299 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 7.4 scrub starts 2024-03-24T11:57:46.299 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 7.4 scrub ok 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 7.3 scrub starts 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 7.3 scrub ok 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 7.e scrub starts 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: 7.e scrub ok 2024-03-24T11:57:46.300 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[42243]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:57:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 7.4 scrub starts 2024-03-24T11:57:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 7.4 scrub ok 2024-03-24T11:57:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T11:57:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T11:57:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T11:57:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T11:57:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 7.3 scrub starts 2024-03-24T11:57:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 7.3 scrub ok 2024-03-24T11:57:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 7.e scrub starts 2024-03-24T11:57:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:57:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:57:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T11:57:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T11:57:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: 7.e scrub ok 2024-03-24T11:57:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:45 smithi064 ceph-mon[30223]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:57:46.332 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:46.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 7.4 scrub starts 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 7.4 scrub ok 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 7.3 scrub starts 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 7.3 scrub ok 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 7.e scrub starts 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T11:57:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T11:57:46.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: 7.e scrub ok 2024-03-24T11:57:46.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:45 smithi193 ceph-mon[33593]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:57:46.337 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:46.434 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:46.465 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:46.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:46.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:46.617 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:46.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:46.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:46.745 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:46.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:46.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:46.888 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:46.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:46.989 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:47.020 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:47.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:47.122 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:47.153 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:47.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:47.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:47.297 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:47.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: pgmap v217: 158 pgs: 10 unknown, 148 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 755 op/s 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.6 scrub starts 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.6 scrub ok 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13476 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13303 2024-03-24T11:57:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13534 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 148'13539 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13250 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13729 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13250 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13303 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13476 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 148'13084 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[30223]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: pgmap v217: 158 pgs: 10 unknown, 148 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 755 op/s 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.6 scrub starts 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.6 scrub ok 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13476 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13303 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13534 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 148'13539 2024-03-24T11:57:47.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13250 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13729 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13250 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13303 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13476 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 148'13084 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:46 smithi064 ceph-mon[42243]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:57:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:57:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: pgmap v217: 158 pgs: 10 unknown, 148 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 755 op/s 2024-03-24T11:57:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.6 scrub starts 2024-03-24T11:57:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.6 scrub ok 2024-03-24T11:57:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T11:57:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T11:57:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T11:57:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T11:57:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13476 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13303 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13534 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 148'13539 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 148'13250 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13729 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13250 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13303 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13476 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 148'13084 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13717 2024-03-24T11:57:47.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 148'13451 2024-03-24T11:57:47.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:46 smithi193 ceph-mon[33593]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 148'13515 2024-03-24T11:57:47.399 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:47.431 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:47.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:47.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:47.576 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:47.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:47.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:47.716 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:47.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:47.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:57:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:57:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:47.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:47.862 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:47.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:47.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:48.005 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:48.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:48.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:48.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:48.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:48.248 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:48.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:47 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:48.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:47 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:48.277 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:48.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:48.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:47 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:48.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:57:48.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:48.409 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:48.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:48.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:48.547 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:48.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:48.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:48.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:48.800 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:48.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:48.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:48.983 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:48.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:49.042 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[30223]: pgmap v219: 158 pgs: 5 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 1.6 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14638/77163 objects misplaced (18.970%); 3.8 MiB/s, 60 objects/s recovering 2024-03-24T11:57:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[30223]: 4.e scrub starts 2024-03-24T11:57:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[30223]: 4.e scrub ok 2024-03-24T11:57:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[30223]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:57:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:57:49.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:57:49.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[42243]: pgmap v219: 158 pgs: 5 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 1.6 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14638/77163 objects misplaced (18.970%); 3.8 MiB/s, 60 objects/s recovering 2024-03-24T11:57:49.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[42243]: 4.e scrub starts 2024-03-24T11:57:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[42243]: 4.e scrub ok 2024-03-24T11:57:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[42243]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:57:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:57:49.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:48 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:57:49.085 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:49.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:49.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:49.218 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:49.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:49.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:48 smithi193 ceph-mon[33593]: pgmap v219: 158 pgs: 5 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 1.6 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14638/77163 objects misplaced (18.970%); 3.8 MiB/s, 60 objects/s recovering 2024-03-24T11:57:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:48 smithi193 ceph-mon[33593]: 4.e scrub starts 2024-03-24T11:57:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:48 smithi193 ceph-mon[33593]: 4.e scrub ok 2024-03-24T11:57:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:48 smithi193 ceph-mon[33593]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:57:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:48 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:57:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:48 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:57:49.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:49.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:49.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:49.591 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:49.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:49.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:49.746 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:49.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:49.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:49.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:49.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:49.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:50.023 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:50.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:50.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:50.167 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:50.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:50.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:49 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T11:57:50.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:49 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T11:57:50.238 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:49 smithi193 ceph-mon[33593]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:57:50.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:50.300 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:50.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:50.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:50.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:49 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T11:57:50.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:49 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T11:57:50.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:49 smithi064 ceph-mon[30223]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:57:50.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:49 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T11:57:50.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:49 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T11:57:50.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:49 smithi064 ceph-mon[42243]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:57:50.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:50.446 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:50.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:50.548 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:50.580 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:50.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:50.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:50.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:50.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:50.821 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:50.851 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:50.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:50.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:50.992 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:50.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:51.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:50 smithi064 ceph-mon[30223]: pgmap v221: 158 pgs: 5 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 1.6 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 963 B/s rd, 0 op/s; 14638/77163 objects misplaced (18.970%); 3.1 MiB/s, 49 objects/s recovering 2024-03-24T11:57:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:50 smithi064 ceph-mon[42243]: pgmap v221: 158 pgs: 5 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 1.6 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 963 B/s rd, 0 op/s; 14638/77163 objects misplaced (18.970%); 3.1 MiB/s, 49 objects/s recovering 2024-03-24T11:57:51.095 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:51.125 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:51.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:51.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:51.264 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:51.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:50 smithi193 ceph-mon[33593]: pgmap v221: 158 pgs: 5 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 1.6 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 963 B/s rd, 0 op/s; 14638/77163 objects misplaced (18.970%); 3.1 MiB/s, 49 objects/s recovering 2024-03-24T11:57:51.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:51.406 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:51.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:51.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:51.536 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:51.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:51.640 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:51.669 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:51.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:51.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:51.797 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:51.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:51.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:51.925 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:51.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:52.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:52.060 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:52.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:52.161 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:52.189 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:52.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:52.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:52.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:52 smithi064 ceph-mon[30223]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:57:52.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:52 smithi064 ceph-mon[42243]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:57:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:52.323 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:52.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:52.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:52 smithi193 ceph-mon[33593]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:57:52.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:52.454 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:52.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:52.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:52.584 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:52.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:52.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:52.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:52.714 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:52.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:52.815 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:52.840 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:52.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:52.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:52.965 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:52.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:53.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:53.114 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:53.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:53.215 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:53.242 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:53.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[30223]: pgmap v223: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 13863/72495 objects misplaced (19.123%); 8.8 MiB/s, 141 objects/s recovering 2024-03-24T11:57:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T11:57:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T11:57:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[30223]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:57:53.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[30223]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:57:53.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[42243]: pgmap v223: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 13863/72495 objects misplaced (19.123%); 8.8 MiB/s, 141 objects/s recovering 2024-03-24T11:57:53.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T11:57:53.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T11:57:53.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[42243]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:57:53.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:53 smithi064 ceph-mon[42243]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:57:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:53 smithi193 ceph-mon[33593]: pgmap v223: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 13863/72495 objects misplaced (19.123%); 8.8 MiB/s, 141 objects/s recovering 2024-03-24T11:57:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:53 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T11:57:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:53 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T11:57:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:53 smithi193 ceph-mon[33593]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:57:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:53 smithi193 ceph-mon[33593]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:57:53.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:53.368 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:53.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:53.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:53.493 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:53.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:53.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:53.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:53.631 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:57:53.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:53.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:53.751 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:53.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:53.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:53.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:53.877 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:53.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:53.978 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:54.005 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:54.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:54.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:54.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:54.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:54.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 7.d scrub starts 2024-03-24T11:57:54.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T11:57:54.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T11:57:54.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 7.d scrub ok 2024-03-24T11:57:54.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T11:57:54.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T11:57:54.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 7.11 scrub starts 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 7.11 scrub ok 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 5.12 scrub starts 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: 5.12 scrub ok 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[30223]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 7.d scrub starts 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 7.d scrub ok 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 7.11 scrub starts 2024-03-24T11:57:54.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 7.11 scrub ok 2024-03-24T11:57:54.158 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 5.12 scrub starts 2024-03-24T11:57:54.158 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: 5.12 scrub ok 2024-03-24T11:57:54.158 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:54 smithi064 ceph-mon[42243]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:57:54.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:54.283 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:54.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 7.d scrub starts 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 7.d scrub ok 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 7.11 scrub starts 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 7.11 scrub ok 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 5.12 scrub starts 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: 5.12 scrub ok 2024-03-24T11:57:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:54 smithi193 ceph-mon[33593]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:57:54.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:54.417 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:54.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:54.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:54.563 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:54.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:54.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:54.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:54.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:54.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:223114 7.15 e156/150 7:afb0f9d7:::benchmark_data_smithi064_87013_object102157:head v 156'13746, mlcod=156'13744)", 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:54.800701+0000", 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0079283389999999995, 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079592210000000007, 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800701+0000", 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800701+0000", 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800703+0000", 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800711+0000", 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0309999999999998e-06 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800734+0000", 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3493999999999999e-05 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800746+0000", 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2181000000000001e-05 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.801067+0000", 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032059100000000001 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.801092+0000", 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5171e-05 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:223113 7.d e156/97 7:b42cca19:::benchmark_data_smithi064_87013_object102156:head v 156'13905, mlcod=156'13902)", 2024-03-24T11:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:54.800809+0000", 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0078203079999999998, 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079195189999999999, 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800809+0000", 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800809+0000", 2024-03-24T11:57:54.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800810+0000", 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2049999999999999e-06 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800817+0000", 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9739999999999996e-06 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800831+0000", 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3297e-05 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.800838+0000", 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1080000000000004e-06 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.802651+0000", 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001813222 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:54.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.802674+0000", 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3294000000000001e-05 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:223118 7.15 e156/150 7:aa9fc0a1:::benchmark_data_smithi064_87013_object102161:head v 156'13747, mlcod=156'13744)", 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:54.804173+0000", 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0044565730000000001, 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046208480000000003, 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:57:54.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.804173+0000", 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.804173+0000", 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.804174+0000", 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0979999999999999e-06 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.804181+0000", 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7129999999999999e-06 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.804189+0000", 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.259e-06 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.804196+0000", 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5610000000000004e-06 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.805602+0000", 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014068360000000001 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.805623+0000", 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0837000000000001e-05 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 3 2024-03-24T11:57:54.819 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:57:54.833 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:54.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:54.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:54.977 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:54.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T11:57:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T11:57:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T11:57:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T11:57:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T11:57:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T11:57:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T11:57:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:57:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T11:57:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T11:57:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T11:57:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T11:57:55.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:55.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:55.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:55.261 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:55.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: pgmap v226: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.2 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 7559/62016 objects misplaced (12.189%); 34 MiB/s, 538 objects/s recovering 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: 7.13 scrub starts 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: 7.12 scrub starts 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: 7.12 scrub ok 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: 7.13 scrub ok 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: 7.17 scrub starts 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: 7.17 scrub ok 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: 7.14 scrub starts 2024-03-24T11:57:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[42243]: 7.14 scrub ok 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: pgmap v226: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.2 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 7559/62016 objects misplaced (12.189%); 34 MiB/s, 538 objects/s recovering 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: 7.13 scrub starts 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: 7.12 scrub starts 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: 7.12 scrub ok 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: 7.13 scrub ok 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: 7.17 scrub starts 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: 7.17 scrub ok 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: 7.14 scrub starts 2024-03-24T11:57:55.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:55 smithi064 ceph-mon[30223]: 7.14 scrub ok 2024-03-24T11:57:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: pgmap v226: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.2 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 7559/62016 objects misplaced (12.189%); 34 MiB/s, 538 objects/s recovering 2024-03-24T11:57:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: 7.13 scrub starts 2024-03-24T11:57:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: 7.12 scrub starts 2024-03-24T11:57:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: 7.12 scrub ok 2024-03-24T11:57:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: 7.13 scrub ok 2024-03-24T11:57:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: 7.17 scrub starts 2024-03-24T11:57:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: 7.17 scrub ok 2024-03-24T11:57:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: 7.14 scrub starts 2024-03-24T11:57:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:55 smithi193 ceph-mon[33593]: 7.14 scrub ok 2024-03-24T11:57:55.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:55.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:55.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:55.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:55.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:57:55.681 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:55.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:57:55.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:55.823 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:55.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:55.925 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:57:55.943 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:57:55.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-24T11:57:55.957 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:55.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:56.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:56.115 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:56.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:56.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T11:57:56.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T11:57:56.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T11:57:56.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T11:57:56.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:56.162 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:56.162 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T11:57:56.162 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T11:57:56.162 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T11:57:56.162 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T11:57:56.162 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:56.162 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:56.217 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:56.245 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:56.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:56 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T11:57:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:56 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T11:57:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:56 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T11:57:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:56 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T11:57:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:57:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:56.347 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:56.382 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:56.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:56.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:56.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:56.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:56.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:56.659 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:56.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:56.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:56.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:56.799 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:56.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:56.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T11:57:56.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:56.976 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:56.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:57.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:57.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:57.225 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[30223]: pgmap v228: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.2 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 7559/62016 objects misplaced (12.189%); 34 MiB/s, 545 objects/s recovering 2024-03-24T11:57:57.225 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[30223]: 4.8 scrub starts 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[30223]: 4.8 scrub ok 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[42243]: pgmap v228: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.2 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 7559/62016 objects misplaced (12.189%); 34 MiB/s, 545 objects/s recovering 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[42243]: 4.8 scrub starts 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[42243]: 4.8 scrub ok 2024-03-24T11:57:57.226 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:57.229 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:57.257 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:57.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:57.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:57:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:57:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:57 smithi193 ceph-mon[33593]: pgmap v228: 158 pgs: 1 peering, 5 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 1.2 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 7559/62016 objects misplaced (12.189%); 34 MiB/s, 545 objects/s recovering 2024-03-24T11:57:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:57 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T11:57:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:57 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T11:57:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:57 smithi193 ceph-mon[33593]: 4.8 scrub starts 2024-03-24T11:57:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:57 smithi193 ceph-mon[33593]: 4.8 scrub ok 2024-03-24T11:57:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:57 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:57.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:57.405 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:57.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:57.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:57.536 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:57.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:57.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:57:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:57:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:57.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:57.707 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:57.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:57.808 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:57.839 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:57.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:57.940 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:57:57.972 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:57.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:57:58.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:58.081 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T11:57:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:58.101 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:58.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:58.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:57:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:58.243 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:58.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: 5.1 scrub starts 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: 5.1 scrub ok 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: pgmap v229: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 1.0 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4189/50241 objects misplaced (8.338%) 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2406000159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:58 smithi193 ceph-mon[33593]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:58.344 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:58.374 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:58.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:58.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: 5.1 scrub starts 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: 5.1 scrub ok 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: pgmap v229: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 1.0 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4189/50241 objects misplaced (8.338%) 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2406000159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:58.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[30223]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: 5.1 scrub starts 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: 5.1 scrub ok 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: pgmap v229: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 1.0 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4189/50241 objects misplaced (8.338%) 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2406000159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:58.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:58 smithi064 ceph-mon[42243]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:57:58.517 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:58.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:58.620 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:58.651 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:58.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:58.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:58.792 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:58.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:58.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:58.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:59.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:59.077 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:59.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:59.179 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:59.209 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:59.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: 6.a scrub starts 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[30223]: 6.a scrub ok 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T11:57:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T11:57:59.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T11:57:59.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:57:59.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:59.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:57:59.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: 6.a scrub starts 2024-03-24T11:57:59.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:57:59 smithi064 ceph-mon[42243]: 6.a scrub ok 2024-03-24T11:57:59.311 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:57:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T11:57:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T11:57:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T11:57:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T11:57:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:57:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:57:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:57:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: 6.a scrub starts 2024-03-24T11:57:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:57:59 smithi193 ceph-mon[33593]: 6.a scrub ok 2024-03-24T11:57:59.343 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:59.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:57:59.445 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:57:59.475 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:57:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:59.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:57:59.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:57:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:59.618 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:59.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:57:59.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:57:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:59.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:57:59.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:57:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:57:59.901 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:57:59.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:57:59.905 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:00.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:00.056 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:00.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:00.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:00.199 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:00.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:00.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T11:58:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T11:58:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T11:58:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T11:58:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[30223]: pgmap v231: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 1.0 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 825 B/s rd, 0 op/s; 4189/50241 objects misplaced (8.338%) 2024-03-24T11:58:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T11:58:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T11:58:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T11:58:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T11:58:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T11:58:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T11:58:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[42243]: pgmap v231: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 1.0 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 825 B/s rd, 0 op/s; 4189/50241 objects misplaced (8.338%) 2024-03-24T11:58:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T11:58:00.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:00 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T11:58:00.342 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:00.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:00.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:00.445 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:00.469 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:00.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:00.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:00 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T11:58:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:00 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T11:58:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:00 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T11:58:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:00 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T11:58:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:00 smithi193 ceph-mon[33593]: pgmap v231: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 1.0 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 825 B/s rd, 0 op/s; 4189/50241 objects misplaced (8.338%) 2024-03-24T11:58:00.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:00 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T11:58:00.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:00 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T11:58:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:00.602 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:00.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:00.703 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:00.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:00.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:00.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:00.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:00.879 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:00.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:00.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:00.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:01.007 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:01.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:01.108 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:01.137 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:01.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:01.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:01.277 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:01.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:01.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:01.409 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:01.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:01.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T11:58:01.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T11:58:01.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[30223]: 5.10 deep-scrub starts 2024-03-24T11:58:01.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[30223]: 5.10 deep-scrub ok 2024-03-24T11:58:01.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T11:58:01.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T11:58:01.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T11:58:01.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T11:58:01.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[42243]: 5.10 deep-scrub starts 2024-03-24T11:58:01.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[42243]: 5.10 deep-scrub ok 2024-03-24T11:58:01.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T11:58:01.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:01 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T11:58:01.493 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:01.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:01.563 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:01.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:01 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T11:58:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:01 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T11:58:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:01 smithi193 ceph-mon[33593]: 5.10 deep-scrub starts 2024-03-24T11:58:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:01 smithi193 ceph-mon[33593]: 5.10 deep-scrub ok 2024-03-24T11:58:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:01 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T11:58:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:01 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T11:58:01.663 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:01.694 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:01.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:01.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:01.833 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:01.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:01.934 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:01.967 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:01.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:02.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:02.105 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:02.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:02.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:02.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T11:58:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T11:58:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[30223]: 6.12 deep-scrub starts 2024-03-24T11:58:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[30223]: 6.12 deep-scrub ok 2024-03-24T11:58:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[30223]: pgmap v232: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 886 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 3990/43107 objects misplaced (9.256%) 2024-03-24T11:58:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[30223]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:58:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[30223]: 2.16 scrub starts 2024-03-24T11:58:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[30223]: 2.16 scrub ok 2024-03-24T11:58:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T11:58:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T11:58:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[42243]: 6.12 deep-scrub starts 2024-03-24T11:58:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[42243]: 6.12 deep-scrub ok 2024-03-24T11:58:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[42243]: pgmap v232: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 886 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 3990/43107 objects misplaced (9.256%) 2024-03-24T11:58:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[42243]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:58:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[42243]: 2.16 scrub starts 2024-03-24T11:58:02.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:02 smithi064 ceph-mon[42243]: 2.16 scrub ok 2024-03-24T11:58:02.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:02.391 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:02.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:02.493 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:02.526 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:02.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:02 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T11:58:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:02 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T11:58:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:02 smithi193 ceph-mon[33593]: 6.12 deep-scrub starts 2024-03-24T11:58:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:02 smithi193 ceph-mon[33593]: 6.12 deep-scrub ok 2024-03-24T11:58:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:02 smithi193 ceph-mon[33593]: pgmap v232: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 886 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 3990/43107 objects misplaced (9.256%) 2024-03-24T11:58:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:02 smithi193 ceph-mon[33593]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:58:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:02 smithi193 ceph-mon[33593]: 2.16 scrub starts 2024-03-24T11:58:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:02 smithi193 ceph-mon[33593]: 2.16 scrub ok 2024-03-24T11:58:02.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:02.679 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:02.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:02.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:02.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:02.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:02.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:03.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:03.090 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:03.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:03.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:03.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:03.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:03 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T11:58:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:03 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T11:58:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:03 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T11:58:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:03 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T11:58:03.340 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:03.369 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:03.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:03.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:03.502 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:03.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:03.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:58:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:03 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T11:58:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:03 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T11:58:03.605 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:03.634 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:03.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:03.701 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:58:03.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:03.748 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:58:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:03.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:03.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:03.792 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:58:03.834 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:58:03.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:03.872 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:58:03.912 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:58:03.914 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:03.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:03.953 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:58:03.991 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:58:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:58:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:58:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:58:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:58:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:58:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:58:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:58:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:58:04.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:04.054 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:04.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:04.156 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:04.189 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:04.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[30223]: 7.c scrub starts 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[30223]: 7.c scrub ok 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[30223]: pgmap v234: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 646 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2568/31578 objects misplaced (8.132%) 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2207350141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[30223]: 5.1b scrub starts 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[30223]: 5.1b scrub ok 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[42243]: 7.c scrub starts 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[42243]: 7.c scrub ok 2024-03-24T11:58:04.218 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[42243]: pgmap v234: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 646 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2568/31578 objects misplaced (8.132%) 2024-03-24T11:58:04.219 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2207350141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:04.219 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[42243]: 5.1b scrub starts 2024-03-24T11:58:04.219 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:04 smithi064 ceph-mon[42243]: 5.1b scrub ok 2024-03-24T11:58:04.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:04.321 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:04.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:04.423 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:04.453 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:04.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:04.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:04.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:04 smithi193 ceph-mon[33593]: 7.c scrub starts 2024-03-24T11:58:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:04 smithi193 ceph-mon[33593]: 7.c scrub ok 2024-03-24T11:58:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:04 smithi193 ceph-mon[33593]: pgmap v234: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 646 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2568/31578 objects misplaced (8.132%) 2024-03-24T11:58:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:04 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2207350141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:04 smithi193 ceph-mon[33593]: 5.1b scrub starts 2024-03-24T11:58:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:04 smithi193 ceph-mon[33593]: 5.1b scrub ok 2024-03-24T11:58:04.592 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:04.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:04.693 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:04.724 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:04.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:04.828 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:04.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:04.861 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:04.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:04.962 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:04.996 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:04.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:05.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:05.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:05.242 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:05.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:05.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:05.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:05.410 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:05.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:05.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:05.545 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:05.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:05.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:05 smithi064 ceph-mon[30223]: 7.f scrub starts 2024-03-24T11:58:05.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:05 smithi064 ceph-mon[30223]: 7.f scrub ok 2024-03-24T11:58:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:05 smithi064 ceph-mon[42243]: 7.f scrub starts 2024-03-24T11:58:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:05 smithi064 ceph-mon[42243]: 7.f scrub ok 2024-03-24T11:58:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:05 smithi193 ceph-mon[33593]: 7.f scrub starts 2024-03-24T11:58:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:05 smithi193 ceph-mon[33593]: 7.f scrub ok 2024-03-24T11:58:05.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:05.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:05.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:05.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:05.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:05.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:05.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:06.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:06.139 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:06.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:06.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:06.269 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:06.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:06.370 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:06.403 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:06.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:06.468 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:06 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T11:58:06.468 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:06 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T11:58:06.468 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:06 smithi064 ceph-mon[30223]: pgmap v235: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 646 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 777 op/s; 2568/31578 objects misplaced (8.132%) 2024-03-24T11:58:06.468 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:06 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T11:58:06.468 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:06 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T11:58:06.468 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:06 smithi064 ceph-mon[42243]: pgmap v235: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 646 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 777 op/s; 2568/31578 objects misplaced (8.132%) 2024-03-24T11:58:06.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:06.536 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:06.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:06.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:06 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T11:58:06.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:06 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T11:58:06.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:06 smithi193 ceph-mon[33593]: pgmap v235: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 646 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 777 op/s; 2568/31578 objects misplaced (8.132%) 2024-03-24T11:58:06.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:06.679 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:06.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:06.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:06.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:06.811 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:06.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:06.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:06.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:06.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:06.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:07.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:07.097 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:07.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:07.146 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T11:58:07.146 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T11:58:07.147 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T11:58:07.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:07.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:58:07.230 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:07.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:07.319 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:58:07.319 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:58:07.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:07.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 5.16 scrub starts 2024-03-24T11:58:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: 5.16 scrub ok 2024-03-24T11:58:07.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:07 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:07.376 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:07.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 5.16 scrub starts 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: 5.16 scrub ok 2024-03-24T11:58:07.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:58:07.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 5.16 scrub starts 2024-03-24T11:58:07.387 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: 5.16 scrub ok 2024-03-24T11:58:07.387 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:07.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:07.526 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:07.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:07.627 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:07.661 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:07.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:07.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:07.802 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:58:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:58:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:07.804 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:07.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:07.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:07.906 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:07.939 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:07.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:08.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:08.078 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:08.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:08.180 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:08.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:08.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:58:08.284 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:58:08.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-24T11:58:08.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:08.342 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:08.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:08.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:08.502 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:08.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:08.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T11:58:08.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T11:58:08.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[30223]: pgmap v237: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 234 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 897/11826 objects misplaced (7.585%) 2024-03-24T11:58:08.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[30223]: 2.1e scrub starts 2024-03-24T11:58:08.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[30223]: 2.1e scrub ok 2024-03-24T11:58:08.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T11:58:08.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T11:58:08.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[42243]: pgmap v237: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 234 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 897/11826 objects misplaced (7.585%) 2024-03-24T11:58:08.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[42243]: 2.1e scrub starts 2024-03-24T11:58:08.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:08 smithi064 ceph-mon[42243]: 2.1e scrub ok 2024-03-24T11:58:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:08 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T11:58:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:08 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T11:58:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:08 smithi193 ceph-mon[33593]: pgmap v237: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 234 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 897/11826 objects misplaced (7.585%) 2024-03-24T11:58:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:08 smithi193 ceph-mon[33593]: 2.1e scrub starts 2024-03-24T11:58:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:08 smithi193 ceph-mon[33593]: 2.1e scrub ok 2024-03-24T11:58:08.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:08.635 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:08.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:08.737 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:08.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:08.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:08.869 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:08.901 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:08.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:08.946 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:08.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T11:58:09.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:09.090 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:09.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:09.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:09.237 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:09.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:09.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[30223]: 5.1e deep-scrub starts 2024-03-24T11:58:09.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[30223]: 5.1e deep-scrub ok 2024-03-24T11:58:09.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T11:58:09.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T11:58:09.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T11:58:09.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[30223]: 4.0 scrub starts 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[30223]: 4.0 scrub ok 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[42243]: 5.1e deep-scrub starts 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[42243]: 5.1e deep-scrub ok 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[42243]: 4.0 scrub starts 2024-03-24T11:58:09.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:09 smithi064 ceph-mon[42243]: 4.0 scrub ok 2024-03-24T11:58:09.338 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:09.368 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:09.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:09.425 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Removed 120954 objects 2024-03-24T11:58:09.425 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Clean up completed and total clean up time :123.686 2024-03-24T11:58:09.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:09.494 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:09.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:09 smithi193 ceph-mon[33593]: 5.1e deep-scrub starts 2024-03-24T11:58:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:09 smithi193 ceph-mon[33593]: 5.1e deep-scrub ok 2024-03-24T11:58:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:09 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T11:58:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:09 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T11:58:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:09 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T11:58:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:09 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T11:58:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:09 smithi193 ceph-mon[33593]: 4.0 scrub starts 2024-03-24T11:58:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:09 smithi193 ceph-mon[33593]: 4.0 scrub ok 2024-03-24T11:58:09.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:09.620 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:09.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:09.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:09.761 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:09.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:09.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:09.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:09.988 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:10.013 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:10.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:10.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:10.143 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:10.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:10.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:10.269 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:10.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:10.370 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:10.396 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:10.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:10.455 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T11:58:10.455 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T11:58:10.455 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[30223]: 1.0 deep-scrub starts 2024-03-24T11:58:10.456 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[30223]: 1.0 deep-scrub ok 2024-03-24T11:58:10.456 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[30223]: pgmap v238: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 234 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 897/11826 objects misplaced (7.585%) 2024-03-24T11:58:10.456 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T11:58:10.456 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T11:58:10.456 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[42243]: 1.0 deep-scrub starts 2024-03-24T11:58:10.456 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[42243]: 1.0 deep-scrub ok 2024-03-24T11:58:10.456 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:10 smithi064 ceph-mon[42243]: pgmap v238: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 234 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 897/11826 objects misplaced (7.585%) 2024-03-24T11:58:10.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:10.522 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:10.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:10 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T11:58:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:10 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T11:58:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:10 smithi193 ceph-mon[33593]: 1.0 deep-scrub starts 2024-03-24T11:58:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:10 smithi193 ceph-mon[33593]: 1.0 deep-scrub ok 2024-03-24T11:58:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:10 smithi193 ceph-mon[33593]: pgmap v238: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 234 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 897/11826 objects misplaced (7.585%) 2024-03-24T11:58:10.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:10.648 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:10.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:10.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:10.778 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:10.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:10.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:10.902 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:10.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:11.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:11.028 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:11.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:11.130 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:11.158 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:11.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:11.260 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:11.289 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:11.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:11.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:11.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T11:58:11.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T11:58:11.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T11:58:11.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T11:58:11.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:11.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:11.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T11:58:11.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: 2.a scrub starts 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[30223]: 2.a scrub ok 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T11:58:11.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: 2.a scrub starts 2024-03-24T11:58:11.414 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:11 smithi064 ceph-mon[42243]: 2.a scrub ok 2024-03-24T11:58:11.414 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:11.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:11.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:11.544 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:11.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T11:58:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: 2.a scrub starts 2024-03-24T11:58:11.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:11 smithi193 ceph-mon[33593]: 2.a scrub ok 2024-03-24T11:58:11.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:11.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:11.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:11.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:11.796 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:11.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:11.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:11.921 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:11.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:12.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:12.047 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:12.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:12.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:12.173 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:12.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:12.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:12.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:12.302 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:12.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:12.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:12.432 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:12.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:12.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T11:58:12.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T11:58:12.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[30223]: pgmap v239: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 134 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 537 B/s rd, 0 B/s wr, 981 op/s; 771/6996 objects misplaced (11.021%) 2024-03-24T11:58:12.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[30223]: 4.4 scrub starts 2024-03-24T11:58:12.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[30223]: 4.4 scrub ok 2024-03-24T11:58:12.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[30223]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:58:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T11:58:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T11:58:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[42243]: pgmap v239: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 134 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 537 B/s rd, 0 B/s wr, 981 op/s; 771/6996 objects misplaced (11.021%) 2024-03-24T11:58:12.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[42243]: 4.4 scrub starts 2024-03-24T11:58:12.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[42243]: 4.4 scrub ok 2024-03-24T11:58:12.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:12 smithi064 ceph-mon[42243]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:58:12.558 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:12.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:12 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T11:58:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:12 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T11:58:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:12 smithi193 ceph-mon[33593]: pgmap v239: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 134 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 537 B/s rd, 0 B/s wr, 981 op/s; 771/6996 objects misplaced (11.021%) 2024-03-24T11:58:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:12 smithi193 ceph-mon[33593]: 4.4 scrub starts 2024-03-24T11:58:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:12 smithi193 ceph-mon[33593]: 4.4 scrub ok 2024-03-24T11:58:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:12 smithi193 ceph-mon[33593]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:58:12.660 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:12.688 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:12.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:12.790 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:12.817 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:12.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:12.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:12.943 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:12.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:13.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:13.070 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:13.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:13.171 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:13.200 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:13.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:13.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:13.326 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:13.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:13.360 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:13 smithi064 ceph-mon[30223]: 6.19 scrub starts 2024-03-24T11:58:13.361 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:13 smithi064 ceph-mon[30223]: 6.19 scrub ok 2024-03-24T11:58:13.361 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:13 smithi064 ceph-mon[30223]: 4.19 scrub starts 2024-03-24T11:58:13.361 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:13 smithi064 ceph-mon[30223]: 4.19 scrub ok 2024-03-24T11:58:13.361 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:13 smithi064 ceph-mon[42243]: 6.19 scrub starts 2024-03-24T11:58:13.361 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:13 smithi064 ceph-mon[42243]: 6.19 scrub ok 2024-03-24T11:58:13.361 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:13 smithi064 ceph-mon[42243]: 4.19 scrub starts 2024-03-24T11:58:13.361 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:13 smithi064 ceph-mon[42243]: 4.19 scrub ok 2024-03-24T11:58:13.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:13.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:13.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:13.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:13.578 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:13.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:13 smithi193 ceph-mon[33593]: 6.19 scrub starts 2024-03-24T11:58:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:13 smithi193 ceph-mon[33593]: 6.19 scrub ok 2024-03-24T11:58:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:13 smithi193 ceph-mon[33593]: 4.19 scrub starts 2024-03-24T11:58:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:13 smithi193 ceph-mon[33593]: 4.19 scrub ok 2024-03-24T11:58:13.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:13.703 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:58:13.705 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:13.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:13.718 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:58:13.737 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:58:13.757 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:58:13.777 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:58:13.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:58:13.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:13.818 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:58:13.830 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:13.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:13.839 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:58:13.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:58:13.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:58:13.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:58:13.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:58:13.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:58:13.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:58:13.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:58:13.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:58:13.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:13.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:13.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:14.057 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:14.086 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:14.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:14.176 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T11:58:14.188 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:14.215 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:14.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:14.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:14.340 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:14.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:14.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:14.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:14.467 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:14.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[30223]: pgmap v241: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 62/735 objects misplaced (8.435%) 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2664200261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1976421439' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T11:58:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[42243]: pgmap v241: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 62/735 objects misplaced (8.435%) 2024-03-24T11:58:14.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2664200261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:14.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T11:58:14.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T11:58:14.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1976421439' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:58:14.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:14 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T11:58:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:14 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T11:58:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:14 smithi193 ceph-mon[33593]: pgmap v241: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 62/735 objects misplaced (8.435%) 2024-03-24T11:58:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2664200261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:14 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T11:58:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:14 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T11:58:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1976421439' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:58:14.592 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:14.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:14.693 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:14.721 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:14.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:14.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:14.851 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:14.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:14.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:14.979 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:14.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:15.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:15.108 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:15.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T11:58:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T11:58:15.173 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T11:58:15.210 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:15.236 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:15.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:15.270 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-24T11:58:15.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-24T11:58:15.294 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[42243]: 2.5 deep-scrub starts 2024-03-24T11:58:15.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[42243]: 2.5 deep-scrub ok 2024-03-24T11:58:15.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:15.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1976421439' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:58:15.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[42243]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:58:15.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[30223]: 2.5 deep-scrub starts 2024-03-24T11:58:15.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[30223]: 2.5 deep-scrub ok 2024-03-24T11:58:15.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:15.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1976421439' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:58:15.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:15 smithi064 ceph-mon[30223]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:58:15.338 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:15.365 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:15.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:15.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:15.493 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:15.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:15 smithi193 ceph-mon[33593]: 2.5 deep-scrub starts 2024-03-24T11:58:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:15 smithi193 ceph-mon[33593]: 2.5 deep-scrub ok 2024-03-24T11:58:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:15 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:15 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1976421439' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:58:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:15 smithi193 ceph-mon[33593]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:58:15.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:15.622 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:15.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:15.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:15.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:15.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:15.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:15.874 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:15.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:58:15.939 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:58:15.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-24T11:58:15.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:16.002 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:16.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:16.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:16.132 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:16.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:16.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:16.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:16.258 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:16.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:16.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:16.383 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:16.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:16.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:16.513 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:16.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:16 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T11:58:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:16 smithi064 ceph-mon[30223]: pgmap v243: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 464 op/s; 62/735 objects misplaced (8.435%) 2024-03-24T11:58:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:16 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T11:58:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:16 smithi064 ceph-mon[42243]: pgmap v243: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 464 op/s; 62/735 objects misplaced (8.435%) 2024-03-24T11:58:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:16 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T11:58:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:16 smithi193 ceph-mon[33593]: pgmap v243: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 464 op/s; 62/735 objects misplaced (8.435%) 2024-03-24T11:58:16.614 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:16.642 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:16.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:16.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:16.769 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:16.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:16.871 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:16.895 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:16.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:16.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:17.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:17.022 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:17.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:17.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:17.148 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:17.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:17.179 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:17 smithi064 ceph-mon[30223]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:58:17.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:17.274 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:17.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:17.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:58:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:58:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:17 smithi193 ceph-mon[33593]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:58:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:17 smithi193 ceph-mon[33593]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:58:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:17 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:17.375 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:17.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:17.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:17.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:17 smithi064 ceph-mon[30223]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:58:17.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:17 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:17.503 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:58:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:58:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:17.503 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:17 smithi064 ceph-mon[42243]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:58:17.503 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:17 smithi064 ceph-mon[42243]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:58:17.503 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:17 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:17.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:17.533 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:17.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:17.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:17.657 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:17.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:17.759 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:17.787 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:17.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:17.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:17.914 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:17.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:18.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:18.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:18.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:18.166 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:18.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:18.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:18.296 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:18.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:18.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:18.425 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:18.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:18.527 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:18 smithi064 ceph-mon[30223]: pgmap v246: 158 pgs: 3 peering, 2 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 B/s wr, 102 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:18 smithi064 ceph-mon[42243]: pgmap v246: 158 pgs: 3 peering, 2 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 B/s wr, 102 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:18.554 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:18.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:18.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:18 smithi193 ceph-mon[33593]: pgmap v246: 158 pgs: 3 peering, 2 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 B/s wr, 102 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:18.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:58:18.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:18.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:18.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:18.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:18.834 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:18.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:18.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:18.936 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:18.965 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:18.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:19.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:19.094 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:19.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:19.197 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:19.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:19.226 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:19.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:19.327 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:19.355 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:19.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:19.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:19.467 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:19 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:58:19.467 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:19 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:58:19.483 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:19.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:19.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:19.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:19 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:58:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:19.613 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:19.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:19.714 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:19.742 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:19.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:19.843 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:19.872 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:19.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:19.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:58:19.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:20.000 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:20.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:20.057 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:20.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:20.127 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:20.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:20.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:20.253 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:20.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:20.354 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:20.381 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:20.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:20.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:20 smithi193 ceph-mon[33593]: pgmap v247: 158 pgs: 3 peering, 2 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:20.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:20 smithi064 ceph-mon[30223]: pgmap v247: 158 pgs: 3 peering, 2 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:20.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:20 smithi064 ceph-mon[42243]: pgmap v247: 158 pgs: 3 peering, 2 active+remapped+backfill_toofull, 153 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:20.507 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:20.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:20.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:20.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:20.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:20.764 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:20.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:20.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:20.893 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:20.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:20.995 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:21.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:21.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:21.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:21.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:21.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:21.280 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:21.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:21.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:21.405 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:21.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:21.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:21.535 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:21.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:21.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:21.664 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:21.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:21.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:21.791 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:21.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:21.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:21.893 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:21.921 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:21.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:22.022 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:22.051 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:22.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:22.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:22.180 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:22.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:22.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:22.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:22.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:22 smithi064 ceph-mon[30223]: pgmap v248: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:22.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:22 smithi064 ceph-mon[42243]: pgmap v248: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:22.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:22.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:22.440 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:22.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:22.542 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:22.570 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:22.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:22 smithi193 ceph-mon[33593]: pgmap v248: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:22.672 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:22.699 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:22.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:22.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:22.829 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:22.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:22.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:22.957 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:22.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:23.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:23.076 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:49:57.905681+0000", 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 503.494741777, 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.270907695, 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.14158", 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3351953651", 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905681+0000", 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905681+0000", 2024-03-24T11:58:23.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905683+0000", 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6060000000000001e-06 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905692+0000", 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3979999999999999e-06 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905704+0000", 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2123000000000001e-05 2024-03-24T11:58:23.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905715+0000", 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1523e-05 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905796+0000", 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0366999999999998e-05 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:57.905809+0000", 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2879e-05 2024-03-24T11:58:23.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.175618+0000", 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.269809356 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.175755+0000", 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013682000000000001 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.175884+0000", 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012961900000000001 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176237+0000", 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0003522 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176480+0000", 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243238 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176513+0000", 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870999999999997e-05 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176546+0000", 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3628999999999999e-05 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:49:58.176588+0000", 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2066e-05 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:7146 7.b 7:d672a46e:::benchmark_data_smithi064_87013_object7145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:38.009511+0000", 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.39091165599999, 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065759936000000005, 2024-03-24T11:58:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7146 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009511+0000", 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009511+0000", 2024-03-24T11:58:23.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009513+0000", 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0140000000000001e-06 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009536+0000", 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3357999999999999e-05 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009548+0000", 2024-03-24T11:58:23.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1867e-05 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009556+0000", 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2269999999999993e-06 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009617+0000", 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0744e-05 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009758+0000", 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141366 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.009906+0000", 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147517 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.010525+0000", 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061947 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.011838+0000", 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013121949999999999 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.075169+0000", 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063331081999999997 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.075218+0000", 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9199000000000001e-05 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:38.075271+0000", 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2896999999999999e-05 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:29171 7.2 e108/97 7:4440de80:::benchmark_data_smithi064_87013_object29170:head v 108'1739, mlcod=108'1737)", 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.116243+0000", 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 213.28417924600001, 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082093063999999993, 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116243+0000", 2024-03-24T11:58:23.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116243+0000", 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116245+0000", 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.674e-06 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116272+0000", 2024-03-24T11:58:23.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6738999999999999e-05 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116282+0000", 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0219e-05 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116290+0000", 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2919999999999996e-06 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116378+0000", 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7276999999999998e-05 2024-03-24T11:58:23.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.116401+0000", 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3910999999999999e-05 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.117630+0000", 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001228951 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198336+0000", 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080706000999999999 2024-03-24T11:58:23.093 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:29183 7.0 e108/97 7:0db6911e:::benchmark_data_smithi064_87013_object29182:head v 108'1867, mlcod=108'1865)", 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.118079+0000", 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 213.28234361099999, 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084733374, 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.118079+0000", 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.118079+0000", 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.118081+0000", 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7260000000000001e-06 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.118133+0000", 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2744000000000002e-05 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198323+0000", 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080189773000000006 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198332+0000", 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5129999999999996e-06 2024-03-24T11:58:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198415+0000", 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3007e-05 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198438+0000", 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3723e-05 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.202601+0000", 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041624940000000001 2024-03-24T11:58:23.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.202812+0000", 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211394 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:29191 7.d e108/97 7:bfe44579:::benchmark_data_smithi064_87013_object29190:head v 108'1849, mlcod=108'1847)", 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.120732+0000", 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 213.27969054299999, 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079646350000000005, 2024-03-24T11:58:23.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.120732+0000", 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.120732+0000", 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.120734+0000", 2024-03-24T11:58:23.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1339999999999998e-06 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.120902+0000", 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016809600000000001 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198312+0000", 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077409338999999994 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198324+0000", 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2622e-05 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198389+0000", 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4331999999999993e-05 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198414+0000", 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5607000000000001e-05 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.200266+0000", 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00185216 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.200378+0000", 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011205999999999999 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.34742.0:29182 7.4 e108/97 7:2bc31f0c:::benchmark_data_smithi064_87013_object29181:head v 108'1855, mlcod=108'1853)", 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.121083+0000", 2024-03-24T11:58:23.102 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 213.279339126, 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080973751999999996, 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.121083+0000", 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.121083+0000", 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.121085+0000", 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.121122+0000", 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7608999999999999e-05 2024-03-24T11:58:23.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198316+0000", 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077193500999999998 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198326+0000", 2024-03-24T11:58:23.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0363999999999999e-05 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198389+0000", 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-05 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.198414+0000", 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4922e-05 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.202012+0000", 2024-03-24T11:58:23.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035973120000000001 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.202057+0000", 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5639000000000001e-05 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30169 7.b 7:d7bf31c7:::benchmark_data_smithi064_87013_object30168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.766037+0000", 2024-03-24T11:58:23.107 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.63438580799999, 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092283463999999996, 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30169 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766037+0000", 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766037+0000", 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766039+0000", 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9319999999999999e-06 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766063+0000", 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4379000000000002e-05 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766074+0000", 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0791000000000001e-05 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766081+0000", 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6120000000000001e-06 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.766127+0000", 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5946999999999999e-05 2024-03-24T11:58:23.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767175+0000", 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010475689999999999 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767459+0000", 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000284505 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.770793+0000", 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033337010000000001 2024-03-24T11:58:23.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.770836+0000", 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.286e-05 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858166+0000", 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087330388999999994 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858273+0000", 2024-03-24T11:58:23.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106161 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858320+0000", 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7617999999999997e-05 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30176 7.b 7:de597230:::benchmark_data_smithi064_87013_object30175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.767071+0000", 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.63335156700001, 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091388602999999999, 2024-03-24T11:58:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30176 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767071+0000", 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767071+0000", 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767072+0000", 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.508e-06 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767095+0000", 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2299000000000001e-05 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767106+0000", 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1277000000000001e-05 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.767116+0000", 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7259999999999997e-06 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.769209+0000", 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020930699999999998 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.769317+0000", 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108312 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.769459+0000", 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014156499999999999 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.770991+0000", 2024-03-24T11:58:23.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00153232 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.771002+0000", 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0818999999999999e-05 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858366+0000", 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087364081999999996 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858419+0000", 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3114e-05 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.858460+0000", 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0510999999999999e-05 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30186 7.b 7:d1250a05:::benchmark_data_smithi064_87013_object30185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.773372+0000", 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.62705073399999, 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086805445999999994, 2024-03-24T11:58:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30186 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773372+0000", 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773372+0000", 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773374+0000", 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0889999999999998e-06 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773397+0000", 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3589999999999999e-05 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773410+0000", 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2602000000000001e-05 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773419+0000", 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.67e-06 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773472+0000", 2024-03-24T11:58:23.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3619000000000003e-05 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773578+0000", 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105398 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.773724+0000", 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014664800000000001 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.774987+0000", 2024-03-24T11:58:23.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001262195 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.775084+0000", 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7656000000000001e-05 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860075+0000", 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084991026999999997 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860134+0000", 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9039999999999997e-05 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860177+0000", 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2911999999999998e-05 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30194 7.b 7:df8f0b15:::benchmark_data_smithi064_87013_object30193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.779309+0000", 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.62111336300001, 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081026438000000006, 2024-03-24T11:58:23.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30194 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779309+0000", 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779309+0000", 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779311+0000", 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6840000000000001e-06 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779330+0000", 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9006999999999999e-05 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779340+0000", 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0271999999999999e-05 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779346+0000", 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8119999999999996e-06 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779398+0000", 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1557999999999997e-05 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779490+0000", 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2237000000000005e-05 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.779579+0000", 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9030000000000006e-05 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.780593+0000", 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001013746 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.780742+0000", 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149357 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860246+0000", 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079504107000000004 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860296+0000", 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0071999999999998e-05 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.860336+0000", 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9555999999999999e-05 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:30196 7.b 7:d819836d:::benchmark_data_smithi064_87013_object30195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:54:48.783097+0000", 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.61732591200001, 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079021147, 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30196 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783097+0000", 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783097+0000", 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783098+0000", 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.55e-06 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783136+0000", 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8141e-05 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783146+0000", 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783152+0000", 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.677e-06 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783194+0000", 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2116000000000001e-05 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783284+0000", 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9914999999999995e-05 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.783375+0000", 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0919000000000001e-05 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.784220+0000", 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084454199999999999 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.784411+0000", 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019179000000000001 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.861959+0000", 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077547952000000003 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.862068+0000", 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109158 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:54:48.862118+0000", 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9378000000000001e-05 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:128890 7.b 7:d38a7938:::benchmark_data_smithi064_87013_object7933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:13.809815+0000", 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.590607577, 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11845847499999999, 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128890 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.809815+0000", 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.809815+0000", 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.809815+0000", 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000005e-07 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.809818+0000", 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2029999999999999e-06 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.809823+0000", 2024-03-24T11:58:23.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0690000000000001e-06 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.809827+0000", 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6870000000000002e-06 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.813454+0000", 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036263440000000001 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.813567+0000", 2024-03-24T11:58:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011366699999999999 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.813683+0000", 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011524299999999999 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.818336+0000", 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004653229 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.818581+0000", 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024459500000000001 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.928168+0000", 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109587093 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.928223+0000", 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4963999999999998e-05 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.928273+0000", 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0872999999999999e-05 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:128916 7.b 7:d595426e:::benchmark_data_smithi064_87013_object7959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:13.835361+0000", 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.56506154, 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093074902000000001, 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128916 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835361+0000", 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835361+0000", 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835363+0000", 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.841e-06 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835368+0000", 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1610000000000002e-06 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835380+0000", 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1491000000000001e-05 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835386+0000", 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4330000000000001e-06 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835434+0000", 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8262e-05 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835543+0000", 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108578 2024-03-24T11:58:23.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.835671+0000", 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012804299999999999 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.841279+0000", 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056080660000000001 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.841317+0000", 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8485000000000002e-05 2024-03-24T11:58:23.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.928312+0000", 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086994872000000001 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.928395+0000", 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3228999999999999e-05 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.928436+0000", 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0441000000000001e-05 2024-03-24T11:58:23.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:128919 7.b 7:d82b645d:::benchmark_data_smithi064_87013_object7962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:13.838119+0000", 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.562303482, 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092515010999999994, 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128919 2024-03-24T11:58:23.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838119+0000", 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838119+0000", 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838121+0000", 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5260000000000001e-06 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838126+0000", 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4090000000000002e-06 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838139+0000", 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2877999999999999e-05 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838147+0000", 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3639999999999999e-06 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838412+0000", 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026490299999999998 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838577+0000", 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016471500000000001 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.838693+0000", 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116042 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.841390+0000", 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026970700000000002 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.841525+0000", 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013487199999999999 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.930537+0000", 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089011902000000004 2024-03-24T11:58:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.930591+0000", 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4410000000000003e-05 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.930634+0000", 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2920000000000002e-05 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:128928 7.b 7:d87f8fdf:::benchmark_data_smithi064_87013_object7971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:13.854671+0000", 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.54575178100001, 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076246126999999997, 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128928 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854671+0000", 2024-03-24T11:58:23.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854671+0000", 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854673+0000", 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0849999999999999e-06 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854678+0000", 2024-03-24T11:58:23.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1059999999999999e-06 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854689+0000", 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0958000000000001e-05 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854695+0000", 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1109999999999996e-06 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854743+0000", 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7939999999999998e-05 2024-03-24T11:58:23.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854829+0000", 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6411999999999999e-05 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.854909+0000", 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9954000000000001e-05 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.856990+0000", 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020810669999999998 2024-03-24T11:58:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.857031+0000", 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0577999999999997e-05 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.930827+0000", 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073796031999999998 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.930881+0000", 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4392999999999999e-05 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:13.930917+0000", 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5491000000000003e-05 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:133351 7.b 7:d5aec8e3:::benchmark_data_smithi064_87013_object12394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:18.614675+0000", 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.7857478, 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.068542967999999996, 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 133351 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.614675+0000", 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.614675+0000", 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.614677+0000", 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.277e-06 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.614683+0000", 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4690000000000002e-06 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.614696+0000", 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2872e-05 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.614706+0000", 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1400000000000006e-06 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.614765+0000", 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9655999999999999e-05 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.614890+0000", 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000125272 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.615006+0000", 2024-03-24T11:58:23.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115927 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.618540+0000", 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035337459999999999 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.618941+0000", 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040095499999999999 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.682978+0000", 2024-03-24T11:58:23.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064036957000000005 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.683039+0000", 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0711000000000002e-05 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.683218+0000", 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017898600000000001 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:177298 7.b 7:d3c6d09e:::benchmark_data_smithi064_87013_object56341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:04.963835+0000", 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.436587149999994, 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078906805999999996, 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 177298 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.963835+0000", 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.963835+0000", 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.963838+0000", 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8499999999999998e-06 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.963847+0000", 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7679999999999997e-06 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.963877+0000", 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9986999999999999e-05 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.963889+0000", 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1528999999999999e-05 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.964077+0000", 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018801199999999999 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.964226+0000", 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014928599999999999 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.964345+0000", 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011903799999999999 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.967010+0000", 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00266541 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:04.967156+0000", 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014524899999999999 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:05.042635+0000", 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075479409999999997 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:05.042693+0000", 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8257999999999997e-05 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:05.042742+0000", 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215974 7.12 7:49827d29:::benchmark_data_smithi064_87013_object95017:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T11:58:23.171 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.912925+0000", 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.487497634, 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1524356259999999, 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215974 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912925+0000", 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912925+0000", 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912927+0000", 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9910000000000001e-06 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912932+0000", 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5979999999999999e-06 2024-03-24T11:58:23.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.004832+0000", 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091900446999999996 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.004846+0000", 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4114000000000001e-05 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009898+0000", 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050523190000000004 2024-03-24T11:58:23.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009908+0000", 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.065275+0000", 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0553668060000001 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.065361+0000", 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5686999999999995e-05 2024-03-24T11:58:23.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:227695 7.b 7:d4c7ce98:::benchmark_data_smithi064_87013_object106738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:58.356688+0000", 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.043734706999999, 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065199378000000002, 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 227695 2024-03-24T11:58:23.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.356688+0000", 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.356688+0000", 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.356690+0000", 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9889999999999999e-06 2024-03-24T11:58:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.356696+0000", 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7710000000000003e-06 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.356708+0000", 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2656e-05 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.356717+0000", 2024-03-24T11:58:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2420000000000006e-06 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.356777+0000", 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9191000000000003e-05 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.356906+0000", 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129264 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.357019+0000", 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112557 2024-03-24T11:58:23.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360108+0000", 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030894189999999999 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360207+0000", 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8871999999999998e-05 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.421787+0000", 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061579912000000001 2024-03-24T11:58:23.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.421839+0000", 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2049999999999998e-05 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.421887+0000", 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8455000000000003e-05 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.181 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:227702 7.b 7:d2a1f8e6:::benchmark_data_smithi064_87013_object106745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:58.360288+0000", 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.040135024000001, 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073568538000000003, 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 227702 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360288+0000", 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360288+0000", 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360290+0000", 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360295+0000", 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360306+0000", 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1354e-05 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360314+0000", 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7659999999999998e-06 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.360875+0000", 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056080500000000005 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.418914+0000", 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058039009000000003 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.419049+0000", 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135253 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.422021+0000", 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029712779999999999 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.433699+0000", 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011678457999999999 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.433757+0000", 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7519000000000003e-05 2024-03-24T11:58:23.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.433810+0000", 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3378999999999998e-05 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:58.433856+0000", 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6110999999999999e-05 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:58:23.187 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:58:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:23.189 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:23.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:23.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:23.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:23.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:23.416 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:23.444 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:23.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:23.546 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:23.574 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:23.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:23.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:23.702 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:23.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:23.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:23.830 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:23.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:23.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T11:58:23.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:23.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:23.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:24.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:24.082 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:24.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:24.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:24.208 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:24.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:24.309 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:24.338 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:24.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:24.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:24 smithi064 ceph-mon[30223]: pgmap v249: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:24.393 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:24 smithi064 ceph-mon[42243]: pgmap v249: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:24.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:24.463 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:24.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:24.526 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T11:58:24.527 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T11:58:24.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:24 smithi193 ceph-mon[33593]: pgmap v249: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:24.588 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:24.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:24.690 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:24.719 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:24.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:24.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:24.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:24.845 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:24.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:24.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:24.971 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:24.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:25.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:25.096 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:25.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:25.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:25.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:25.323 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:25.352 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:25.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:25.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:25.476 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:25.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:25.578 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:25.605 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:25.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:25.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:25.707 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:25.735 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:25.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:25.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:58:25.845 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:58:25.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-24T11:58:25.864 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:25.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:25.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:25.990 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:25.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:26.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:26.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:26.116 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:26.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:26.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:26.240 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:26.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:26.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:26.371 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:26.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:26.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:26.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:26.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:26.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:26.624 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:26.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:26.725 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:26.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:26 smithi064 ceph-mon[30223]: pgmap v250: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:26.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:26.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:26.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:26 smithi064 ceph-mon[42243]: pgmap v250: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:26.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:26.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:26.752 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:26.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:26.854 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:26.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:26.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:26.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:27.007 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:27.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:27.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:26 smithi193 ceph-mon[33593]: pgmap v250: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:27.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:27.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:27.087 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:58:27.087 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:58:27.088 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:58:27.088 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:58:27.090 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:58:27.095 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:27.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:27.114 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:58:27.135 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:27.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:27.137 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:58:27.157 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:58:27.158 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:58:27.158 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:58:27.158 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:58:27.158 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:58:27.158 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:58:27.159 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:58:27.159 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:58:27.159 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:58:27.237 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:27.264 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:27.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:27.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:58:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:58:27.366 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:27.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:27.494 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:27.518 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:27.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:27.620 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:27.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:27.721 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:58:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:27.735 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_0' removed 2024-03-24T11:58:27.747 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:27.774 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:27.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:27.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:27.913 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:27.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:28.014 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:28.038 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:28.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:28.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1870006774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:28.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2754532239' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:58:28.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-mon[30223]: from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:58:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1870006774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2754532239' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:58:28.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:27 smithi064 ceph-mon[42243]: from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:58:28.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:27 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1870006774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:28.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:27 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:28.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:27 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2754532239' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:58:28.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:27 smithi193 ceph-mon[33593]: from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:58:28.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:28.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:28.164 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:28.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:28.265 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:28.289 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:28.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:28.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:28.413 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:28.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:28.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:28.539 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:28.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:28.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:28.645 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T11:58:28.646 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T11:58:28.646 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-24T11:58:28.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-24T11:58:28.666 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:28.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:28.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:28.792 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:28.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:28.874 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:28 smithi064 ceph-mon[30223]: pgmap v251: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 973 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:28.874 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:28 smithi064 ceph-mon[30223]: from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:58:28.874 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:28 smithi064 ceph-mon[30223]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:58:28.874 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:28 smithi064 ceph-mon[42243]: pgmap v251: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 973 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:28.874 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:28 smithi064 ceph-mon[42243]: from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:58:28.874 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:28 smithi064 ceph-mon[42243]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:58:28.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:28.918 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:28.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:29.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:29.044 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:29.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:28 smithi193 ceph-mon[33593]: pgmap v251: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 973 B/s rd, 0 op/s; 14/735 objects misplaced (1.905%) 2024-03-24T11:58:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:28 smithi193 ceph-mon[33593]: from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:58:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:28 smithi193 ceph-mon[33593]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:58:29.146 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:29.175 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:29.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:29.275 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:29.275 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:29.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:29.304 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:29.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:29.406 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:29.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:29.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:29.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:29.566 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:29.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:29.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:29.693 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:29.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:29.794 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:29.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:29 smithi064 ceph-mon[30223]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-24T11:58:29.812 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:29 smithi064 ceph-mon[42243]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-24T11:58:29.821 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:29.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:29.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:29.947 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:29.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:29.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:30.049 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:30.077 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:30.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:30.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:30.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:29 smithi193 ceph-mon[33593]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-24T11:58:30.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:30.204 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:30.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:30.305 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:30.335 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:30.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:30.436 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:30.464 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:30.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:30.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:30.592 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:30.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:30.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:30.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:30.719 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:30.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:30.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:30.847 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:30.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:30.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:30.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:30.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:31.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:30 smithi064 ceph-mon[30223]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:58:31.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:30 smithi064 ceph-mon[42243]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:58:31.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:31.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:30 smithi193 ceph-mon[33593]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:58:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:31.100 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:31.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:31.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:31.227 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:31.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:31.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:58:31.329 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:31.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:31.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:31.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:31.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:31.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:31.619 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:31.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:31.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:31.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:31.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:31.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:31.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:31.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:32.001 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:32.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:32.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:32.130 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:32.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:32.232 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:32.260 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:32.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:32.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:32.387 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:32.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:32.488 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:32.518 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:32.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:32.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:32.646 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:32.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:32.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:32.758 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T11:58:32.758 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T11:58:32.758 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T11:58:32.758 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T11:58:32.759 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T11:58:32.759 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T11:58:32.759 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T11:58:32.759 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:58:32.759 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T11:58:32.759 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T11:58:32.759 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T11:58:32.759 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T11:58:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:32.777 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:32.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:32.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:32.903 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:32.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:33.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:33.028 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:33.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:33.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:32 smithi064 ceph-mon[30223]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:58:33.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:32 smithi064 ceph-mon[30223]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:58:33.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:32 smithi064 ceph-mon[42243]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:58:33.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:32 smithi064 ceph-mon[42243]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:58:33.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:32 smithi193 ceph-mon[33593]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:58:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:32 smithi193 ceph-mon[33593]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:58:33.130 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:33.158 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:33.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:33.260 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:33.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:33.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:33.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:33.413 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:33.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:58:33.490 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:58:33.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-24T11:58:33.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:33.557 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:33.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:33.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:33.683 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:33.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:33.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:33.810 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:33.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:33.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:33.940 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:33.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:34.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:33 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:58:34.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:33 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:58:34.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:33 smithi064 ceph-mon[30223]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:58:34.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:33 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:58:34.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:33 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:58:34.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:33 smithi064 ceph-mon[42243]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:58:34.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:34.065 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:34.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:34.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:33 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:58:34.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:33 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:58:34.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:33 smithi193 ceph-mon[33593]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:58:34.167 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:34.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:34.197 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:34.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:34.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:34.324 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:34.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:34.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:34.449 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:34.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:34.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:34.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:34.676 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:34.704 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:34.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:34.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:34.834 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:34.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:34.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:34.960 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:34.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:34.994 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:34 smithi064 ceph-mon[30223]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:58:34.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:34 smithi064 ceph-mon[42243]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:58:35.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:35.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:34 smithi193 ceph-mon[33593]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:58:35.087 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:35.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:35.189 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:35.217 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:35.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:35.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:35.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:35.400 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:58:35.400 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:58:35.445 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:35.474 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:35.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:35.575 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:35.602 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:35.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:35.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:35.728 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:35.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:35.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:35.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:35.854 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:35.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:35.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:35.978 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:35.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:36.067 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:36.067 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:36.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T11:58:36.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:36.108 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:36.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:36.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:36.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:36.234 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:36.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:36.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:36.361 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:36.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:36.462 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:36.490 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:36.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:36.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:36.616 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:36.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:36.717 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:36.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:36.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:36.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:36.875 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:36.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:36.918 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:36 smithi064 ceph-mon[30223]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-24T11:58:36.918 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:36 smithi064 ceph-mon[30223]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:58:36.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:36 smithi064 ceph-mon[42243]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-24T11:58:36.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:36 smithi064 ceph-mon[42243]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:58:36.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:37.000 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:37.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:37.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:36 smithi193 ceph-mon[33593]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-24T11:58:37.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:36 smithi193 ceph-mon[33593]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:58:37.102 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:37.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:37.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:37.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:37.256 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:37.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:37.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:58:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:58:37.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:37.382 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:37.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:37.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:37.506 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:37.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:37.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:58:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:58:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:37.609 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:37.637 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:37.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:37.739 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:37.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:37.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:37.804 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T11:58:37.868 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:37.895 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:37.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:37.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:38.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:37 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1874376905' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:58:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:37 smithi064 ceph-mon[30223]: from='client.34976 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:58:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:37 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1874376905' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:58:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:37 smithi064 ceph-mon[42243]: from='client.34976 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:58:38.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:37 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1874376905' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:58:38.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:37 smithi193 ceph-mon[33593]: from='client.34976 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:58:38.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:38.148 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:38.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:38.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:38.274 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:38.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:38.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:38.401 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:38.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:38.503 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:38.527 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:38.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:38.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:38.652 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:38.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:38.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:38.777 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:38.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:38.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:38.903 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:38.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:38.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:39.006 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:39.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:39.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[30223]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:39.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:39.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[30223]: from='client.34976 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:58:39.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[30223]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:58:39.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:39.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:39.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:39.023 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:39.023 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[42243]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:39.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:39.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[42243]: from='client.34976 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:58:39.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[42243]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:58:39.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:39.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:39.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:38 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:39.029 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:39.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:39.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:38 smithi193 ceph-mon[33593]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:38 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:38 smithi193 ceph-mon[33593]: from='client.34976 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:58:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:38 smithi193 ceph-mon[33593]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:58:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:38 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:38 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:39.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:38 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:39.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:39.157 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:39.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:39.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:39.282 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:39.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:39.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:39.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:39.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:39.533 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:39.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:39.635 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:39.659 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:39.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:39.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:39.785 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:39.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:39.886 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:39.911 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:39.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:40.012 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:40.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:40.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:40.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:39 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:39 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:58:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:39 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:40.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:39 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:40.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:39 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:58:40.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:39 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:40.059 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:58:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:58:40.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:39 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:40.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:39 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:58:40.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:39 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:40.097 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:58:40.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:58:40.137 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:58:40.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:40.156 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:58:40.163 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:40.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:40.177 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:58:40.198 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:58:40.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:58:40.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:58:40.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:58:40.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:58:40.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:58:40.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:58:40.200 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:58:40.200 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:58:40.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:40.289 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:40.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:40.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:40.461 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:40.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:40.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:40.586 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:40.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:40.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:40.716 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:40.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:40.818 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:40.846 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:40.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:40.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:40.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:40.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:41.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[30223]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-24T11:58:41.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/119938477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[42243]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/119938477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:41.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:40 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:41.074 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:41.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:40 smithi193 ceph-mon[33593]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-24T11:58:41.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:40 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/119938477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:41.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:40 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:58:41.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:40 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:58:41.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:41.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:40 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:41.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:41.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:41.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:41.231 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:41.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:41.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:41.309 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T11:58:41.309 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T11:58:41.310 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T11:58:41.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:41.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:41.358 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:41.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:41.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:41.484 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:41.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:41.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:41.610 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:41.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:41.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:41.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:41.736 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:41.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:41.814 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_1' created 2024-03-24T11:58:41.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:58:41.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:41.901 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:41.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:42.001 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:42.030 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:42.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:42.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:41 smithi064 ceph-mon[30223]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:58:42.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:41 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3237001834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:58:42.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:41 smithi064 ceph-mon[30223]: from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:58:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:41 smithi064 ceph-mon[42243]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:58:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:41 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3237001834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:58:42.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:41 smithi064 ceph-mon[42243]: from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:58:42.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:41 smithi193 ceph-mon[33593]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:58:42.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:41 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3237001834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:58:42.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:41 smithi193 ceph-mon[33593]: from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:58:42.132 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:42.160 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:42.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:42.261 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:42.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:42.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:42.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:42.415 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:42.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:42.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:42.541 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:42.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:42.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:42.667 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:42.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:42.769 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:42.797 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:42.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:42.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:42.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:42.924 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:42.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:42.936 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:42 smithi064 ceph-mon[30223]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:42.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:42 smithi064 ceph-mon[30223]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T11:58:42.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:42 smithi064 ceph-mon[30223]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:58:42.937 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:42 smithi064 ceph-mon[42243]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:42.937 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:42 smithi064 ceph-mon[42243]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T11:58:42.937 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:42 smithi064 ceph-mon[42243]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:58:42.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-24T11:58:43.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:43.082 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:43.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:43.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:42 smithi193 ceph-mon[33593]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:43.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:42 smithi193 ceph-mon[33593]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T11:58:43.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:42 smithi193 ceph-mon[33593]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:58:43.183 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:43.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:43.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:43.314 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:43.342 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:43.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:58:43.422 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:58:43.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-24T11:58:43.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:43.489 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:43.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:43.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:43.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:43.616 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:43.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:43.717 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:43.746 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:43.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:43.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:43.876 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:43.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:43.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:44.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:44.005 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:44.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:44.080 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:43 smithi064 ceph-mon[30223]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:58:44.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:43 smithi064 ceph-mon[42243]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:58:44.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:43 smithi193 ceph-mon[33593]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:58:44.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:44.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:44.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:44.257 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:44.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:44.359 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:44.389 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:44.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:44.456 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:44.489 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:44.499 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:58:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-03-24T11:58:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-03-24T11:58:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-03-24T11:58:44.499 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:58:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:44.518 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:44.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:44.620 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:44.647 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:44.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:44.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:44.777 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:44.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:44.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:58:44.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:44 smithi064 ceph-mon[30223]: pgmap v267: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-24T11:58:44.872 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:44 smithi064 ceph-mon[30223]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:58:44.872 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:44 smithi064 ceph-mon[30223]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:58:44.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:44 smithi064 ceph-mon[42243]: pgmap v267: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-24T11:58:44.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:44 smithi064 ceph-mon[42243]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:58:44.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:44 smithi064 ceph-mon[42243]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:58:44.878 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:44.907 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:44.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:45.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:45.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:45.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:45.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:44 smithi193 ceph-mon[33593]: pgmap v267: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-24T11:58:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:44 smithi193 ceph-mon[33593]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:58:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:44 smithi193 ceph-mon[33593]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:58:45.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:45.164 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:45.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:45.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:45.289 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:45.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:45.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:45.419 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:45.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:45.520 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:45.549 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:45.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:45.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:45.680 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:45.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:45.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T11:58:45.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:45.809 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:45.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:45.910 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:45.939 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:45.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:45.959 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:46.006 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:46.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:46.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:46.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:46.189 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:46.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:46.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:46.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:46.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:46.416 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:46.446 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:46.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:46.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:46.571 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:46.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:46.672 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:46.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:46.701 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:46.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:46.802 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:46.829 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:46.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:46.881 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:46 smithi064 ceph-mon[30223]: pgmap v269: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:58:46.881 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:46 smithi064 ceph-mon[42243]: pgmap v269: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:58:46.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:46.954 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:46.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:47.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:47.080 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:47.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:47.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:46 smithi193 ceph-mon[33593]: pgmap v269: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:58:47.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:58:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:58:47.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:47.206 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:47.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:47.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:47.331 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:47.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:47.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:47.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:47.458 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:47.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:47.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:47.587 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:47.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:47.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:47.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:47.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:47.724 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:58:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:58:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:47.816 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:47.845 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:47.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:47.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:47.970 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:47.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:48.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:48.100 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:48.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:48.201 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:48.229 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:48.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:48.331 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:48.359 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:48.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:48.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:48.485 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:48.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:48.498 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:48.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:48.611 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:48.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:48.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:48.736 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:48.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:48.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:48.866 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:48.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:48.968 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:48.997 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:48.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:48 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:49.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:48 smithi193 ceph-mon[33593]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:49.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:49.122 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:49.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:49.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:49.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:49.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:48 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:48 smithi064 ceph-mon[30223]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:48 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:48 smithi064 ceph-mon[42243]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:49.349 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:49.376 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:49.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:49.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:49.502 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:49.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:49.604 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:49.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:49.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:49.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:49.760 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:49.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:49.861 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:49.889 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:49.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:49.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:50.019 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:50.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:50.121 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:50.148 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:50.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:50.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:50.274 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:50.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:50.362 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T11:58:50.362 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T11:58:50.362 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T11:58:50.362 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T11:58:50.362 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T11:58:50.363 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T11:58:50.363 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T11:58:50.363 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:58:50.363 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T11:58:50.363 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T11:58:50.363 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T11:58:50.363 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T11:58:50.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:50.404 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:50.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:50.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:50.534 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:50.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:50.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:50.663 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:50.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:50.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:50.790 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:50.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:50.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:50.920 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:50.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:50.939 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:50 smithi064 ceph-mon[30223]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-03-24T11:58:50.939 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:50 smithi064 ceph-mon[42243]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-03-24T11:58:50.973 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:58:50.991 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:58:51.010 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:58:51.022 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:51.029 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:58:51.049 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:58:51.050 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:51.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:51.070 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:58:51.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:50 smithi193 ceph-mon[33593]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-03-24T11:58:51.091 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:58:51.115 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:58:51.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:58:51.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:58:51.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:58:51.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:58:51.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:58:51.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:58:51.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:58:51.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:58:51.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:51.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:51.176 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:51.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:51.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:51.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:51.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:51.402 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:51.430 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:51.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:51.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:51.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:51.559 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:51.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:51.660 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:51.687 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:51.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:51.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:51.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:58:51.858 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:58:51.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-24T11:58:51.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:51.944 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:51.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:52.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:52.074 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:52.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:52.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:52.200 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:52.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:52.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:51 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2350508301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:52.262 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:51 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2350508301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:52.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:52.330 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:52.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:52.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:51 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2350508301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:58:52.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:52.460 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:52.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:52.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:52.587 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:52.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:52.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:52.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:52.713 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:52.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:52.814 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:52.842 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:52.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:52.912 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T11:58:52.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:52.972 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:52.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:53.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:53.098 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:53.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:53.104 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:52 smithi064 ceph-mon[30223]: pgmap v272: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-03-24T11:58:53.104 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:52 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:58:53.104 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:52 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:58:53.104 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/372539758' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:58:53.105 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:52 smithi064 ceph-mon[42243]: pgmap v272: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-03-24T11:58:53.105 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:52 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:58:53.105 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:52 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:58:53.105 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:52 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/372539758' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:58:53.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:53.225 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:53.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:53.327 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:52 smithi193 ceph-mon[33593]: pgmap v272: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-03-24T11:58:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:52 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:58:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:52 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:58:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:52 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/372539758' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:58:53.355 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:53.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:53.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:53.485 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:53.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:53.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:53.610 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:53.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:53.712 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:53.740 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:53.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:53.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:53.870 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:53.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:53.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:53.996 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:53.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:54.044 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:54.044 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/372539758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:58:54.044 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[30223]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:58:54.044 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[30223]: 4.1d scrub starts 2024-03-24T11:58:54.044 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[30223]: 4.1d scrub ok 2024-03-24T11:58:54.044 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:58:54.044 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:58:54.045 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:54.045 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/372539758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:58:54.045 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[42243]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:58:54.045 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[42243]: 4.1d scrub starts 2024-03-24T11:58:54.045 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[42243]: 4.1d scrub ok 2024-03-24T11:58:54.045 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:58:54.045 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:53 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:58:54.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:54.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:54.227 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:54.256 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:54.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:53 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:58:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/372539758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:58:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:53 smithi193 ceph-mon[33593]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:58:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:53 smithi193 ceph-mon[33593]: 4.1d scrub starts 2024-03-24T11:58:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:53 smithi193 ceph-mon[33593]: 4.1d scrub ok 2024-03-24T11:58:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:53 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:58:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:53 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:58:54.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:54.381 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:54.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:54.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:54.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:54.506 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:54.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:54.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:58:54.608 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:54.636 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:54.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:54.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:54.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:54.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:54.894 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:54.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:54.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:55.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T11:58:55.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T11:58:55.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[30223]: pgmap v274: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:58:55.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[30223]: 4.1b scrub starts 2024-03-24T11:58:55.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[30223]: 4.1b scrub ok 2024-03-24T11:58:55.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[30223]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:58:55.021 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T11:58:55.021 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T11:58:55.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[42243]: pgmap v274: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:58:55.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[42243]: 4.1b scrub starts 2024-03-24T11:58:55.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[42243]: 4.1b scrub ok 2024-03-24T11:58:55.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:54 smithi064 ceph-mon[42243]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:58:55.025 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:55.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:55.127 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:55.157 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:55.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:55.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:55.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:54 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T11:58:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:54 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T11:58:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:54 smithi193 ceph-mon[33593]: pgmap v274: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:58:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:54 smithi193 ceph-mon[33593]: 4.1b scrub starts 2024-03-24T11:58:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:54 smithi193 ceph-mon[33593]: 4.1b scrub ok 2024-03-24T11:58:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:54 smithi193 ceph-mon[33593]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:58:55.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:55.418 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:55.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:55.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:55.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:55.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:55.674 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:55.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:55.775 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:55.804 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:55.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:55.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:55.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T11:58:55.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T11:58:55.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: 6.10 scrub starts 2024-03-24T11:58:55.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: 6.10 scrub ok 2024-03-24T11:58:55.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T11:58:55.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T11:58:55.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: 6.10 scrub starts 2024-03-24T11:58:55.928 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-24T11:58:55.933 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:55.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:56.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:56.058 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:56.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:56.059 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:58:56.059 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:58:56.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:58:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:56.184 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:56.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:58:56.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: 4.c scrub starts 2024-03-24T11:58:56.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: 4.c scrub ok 2024-03-24T11:58:56.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T11:58:56.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T11:58:56.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1167650354' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:58:56.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:56.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:56.269 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: 6.10 scrub ok 2024-03-24T11:58:56.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: 4.c scrub starts 2024-03-24T11:58:56.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: 4.c scrub ok 2024-03-24T11:58:56.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T11:58:56.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T11:58:56.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1167650354' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:58:56.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:56.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:56.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:56.308 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:56.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:56.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: 6.10 scrub starts 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: 6.10 scrub ok 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: 4.c scrub starts 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: 4.c scrub ok 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1167650354' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:58:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:56.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:56.433 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:56.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:56.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:56.558 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:56.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:56.660 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:56.689 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:56.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:56.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:56.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:58:56.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:56.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:56.920 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:56.946 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:56.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:56.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:57.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:57.073 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:57.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:57.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:57.202 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:57.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[30223]: pgmap v276: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:58:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T11:58:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T11:58:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T11:58:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T11:58:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1167650354' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:58:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[30223]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:58:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[42243]: pgmap v276: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:58:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T11:58:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T11:58:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T11:58:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T11:58:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1167650354' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:58:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:56 smithi064 ceph-mon[42243]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:58:57.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:57.333 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:57.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:57.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:58:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:58:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:56 smithi193 ceph-mon[33593]: pgmap v276: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:58:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:56 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T11:58:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:56 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T11:58:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:56 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T11:58:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:56 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T11:58:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:56 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1167650354' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:58:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:56 smithi193 ceph-mon[33593]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:58:57.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:57.452 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:57.459 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:57.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:57.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T11:58:57.561 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:57.590 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:57.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:57.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:57.716 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:57.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:57.777 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:58:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:57.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:58:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:57.843 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:57.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:58:57.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:57.972 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:57.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:58.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T11:58:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T11:58:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T11:58:58.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T11:58:58.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T11:58:58.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T11:58:58.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T11:58:58.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:57 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T11:58:58.074 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:58.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:58.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:58.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:58:58.230 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:58.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:58:58.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:57 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T11:58:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:57 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T11:58:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:57 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T11:58:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:57 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T11:58:58.356 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:58.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:58.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:58:58.486 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:58.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:58.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:58:58.588 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:58.616 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:58.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:58.717 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:58.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:58.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:58.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:58.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:58.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:58.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:58:59.001 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:59.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:58:59.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:59.129 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:59.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:59.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:59.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 3.2 scrub starts 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 3.2 scrub ok 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: pgmap v278: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[42243]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:58:59.187 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 3.2 scrub starts 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 3.2 scrub ok 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: pgmap v278: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T11:58:59.188 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:58 smithi064 ceph-mon[30223]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:58:59.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:58:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:59.255 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:59.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:58:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 3.2 scrub starts 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 3.2 scrub ok 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: pgmap v278: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T11:58:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:58 smithi193 ceph-mon[33593]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:58:59.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:59.380 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:59.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:59.452 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T11:58:59.452 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T11:58:59.452 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T11:58:59.452 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T11:58:59.452 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T11:58:59.452 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T11:58:59.452 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T11:58:59.452 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:58:59.453 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T11:58:59.453 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T11:58:59.453 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T11:58:59.453 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T11:58:59.482 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:59.509 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:59.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:59.611 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:59.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:59.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:59.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:58:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:58:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:59.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:58:59.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:58:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:58:59.893 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:58:59.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:58:59.978 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:58:59.994 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:00.020 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:00.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:00.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:59 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T11:59:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:59 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T11:59:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:59 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T11:59:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:58:59 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T11:59:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:59 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T11:59:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:59 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T11:59:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:59 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T11:59:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:58:59 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T11:59:00.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:00.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:00.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:00.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:00.275 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:00.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:59 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T11:59:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:59 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T11:59:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:59 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T11:59:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:58:59 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T11:59:00.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:00.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:00.405 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:00.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:00.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:00.534 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:00.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:00.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:59:00.659 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:59:00.663 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:00.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:00.765 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:00.766 DEBUG:teuthology.orchestra.run.smithi064:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-24T11:59:00.769 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T11:59:00.769 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T11:59:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:00.793 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:00.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:00.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:hints = 1 2024-03-24T11:59:00.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T11:59:00.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object prefix: benchmark_data_smithi064_120708 2024-03-24T11:59:00.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:59:00.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 0 0 0 0 0 0 - 0 2024-03-24T11:59:00.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:00.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:00.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:01.051 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:01.084 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:01.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:01.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:01.231 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:01.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:01.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[30223]: pgmap v279: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[42243]: pgmap v279: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T11:59:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:00 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T11:59:01.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:00 smithi193 ceph-mon[33593]: pgmap v279: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-03-24T11:59:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:00 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T11:59:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:00 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T11:59:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:00 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T11:59:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:00 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T11:59:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:01.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:01.481 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:01.515 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:01.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:01.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:01.617 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:01.652 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:01.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:01.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:01.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:01.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 1 15 3561 3546 221.613 221.625 0.0107424 0.00449754 2024-03-24T11:59:01.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:01.933 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:01.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:02.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:02.069 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:02.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:02.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:02.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:02.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:01 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T11:59:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:01 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T11:59:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:01 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T11:59:02.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:01 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T11:59:02.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:01 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T11:59:02.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:01 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T11:59:02.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:01 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T11:59:02.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:01 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T11:59:02.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:02.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:01 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T11:59:02.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:01 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T11:59:02.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:01 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T11:59:02.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:01 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T11:59:02.358 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:02.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:02.461 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:02.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:02.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:02.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:02.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:02.737 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:02.773 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:02.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:02.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 2 16 6779 6763 211.329 201.062 0.00323728 0.00472163 2024-03-24T11:59:02.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:02.906 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:02.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:03.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:03.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:03.053 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:59:03.053 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:59:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:59:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:59:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:59:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:59:03.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:03.193 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:03.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:03.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[30223]: pgmap v280: 148 pgs: 148 active+clean; 36 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 4.5 MiB/s wr, 72 op/s 2024-03-24T11:59:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T11:59:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T11:59:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T11:59:03.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T11:59:03.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2744027051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[42243]: pgmap v280: 148 pgs: 148 active+clean; 36 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 4.5 MiB/s wr, 72 op/s 2024-03-24T11:59:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T11:59:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T11:59:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T11:59:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T11:59:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:02 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2744027051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:03.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:02 smithi193 ceph-mon[33593]: pgmap v280: 148 pgs: 148 active+clean; 36 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 4.5 MiB/s wr, 72 op/s 2024-03-24T11:59:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:02 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T11:59:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:02 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T11:59:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:02 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T11:59:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:02 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T11:59:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:02 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2744027051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:03.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:03.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:03.470 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:03.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:03.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:03.607 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:03.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:03.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:03.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:03.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:03.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 3 16 9873 9857 205.339 193.375 0.00304669 0.00485828 2024-03-24T11:59:03.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:03.881 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:03.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:03.983 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:04.018 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:04.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:04.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T11:59:04.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T11:59:04.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T11:59:04.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T11:59:04.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[42243]: 6.b scrub starts 2024-03-24T11:59:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T11:59:04.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T11:59:04.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T11:59:04.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T11:59:04.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:03 smithi064 ceph-mon[30223]: 6.b scrub starts 2024-03-24T11:59:04.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:04.160 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:04.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:04.261 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:04.294 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:04.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:03 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T11:59:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:03 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T11:59:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:03 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T11:59:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:03 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T11:59:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:03 smithi193 ceph-mon[33593]: 6.b scrub starts 2024-03-24T11:59:04.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:04.429 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:04.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:04.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:04.565 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:04.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:04.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:04.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:04.708 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:04.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:04.809 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:04.816 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 4 16 12503 12487 195.093 164.375 0.00368186 0.00511877 2024-03-24T11:59:04.841 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:04.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:04.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:04.978 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:04.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:05.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:05.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:05.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:05.217 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:05.247 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:04 smithi064 ceph-mon[30223]: 6.b scrub ok 2024-03-24T11:59:05.247 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:04 smithi064 ceph-mon[30223]: pgmap v281: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 145 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 248 op/s 2024-03-24T11:59:05.247 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:04 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T11:59:05.247 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:04 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T11:59:05.248 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:04 smithi064 ceph-mon[42243]: 6.b scrub ok 2024-03-24T11:59:05.248 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:04 smithi064 ceph-mon[42243]: pgmap v281: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 145 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 248 op/s 2024-03-24T11:59:05.248 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:04 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T11:59:05.248 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:04 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T11:59:05.250 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:05.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:04 smithi193 ceph-mon[33593]: 6.b scrub ok 2024-03-24T11:59:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:04 smithi193 ceph-mon[33593]: pgmap v281: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 145 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 248 op/s 2024-03-24T11:59:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:04 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T11:59:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:04 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T11:59:05.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:05.395 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:05.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:05.497 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:05.533 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:05.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:05.634 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:05.660 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:59:05.660 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:59:05.660 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-24T11:59:05.660 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.253781 2024-03-24T11:59:05.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 1 0.2537812422099407 2024-03-24T11:59:05.666 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:05.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:14604 8.0 8:07bef1d1:::benchmark_data_smithi064_120708_object14603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:05.605058+0000", 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0058622209999999999, 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005900359, 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:59:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 14604 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605058+0000", 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605058+0000", 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605060+0000", 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9379999999999999e-06 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605083+0000", 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3255999999999999e-05 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605095+0000", 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1941e-05 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605103+0000", 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4719999999999995e-06 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605162+0000", 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9157999999999998e-05 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605302+0000", 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013979800000000001 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605457+0000", 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000154594 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:14607 8.0 8:07443341:::benchmark_data_smithi064_120708_object14606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:05.605352+0000", 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0055675880000000001, 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057031900000000003, 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:59:05.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 14607 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605352+0000", 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605352+0000", 2024-03-24T11:59:05.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605354+0000", 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4899999999999999e-06 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605376+0000", 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2493999999999998e-05 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605390+0000", 2024-03-24T11:59:05.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3463e-05 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605399+0000", 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0080000000000008e-06 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606201+0000", 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080191400000000003 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.607531+0000", 2024-03-24T11:59:05.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013303869999999999 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.608016+0000", 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048463100000000002 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:14608 8.b 8:d479b2f0:::benchmark_data_smithi064_120708_object14607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:05.605787+0000", 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0051323890000000002, 2024-03-24T11:59:05.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053863890000000001, 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 14608 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605787+0000", 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605787+0000", 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605789+0000", 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.291e-06 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605811+0000", 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1872e-05 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605826+0000", 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4980999999999999e-05 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605834+0000", 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4489999999999999e-06 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.605943+0000", 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109292 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:59:05.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606084+0000", 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014044099999999999 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606313+0000", 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022895500000000001 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.607256+0000", 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094311000000000004 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:59:05.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:14610 8.f 8:fa2a4c0c:::benchmark_data_smithi064_120708_object14609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:05.606157+0000", 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0047624269999999996, 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051114869999999996, 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 14610 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606157+0000", 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606157+0000", 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606159+0000", 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2729999999999999e-06 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606181+0000", 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2544999999999998e-05 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606192+0000", 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0412e-05 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606200+0000", 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.943e-06 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606540+0000", 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034066899999999999 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606722+0000", 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182103 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606846+0000", 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000123738 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35033.0:14604 8.0 e175/170)", 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:05.606617+0000", 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.004302864, 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047296839999999996, 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606617+0000", 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606617+0000", 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606618+0000", 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.423e-06 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606627+0000", 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1349999999999992e-06 2024-03-24T11:59:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606635+0000", 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2919999999999996e-06 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.606642+0000", 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1520000000000003e-06 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.693 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:14613 8.f 8:fcb8c975:::benchmark_data_smithi064_120708_object14612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:05.607931+0000", 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0029889949999999999, 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003476186, 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 14613 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.607931+0000", 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.607931+0000", 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.607933+0000", 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.875e-06 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.607969+0000", 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6563999999999998e-05 2024-03-24T11:59:05.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.607982+0000", 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2948999999999999e-05 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.607991+0000", 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0710000000000001e-06 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.608176+0000", 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000184766 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609808+0000", 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016316359999999999 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.610059+0000", 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025084599999999999 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35033.0:14607 8.0 e175/170)", 2024-03-24T11:59:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:05.609132+0000", 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0017880960000000001, 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028109849999999998, 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609132+0000", 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609132+0000", 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609133+0000", 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.59e-06 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609138+0000", 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8860000000000003e-06 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609149+0000", 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0977e-05 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609157+0000", 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9140000000000008e-06 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35033.0:14610 8.f e175/170)", 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:05.609421+0000", 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0014991710000000001, 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025713020000000001, 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T11:59:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609421+0000", 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609421+0000", 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609422+0000", 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860000000000001e-06 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609431+0000", 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8869999999999998e-06 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609443+0000", 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2062000000000001e-05 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:05.609451+0000", 2024-03-24T11:59:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2779999999999993e-06 2024-03-24T11:59:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T11:59:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T11:59:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T11:59:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 8 2024-03-24T11:59:05.705 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T11:59:05.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:05.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:59:05.796 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:05.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:05.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:05.817 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 5 16 15065 15049 188.071 160.125 0.00832136 0.00530777 2024-03-24T11:59:05.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:05.962 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:05.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:06.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:06.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T11:59:06.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T11:59:06.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[30223]: 2.18 scrub starts 2024-03-24T11:59:06.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[30223]: 2.18 scrub ok 2024-03-24T11:59:06.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T11:59:06.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T11:59:06.098 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T11:59:06.098 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T11:59:06.098 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[42243]: 2.18 scrub starts 2024-03-24T11:59:06.098 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[42243]: 2.18 scrub ok 2024-03-24T11:59:06.098 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T11:59:06.098 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:05 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T11:59:06.110 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:06.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:06.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:06.258 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:06.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:06.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:05 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T11:59:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:05 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T11:59:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:05 smithi193 ceph-mon[33593]: 2.18 scrub starts 2024-03-24T11:59:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:05 smithi193 ceph-mon[33593]: 2.18 scrub ok 2024-03-24T11:59:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:05 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T11:59:06.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:05 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T11:59:06.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:06.398 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:06.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:06.500 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:06.536 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:06.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:06.637 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:06.673 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:06.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:06.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:06.811 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:06.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:06.817 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 6 15 17289 17274 179.902 139.062 0.0174677 0.00554668 2024-03-24T11:59:06.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:06.954 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:06.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:06.998 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T11:59:07.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:07.086 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:07.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:07.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:07.231 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:07.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: pgmap v282: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 145 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 14 MiB/s wr, 231 op/s 2024-03-24T11:59:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 6.1d scrub starts 2024-03-24T11:59:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 6.1d scrub ok 2024-03-24T11:59:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 4.1 scrub starts 2024-03-24T11:59:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 4.1 scrub ok 2024-03-24T11:59:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 6.13 scrub starts 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: 6.13 scrub ok 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2773505574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[30223]: from='client.25063 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: pgmap v282: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 145 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 14 MiB/s wr, 231 op/s 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 6.1d scrub starts 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 6.1d scrub ok 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 4.1 scrub starts 2024-03-24T11:59:07.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 4.1 scrub ok 2024-03-24T11:59:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T11:59:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T11:59:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T11:59:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T11:59:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 6.13 scrub starts 2024-03-24T11:59:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: 6.13 scrub ok 2024-03-24T11:59:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2773505574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:07.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-mon[42243]: from='client.25063 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:07.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:07.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:59:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: pgmap v282: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 145 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 14 MiB/s wr, 231 op/s 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 6.1d scrub starts 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 6.1d scrub ok 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 4.1 scrub starts 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 4.1 scrub ok 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 6.13 scrub starts 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: 6.13 scrub ok 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2773505574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:07 smithi193 ceph-mon[33593]: from='client.25063 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:07.372 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:07.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:07.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:07.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:07.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T11:59:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:07.534 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:07.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:07.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:07.668 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:07.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:07.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:07.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:59:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:59:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:07.817 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:07.817 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 7 16 19694 19678 175.665 150.25 0.00282839 0.00568403 2024-03-24T11:59:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:07.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:07.920 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:07.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:08.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:08.098 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:08.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:08.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: 4.d scrub starts 2024-03-24T11:59:08.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: 4.d scrub ok 2024-03-24T11:59:08.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: 4.3 scrub starts 2024-03-24T11:59:08.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: 4.3 scrub ok 2024-03-24T11:59:08.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:59:08.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:59:08.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:59:08.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T11:59:08.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: from='client.25063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:59:08.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:59:08.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: 4.11 scrub starts 2024-03-24T11:59:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: 4.11 scrub ok 2024-03-24T11:59:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T11:59:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T11:59:08.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: 4.d scrub starts 2024-03-24T11:59:08.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: 4.d scrub ok 2024-03-24T11:59:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: 4.3 scrub starts 2024-03-24T11:59:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: 4.3 scrub ok 2024-03-24T11:59:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:59:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:59:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:59:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T11:59:08.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: from='client.25063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:59:08.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:59:08.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: 4.11 scrub starts 2024-03-24T11:59:08.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: 4.11 scrub ok 2024-03-24T11:59:08.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T11:59:08.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:08 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T11:59:08.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:08.231 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:08.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:08.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: 4.d scrub starts 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: 4.d scrub ok 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: 4.3 scrub starts 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: 4.3 scrub ok 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: from='client.25063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:59:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:59:08.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: 4.11 scrub starts 2024-03-24T11:59:08.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: 4.11 scrub ok 2024-03-24T11:59:08.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T11:59:08.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:08 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T11:59:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:08.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:08.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:08.478 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:08.512 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:08.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:08.614 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:08.645 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:08.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:08.747 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:08.783 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:08.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:08.817 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 8 16 22084 22068 172.377 149.375 0.00881894 0.00579511 2024-03-24T11:59:08.885 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:08.922 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:08.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:09.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:09.034 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:09.036 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:09.078 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:09.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:09.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:09.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:09.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[30223]: pgmap v284: 148 pgs: 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[30223]: 6.d deep-scrub starts 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[30223]: 6.d deep-scrub ok 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[42243]: pgmap v284: 148 pgs: 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-24T11:59:09.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T11:59:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T11:59:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[42243]: 6.d deep-scrub starts 2024-03-24T11:59:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:09 smithi064 ceph-mon[42243]: 6.d deep-scrub ok 2024-03-24T11:59:09.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:09.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:09.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:09 smithi193 ceph-mon[33593]: pgmap v284: 148 pgs: 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-24T11:59:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:09 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T11:59:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:09 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T11:59:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:09 smithi193 ceph-mon[33593]: 6.d deep-scrub starts 2024-03-24T11:59:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:09 smithi193 ceph-mon[33593]: 6.d deep-scrub ok 2024-03-24T11:59:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:09.363 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:09.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:09.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:09.509 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:09.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:09.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:09.650 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:09.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:09.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:09.792 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:09.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:09.817 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 9 16 24497 24481 169.98 150.812 0.00218494 0.00587736 2024-03-24T11:59:09.894 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:09.930 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:09.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:10.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:10.071 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:10.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:10.173 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:10.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:59:10.207 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:10.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[30223]: 5.c scrub starts 2024-03-24T11:59:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[30223]: 5.c scrub ok 2024-03-24T11:59:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[30223]: 4.1e deep-scrub starts 2024-03-24T11:59:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[30223]: 4.1e deep-scrub ok 2024-03-24T11:59:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[30223]: 5.9 deep-scrub starts 2024-03-24T11:59:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[30223]: 5.9 deep-scrub ok 2024-03-24T11:59:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[30223]: 6.11 scrub starts 2024-03-24T11:59:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[30223]: 6.11 scrub ok 2024-03-24T11:59:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[42243]: 5.c scrub starts 2024-03-24T11:59:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[42243]: 5.c scrub ok 2024-03-24T11:59:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[42243]: 4.1e deep-scrub starts 2024-03-24T11:59:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[42243]: 4.1e deep-scrub ok 2024-03-24T11:59:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[42243]: 5.9 deep-scrub starts 2024-03-24T11:59:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[42243]: 5.9 deep-scrub ok 2024-03-24T11:59:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[42243]: 6.11 scrub starts 2024-03-24T11:59:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:10 smithi064 ceph-mon[42243]: 6.11 scrub ok 2024-03-24T11:59:10.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:10 smithi193 ceph-mon[33593]: 5.c scrub starts 2024-03-24T11:59:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:10 smithi193 ceph-mon[33593]: 5.c scrub ok 2024-03-24T11:59:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:10 smithi193 ceph-mon[33593]: 4.1e deep-scrub starts 2024-03-24T11:59:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:10 smithi193 ceph-mon[33593]: 4.1e deep-scrub ok 2024-03-24T11:59:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:10 smithi193 ceph-mon[33593]: 5.9 deep-scrub starts 2024-03-24T11:59:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:10 smithi193 ceph-mon[33593]: 5.9 deep-scrub ok 2024-03-24T11:59:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:10 smithi193 ceph-mon[33593]: 6.11 scrub starts 2024-03-24T11:59:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:10 smithi193 ceph-mon[33593]: 6.11 scrub ok 2024-03-24T11:59:10.337 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:10.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:10.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:10.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:10.582 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:10.620 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:10.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:10.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:10.752 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:10.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:10.817 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 10 16 26684 26668 166.649 136.688 0.00218305 0.00599459 2024-03-24T11:59:10.854 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:10.885 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:10.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:10.987 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:11.020 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:11.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:11.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:11.160 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:11.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:11.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:11.294 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:11.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: pgmap v285: 148 pgs: 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-24T11:59:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T11:59:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T11:59:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T11:59:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T11:59:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: 2.2 deep-scrub starts 2024-03-24T11:59:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: 2.2 deep-scrub ok 2024-03-24T11:59:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: 2.19 scrub starts 2024-03-24T11:59:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: 2.19 scrub ok 2024-03-24T11:59:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:11.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: pgmap v285: 148 pgs: 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-24T11:59:11.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T11:59:11.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T11:59:11.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T11:59:11.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T11:59:11.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: 2.2 deep-scrub starts 2024-03-24T11:59:11.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: 2.2 deep-scrub ok 2024-03-24T11:59:11.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: 2.19 scrub starts 2024-03-24T11:59:11.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: 2.19 scrub ok 2024-03-24T11:59:11.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:11.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: pgmap v285: 148 pgs: 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: 2.2 deep-scrub starts 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: 2.2 deep-scrub ok 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: 2.19 scrub starts 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: 2.19 scrub ok 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:11.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:11.434 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:11.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:11.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:11.576 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:11.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:11.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:11.716 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:11.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:11.818 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:11.818 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 11 16 28891 28875 164.033 137.938 0.00379973 0.00609007 2024-03-24T11:59:11.851 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:11.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:11.953 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:11.989 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:11.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:12.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:12.130 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:12.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:12.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:12.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:12.269 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:12.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:12 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T11:59:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:12 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T11:59:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:12 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T11:59:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:12 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T11:59:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:12 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T11:59:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:12 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T11:59:12.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:12.395 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:12.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:12.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:12.534 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:12.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:12.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:12.668 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:12.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:12.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:12.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:12.818 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 12 16 30747 30731 160.03 116 0.00775351 0.00624233 2024-03-24T11:59:12.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:12.945 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:12.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:13.037 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:13 smithi064 ceph-mon[42243]: pgmap v286: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-24T11:59:13.037 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:13 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T11:59:13.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:13.091 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:13.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:13.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:13.216 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:13.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:13.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:13 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T11:59:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:13 smithi064 ceph-mon[30223]: pgmap v286: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-24T11:59:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:13 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T11:59:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:13 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T11:59:13.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:13 smithi193 ceph-mon[33593]: pgmap v286: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-24T11:59:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:13 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T11:59:13.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:13 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T11:59:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:13.356 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:13.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:13.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:13.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:13.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:13.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:13.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:13.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:13.769 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:13.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:13.818 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 13 16 32268 32252 155.032 95.0625 0.00402034 0.00644453 2024-03-24T11:59:13.870 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:13.902 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:13.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:14.003 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:14.034 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:14.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:14.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:14.175 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:14.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:14.276 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:14 smithi064 ceph-mon[30223]: 6.17 deep-scrub starts 2024-03-24T11:59:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:14 smithi064 ceph-mon[30223]: 6.17 deep-scrub ok 2024-03-24T11:59:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:14 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T11:59:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:14 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T11:59:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:14 smithi064 ceph-mon[42243]: 6.17 deep-scrub starts 2024-03-24T11:59:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:14 smithi064 ceph-mon[42243]: 6.17 deep-scrub ok 2024-03-24T11:59:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:14 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T11:59:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:14 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T11:59:14.311 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:14.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:14 smithi193 ceph-mon[33593]: 6.17 deep-scrub starts 2024-03-24T11:59:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:14 smithi193 ceph-mon[33593]: 6.17 deep-scrub ok 2024-03-24T11:59:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:14 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T11:59:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:14 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T11:59:14.412 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:14.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:14.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:14.588 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:14.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:14.689 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:14.722 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:14.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:14.818 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 14 16 34159 34143 152.4 118.188 0.00151517 0.00653381 2024-03-24T11:59:14.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:14.848 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:14.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:14.949 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:14.986 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:14.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:15.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:15.126 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:15.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:15.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:15.265 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:15.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:15.300 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[30223]: pgmap v287: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-03-24T11:59:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T11:59:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T11:59:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[30223]: 5.4 deep-scrub starts 2024-03-24T11:59:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[30223]: 5.4 deep-scrub ok 2024-03-24T11:59:15.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[42243]: pgmap v287: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-03-24T11:59:15.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T11:59:15.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T11:59:15.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[42243]: 5.4 deep-scrub starts 2024-03-24T11:59:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:15 smithi064 ceph-mon[42243]: 5.4 deep-scrub ok 2024-03-24T11:59:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:15 smithi193 ceph-mon[33593]: pgmap v287: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-03-24T11:59:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:15 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T11:59:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:15 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T11:59:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:15 smithi193 ceph-mon[33593]: 5.4 deep-scrub starts 2024-03-24T11:59:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:15 smithi193 ceph-mon[33593]: 5.4 deep-scrub ok 2024-03-24T11:59:15.367 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:15.391 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:15.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:15.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:15.533 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:15.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:15.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:15.674 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:15.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:15.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:15.815 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:15.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:15.818 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 15 15 35751 35736 148.876 99.5625 0.0189812 0.00671021 2024-03-24T11:59:15.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:15.942 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:15.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:16.043 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:16.076 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:16.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:16.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:16.206 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:16.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: 4.18 deep-scrub starts 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: 4.18 deep-scrub ok 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: 5.17 scrub starts 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: 5.17 scrub ok 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[30223]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:59:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T11:59:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T11:59:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:59:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:59:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: 4.18 deep-scrub starts 2024-03-24T11:59:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: 4.18 deep-scrub ok 2024-03-24T11:59:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: 5.17 scrub starts 2024-03-24T11:59:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: 5.17 scrub ok 2024-03-24T11:59:16.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:16 smithi064 ceph-mon[42243]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:59:16.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: 4.18 deep-scrub starts 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: 4.18 deep-scrub ok 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: 5.17 scrub starts 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: 5.17 scrub ok 2024-03-24T11:59:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:16 smithi193 ceph-mon[33593]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:59:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:16.351 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:16.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:16.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:16.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:16.496 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:16.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:16.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:16.643 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:16.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:16.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:16.794 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:16.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:16.818 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 16 16 37041 37025 144.606 80.5625 0.0125657 0.00690867 2024-03-24T11:59:16.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:16.937 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:16.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:17.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:17.080 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:17.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:17.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:17.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:17.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:17.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:17.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:59:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:59:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:17 smithi193 ceph-mon[33593]: pgmap v289: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 620 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T11:59:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:17 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T11:59:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:17 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T11:59:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:17.364 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:17.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:17.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:17 smithi064 ceph-mon[30223]: pgmap v289: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 620 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T11:59:17.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:17 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T11:59:17.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:17 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T11:59:17.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:17 smithi064 ceph-mon[42243]: pgmap v289: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 620 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T11:59:17.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:17 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T11:59:17.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:17 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T11:59:17.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:17.498 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:17.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:17.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:17.641 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:17.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:17.692 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:59:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:59:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:17.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:17.783 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:17.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:17.819 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 17 16 38821 38805 142.644 111.25 0.00434191 0.00700451 2024-03-24T11:59:17.884 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:17.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:17.919 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:17.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:18.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:18.058 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:18.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.1 primary-affinity to 0.253781 (8166312) 2024-03-24T11:59:18.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:18.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:18.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T11:59:18.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T11:59:18.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T11:59:18.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T11:59:18.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T11:59:18.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T11:59:18.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: 5.1a deep-scrub starts 2024-03-24T11:59:18.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: 5.1a deep-scrub ok 2024-03-24T11:59:18.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:18.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1378069066' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.2537812422099407}]: dispatch 2024-03-24T11:59:18.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[30223]: pgmap v290: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 126 MiB/s wr, 2.01k op/s 2024-03-24T11:59:18.265 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T11:59:18.265 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: 5.1a deep-scrub starts 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: 5.1a deep-scrub ok 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1378069066' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.2537812422099407}]: dispatch 2024-03-24T11:59:18.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:18 smithi064 ceph-mon[42243]: pgmap v290: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 126 MiB/s wr, 2.01k op/s 2024-03-24T11:59:18.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:18.296 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:59:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:18.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:18.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:18.317 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:59:18.337 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:59:18.357 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:59:18.376 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:59:18.396 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:59:18.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:18.416 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:59:18.436 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:59:18.437 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:59:18.437 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:59:18.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:59:18.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:59:18.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:59:18.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:59:18.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:59:18.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:59:18.438 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:18.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:18.540 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:18.565 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:18.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:18.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T11:59:18.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: 5.1a deep-scrub starts 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: 5.1a deep-scrub ok 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1378069066' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.2537812422099407}]: dispatch 2024-03-24T11:59:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:18 smithi193 ceph-mon[33593]: pgmap v290: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 126 MiB/s wr, 2.01k op/s 2024-03-24T11:59:18.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:18.692 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:18.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:18.793 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:18.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:18.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:18.819 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 18 16 39397 39381 136.719 36 0.00272987 0.00702348 2024-03-24T11:59:18.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:18.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:19.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:19.073 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:19.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:19.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:19.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:19.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T11:59:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T11:59:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:59:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:59:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1378069066' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.2537812422099407}]': finished 2024-03-24T11:59:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[30223]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:59:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3908477396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[30223]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:59:19.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T11:59:19.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T11:59:19.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:59:19.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:59:19.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1378069066' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.2537812422099407}]': finished 2024-03-24T11:59:19.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[42243]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:59:19.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3908477396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:19.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:19 smithi064 ceph-mon[42243]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:59:19.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:19.358 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:19.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:19.460 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:19.497 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:19.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:19 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T11:59:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:19 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T11:59:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:19 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:59:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:19 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:59:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:19 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1378069066' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.2537812422099407}]': finished 2024-03-24T11:59:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:19 smithi193 ceph-mon[33593]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:59:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:19 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3908477396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:19 smithi193 ceph-mon[33593]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:59:19.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:19.637 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:19.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:19.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:19.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:19.779 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:19.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:19.819 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 19 16 40780 40764 134.072 86.4375 0.00927866 0.00745257 2024-03-24T11:59:19.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:19.916 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:19.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:20.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:20.052 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:20.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:20.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:20.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:20.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:20.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 6.0 deep-scrub starts 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 6.0 deep-scrub ok 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 4.e deep-scrub starts 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 4.e deep-scrub ok 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[30223]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:59:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 6.0 deep-scrub starts 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 6.0 deep-scrub ok 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 4.e deep-scrub starts 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 4.e deep-scrub ok 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T11:59:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:20 smithi064 ceph-mon[42243]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:59:20.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:20.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:20.331 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:20.424 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:59:20.424 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:59:20.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:20.472 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:20.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 6.0 deep-scrub starts 2024-03-24T11:59:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 6.0 deep-scrub ok 2024-03-24T11:59:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 4.e deep-scrub starts 2024-03-24T11:59:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 4.e deep-scrub ok 2024-03-24T11:59:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T11:59:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T11:59:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T11:59:20.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T11:59:20.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T11:59:20.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T11:59:20.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:20 smithi193 ceph-mon[33593]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:59:20.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:20.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:20.619 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:20.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:20.721 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:20.757 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:20.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:20.819 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T11:59:20.820338+0000 min lat: 0.00128588 max lat: 1.02708 avg lat: 0.00750222 2024-03-24T11:59:20.819 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:59:20.819 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 20 16 42642 42626 133.187 116.375 0.00664494 0.00750222 2024-03-24T11:59:20.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:20.894 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:20.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:20.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:21.036 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:21.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:21.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:21.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:21.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:21.311 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:21.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:21.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T11:59:21.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:21.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:21.510 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:21 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T11:59:21.511 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:21 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T11:59:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:21 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T11:59:21.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:21.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:21.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:21.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:21.785 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:21.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:21.820 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 21 16 44454 44438 132.229 113.25 0.00668407 0.0075572 2024-03-24T11:59:21.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:21.935 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:21.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:22.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:22.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:59:22.080 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:22.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:22.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:22.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:22.215 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:22.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:22.317 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:22.349 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:22.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:22.372 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T11:59:22.372 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T11:59:22.372 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[30223]: 2.1a deep-scrub starts 2024-03-24T11:59:22.372 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[30223]: 2.1a deep-scrub ok 2024-03-24T11:59:22.372 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[30223]: 6.1f scrub ok 2024-03-24T11:59:22.372 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[30223]: pgmap v294: 148 pgs: 148 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 662 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-24T11:59:22.372 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:59:22.372 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:59:22.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T11:59:22.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T11:59:22.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[42243]: 2.1a deep-scrub starts 2024-03-24T11:59:22.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[42243]: 2.1a deep-scrub ok 2024-03-24T11:59:22.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[42243]: 6.1f scrub ok 2024-03-24T11:59:22.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[42243]: pgmap v294: 148 pgs: 148 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 662 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-24T11:59:22.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:59:22.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:22 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:59:22.419 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:22.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:22.480 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:22.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:22.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:22 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T11:59:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:22 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T11:59:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:22 smithi193 ceph-mon[33593]: 2.1a deep-scrub starts 2024-03-24T11:59:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:22 smithi193 ceph-mon[33593]: 2.1a deep-scrub ok 2024-03-24T11:59:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:22 smithi193 ceph-mon[33593]: 6.1f scrub ok 2024-03-24T11:59:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:22 smithi193 ceph-mon[33593]: pgmap v294: 148 pgs: 148 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 662 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-24T11:59:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:22 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:59:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:22 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:59:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:22.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:22.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:22.769 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:22.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:22.820 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 22 16 46113 46097 130.931 103.688 0.00611217 0.00763238 2024-03-24T11:59:22.871 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:22.908 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:22.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:23.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:23.052 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:23.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:23.153 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:23.190 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:23.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:23.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:23.331 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:23.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:23.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:23.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:23.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:23.498 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 2.12 scrub starts 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 2.12 scrub ok 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T11:59:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T11:59:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 2.12 scrub starts 2024-03-24T11:59:23.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 2.12 scrub ok 2024-03-24T11:59:23.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T11:59:23.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T11:59:23.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:59:23.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T11:59:23.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:23 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T11:59:23.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 2.12 scrub starts 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 2.12 scrub ok 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T11:59:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:23 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T11:59:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:23.605 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:23.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:23.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:23.746 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:23.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:23.820 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 23 15 47891 47876 130.073 111.188 0.0142412 0.00768294 2024-03-24T11:59:23.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:23.888 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:23.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:23.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:24.038 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:24.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:24.045 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:59:24.045 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:59:24.045 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-24T11:59:24.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:59:24.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:24.185 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:24.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:24.286 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:24.318 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:24.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:24.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:24.469 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:24.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:24.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: pgmap v296: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T11:59:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T11:59:24.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T11:59:24.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T11:59:24.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T11:59:24.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: pgmap v296: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-03-24T11:59:24.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T11:59:24.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:24 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T11:59:24.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:24.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T11:59:24.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T11:59:24.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T11:59:24.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T11:59:24.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T11:59:24.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T11:59:24.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: pgmap v296: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-03-24T11:59:24.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T11:59:24.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:24 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T11:59:24.621 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:24.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:24.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:24.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:24.820 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 24 16 49421 49405 128.634 95.5625 0.0112154 0.00776886 2024-03-24T11:59:24.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:24.904 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:24.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:24.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:25.006 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:25.041 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:25.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:25.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:25.196 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:25.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:25.276 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T11:59:25.297 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:25.328 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:25.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:25.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:25.466 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:25.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:25.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[30223]: 6.7 scrub starts 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[30223]: 6.7 scrub ok 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3461318797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[30223]: from='client.35060 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[30223]: 4.10 deep-scrub starts 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[30223]: 4.10 deep-scrub ok 2024-03-24T11:59:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T11:59:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T11:59:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[42243]: 6.7 scrub starts 2024-03-24T11:59:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[42243]: 6.7 scrub ok 2024-03-24T11:59:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3461318797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:59:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[42243]: from='client.35060 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:59:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[42243]: 4.10 deep-scrub starts 2024-03-24T11:59:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:25 smithi064 ceph-mon[42243]: 4.10 deep-scrub ok 2024-03-24T11:59:25.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:25 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T11:59:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:25 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T11:59:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:25 smithi193 ceph-mon[33593]: 6.7 scrub starts 2024-03-24T11:59:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:25 smithi193 ceph-mon[33593]: 6.7 scrub ok 2024-03-24T11:59:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:25 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3461318797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:59:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:25 smithi193 ceph-mon[33593]: from='client.35060 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:59:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:25 smithi193 ceph-mon[33593]: 4.10 deep-scrub starts 2024-03-24T11:59:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:25 smithi193 ceph-mon[33593]: 4.10 deep-scrub ok 2024-03-24T11:59:25.607 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:25.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:25.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:25.743 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:25.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:25.820 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 25 16 51109 51093 127.709 105.5 0.00642966 0.00782518 2024-03-24T11:59:25.845 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:25.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:25.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:25.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:26.028 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:26.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:26.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:26.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:26.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:26.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:26.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:26.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:26.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:26.448 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:26.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:26.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: 5.1 deep-scrub starts 2024-03-24T11:59:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: 5.1 deep-scrub ok 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: pgmap v297: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: from='client.35060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:26.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: 5.1 deep-scrub starts 2024-03-24T11:59:26.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: 5.1 deep-scrub ok 2024-03-24T11:59:26.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T11:59:26.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T11:59:26.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T11:59:26.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T11:59:26.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: pgmap v297: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-24T11:59:26.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:26.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: from='client.35060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:59:26.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:59:26.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:26.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:26.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: 5.1 deep-scrub starts 2024-03-24T11:59:26.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: 5.1 deep-scrub ok 2024-03-24T11:59:26.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T11:59:26.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T11:59:26.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T11:59:26.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T11:59:26.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: pgmap v297: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-24T11:59:26.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:26.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: from='client.35060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:59:26.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:59:26.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:26.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:26.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:26.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:26.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:26.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:26.822 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 26 16 52593 52577 126.364 92.75 0.00707665 0.00790905 2024-03-24T11:59:26.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:26.866 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:26.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:26.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:27.010 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:27.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:27.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:27.152 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:27.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:27.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:27.254 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:27.287 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:27.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:27.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:27.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:27.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:27.549 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:59:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:59:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:27.582 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:27.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:27.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:59:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:59:27.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:27.725 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:27.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:27.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 27 16 54066 54050 125.093 92.0625 0.00440739 0.00798823 2024-03-24T11:59:27.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:27.877 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:27.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:27.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:28.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:28.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:28.119 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:28.151 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:28.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:28.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:28.281 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:59:28.281 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:59:28.282 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:59:28.283 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:59:28.283 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:59:28.284 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:28.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:28.386 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:28.422 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:28.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:28.524 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:28.558 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:28.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:28.660 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:28.692 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:28.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:28.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:28.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 28 16 55652 55636 124.166 99.125 0.0049693 0.00804818 2024-03-24T11:59:28.831 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:28.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:28.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:28.932 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:28.966 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:28.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:28 smithi064 ceph-mon[30223]: pgmap v299: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-03-24T11:59:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3226353568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:29.021 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:29.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:28 smithi064 ceph-mon[42243]: pgmap v299: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-03-24T11:59:29.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:28 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3226353568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:29.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:28 smithi193 ceph-mon[33593]: pgmap v299: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-03-24T11:59:29.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:28 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3226353568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:29.103 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:29.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:29.205 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:29.224 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T11:59:29.224 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:59:29.241 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:29.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:29.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:29.384 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:29.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:29.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:29.522 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:29.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:29.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:29.659 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:29.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:29.760 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:29.795 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:29.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:29.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 29 16 57257 57241 123.343 100.312 0.00981326 0.00810281 2024-03-24T11:59:29.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:29.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:30.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:30.072 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:30.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:30.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:30.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:30.211 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:30.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:30.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:59:30.278 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T11:59:30.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:30.326 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:30.373 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:30.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:30.474 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:30.521 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:30.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:30.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:30.642 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:59:30.661 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:30.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:30.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:30.762 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:30.795 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:30.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:30.800 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:30 smithi064 ceph-mon[30223]: pgmap v300: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-03-24T11:59:30.800 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:30 smithi064 ceph-mon[42243]: pgmap v300: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-03-24T11:59:30.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 30 16 58728 58712 122.296 91.9375 0.0255225 0.00817137 2024-03-24T11:59:30.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:30.939 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:30.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:31.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:31.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:59:31.079 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:30 smithi193 ceph-mon[33593]: pgmap v300: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-03-24T11:59:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:31.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:31.185 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:31.192 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:31.217 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:31.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:31.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:31.344 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:31.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:31.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:31.480 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:31.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:31.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:31.623 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:31.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:31.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:31.760 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:31.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:31.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 31 16 60147 60131 121.212 88.6875 0.00663693 0.00824486 2024-03-24T11:59:31.861 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:31.896 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:31.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:31.998 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:32.034 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:32.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:32.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:32.174 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:32.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:32.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:32.319 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:32.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:32.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:32.458 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:32.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:32.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:32.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:32.596 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:32.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:32.698 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:32.722 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:32.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:32.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 32 16 61381 61365 119.834 77.125 0.00359434 0.00833885 2024-03-24T11:59:32.827 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:32.868 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:32.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:32.924 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:32 smithi193 ceph-mon[33593]: pgmap v301: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 567 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-24T11:59:32.969 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:33.006 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:33.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:33.018 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:59:33.018 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T11:59:33.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.018 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35033.0:61675 8.b e181/170 8:db0fa11c:::benchmark_data_smithi064_120708_object61674:head v 181'3767, mlcod=181'3764)", 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:33.018515+0000", 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.00034764399999999998, 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00036857600000000002, 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:33.018515+0000", 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:33.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:33.018515+0000", 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:33.018516+0000", 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5300000000000002e-07 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:33.018585+0000", 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8428000000000003e-05 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:33.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:33.018592+0000", 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8120000000000002e-06 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:33.018597+0000", 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0860000000000001e-06 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:33.018650+0000", 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3523000000000002e-05 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:33.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:33.018675+0000", 2024-03-24T11:59:33.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5069999999999999e-05 2024-03-24T11:59:33.022 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:33.022 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:33.022 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:33.022 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:33.022 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T11:59:33.022 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 1 2024-03-24T11:59:33.022 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:59:33.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:32 smithi064 ceph-mon[30223]: pgmap v301: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 567 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-24T11:59:33.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:32 smithi064 ceph-mon[42243]: pgmap v301: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 567 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-24T11:59:33.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:33.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:33.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:33.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:33.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:33.292 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:33.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:33.393 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:33.429 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:33.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:33.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:33.555 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:33.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:33.656 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:33.689 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:33.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:33.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:33.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 33 16 62706 62690 118.712 82.8125 0.00880713 0.00841808 2024-03-24T11:59:33.826 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:33.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:33.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T11:59:33.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:33.980 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:33.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:34.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:34.120 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:34.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:34.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:34.264 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:59:34.267 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:34.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:34.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:34.415 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:34.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:34.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:34.557 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:34.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:34.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:34.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:34.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:34.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:34.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 34 16 64137 64121 117.851 89.4375 0.00829501 0.00848032 2024-03-24T11:59:34.842 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:34.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:34.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:34.979 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:34.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:35.029 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:34 smithi064 ceph-mon[30223]: pgmap v302: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:59:35.029 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:34 smithi064 ceph-mon[42243]: pgmap v302: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:59:35.043 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:34 smithi193 ceph-mon[33593]: pgmap v302: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:59:35.062 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:35.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:35.139 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:35.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:35.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:35.275 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:35.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:35.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:35.408 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:35.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:35.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:59:35.534 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:59:35.549 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:35.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:35.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:35.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:35.683 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:35.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:35.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:35.819 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:35.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:35.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 35 16 65388 65372 116.717 78.1875 0.00569097 0.00856264 2024-03-24T11:59:35.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:35.959 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:35.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:36.061 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:36.086 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:36.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:36.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:36.235 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:36.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:36.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:36.381 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:36.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:36.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:36.524 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:36.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:36.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:36.673 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:36.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:36.774 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:36.807 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:36.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:36.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 36 16 66717 66701 115.782 83.0625 0.00748435 0.00863224 2024-03-24T11:59:36.909 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:36.955 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:36.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:37.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:36 smithi064 ceph-mon[30223]: pgmap v303: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:59:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:36 smithi064 ceph-mon[30223]: from='client.25093 v1:172.21.15.64:0/2731352904' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:59:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:36 smithi064 ceph-mon[42243]: pgmap v303: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:59:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:36 smithi064 ceph-mon[42243]: from='client.25093 v1:172.21.15.64:0/2731352904' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:59:37.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:36 smithi193 ceph-mon[33593]: pgmap v303: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:59:37.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:36 smithi193 ceph-mon[33593]: from='client.25093 v1:172.21.15.64:0/2731352904' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:59:37.057 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:37.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:37.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:37.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:37.230 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:37.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:37.251 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:59:37.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T11:59:37.252 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T11:59:37.252 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T11:59:37.252 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:59:37.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:37.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:59:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:59:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:37.373 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:37.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:37.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:37.512 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:37.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:37.614 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:37.639 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:37.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:37.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:37.777 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:37.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:37.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:59:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:59:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:37.821 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 37 15 67969 67954 114.77 78.3125 0.0132697 0.00870856 2024-03-24T11:59:37.879 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:37.913 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:37.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:38.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:38.053 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:38.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:38.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:38.194 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:38.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:38.296 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:38.331 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:38.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:38.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T11:59:38.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:38.475 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:38.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:38.576 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:38.599 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:38.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:38.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:38.707 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:59:38.738 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:38.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:38.822 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 38 16 69209 69193 113.787 77.4375 0.0093468 0.00878326 2024-03-24T11:59:38.840 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:38.887 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:38.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:38.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:39.013 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:39.013 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:38 smithi193 ceph-mon[33593]: pgmap v304: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-24T11:59:39.022 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:39.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:38 smithi064 ceph-mon[30223]: pgmap v304: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-24T11:59:39.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:39.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:38 smithi064 ceph-mon[42243]: pgmap v304: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-24T11:59:39.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:39.173 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-24T11:59:35.252922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3915520350,"num_objects":59935,"num_object_clones":0,"num_object_copies":179805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59935,"num_whiteouts":0,"num_read":945,"num_read_kb":791,"num_write":60352,"num_write_kb":3825629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1772300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60419,"ondisk_log_size":60419,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12842332,"kb_used_data":11354588,"kb_used_omap":12,"kb_used_meta":1487667,"kb_avail":737118884,"statfs":{"total":767960285184,"available":754809737216,"internally_reserved":0,"allocated":11627098112,"data_stored":11623788282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1523371568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,5,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1033437184,"num_objects":15769,"num_object_clones":0,"num_object_copies":47307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15769,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15769,"num_write_kb":1009216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003154"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":253,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268392+0000","last_change":"2024-03-24T11:59:02.144257+0000","last_active":"2024-03-24T11:59:25.268392+0000","last_peered":"2024-03-24T11:59:25.268392+0000","last_clean":"2024-03-24T11:59:25.268392+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:59:25.268392+0000","last_undegraded":"2024-03-24T11:59:25.268392+0000","last_fullsized":"2024-03-24T11:59:25.268392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:02.144154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:02.144154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:12.729914+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":271,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.266312+0000","last_change":"2024-03-24T11:59:17.681747+0000","last_active":"2024-03-24T11:59:25.266312+0000","last_peered":"2024-03-24T11:59:25.266312+0000","last_clean":"2024-03-24T11:59:25.266312+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:59:25.266312+0000","last_undegraded":"2024-03-24T11:59:25.266312+0000","last_fullsized":"2024-03-24T11:59:25.266312+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:17.681628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:59:17.681628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.437272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":257,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274872+0000","last_change":"2024-03-24T11:56:59.281520+0000","last_active":"2024-03-24T11:59:25.274872+0000","last_peered":"2024-03-24T11:59:25.274872+0000","last_clean":"2024-03-24T11:59:25.274872+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:59:25.274872+0000","last_undegraded":"2024-03-24T11:59:25.274872+0000","last_fullsized":"2024-03-24T11:59:25.274872+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:56:59.278906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T11:56:59.278906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"117'22","reported_seq":293,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268998+0000","last_change":"2024-03-24T11:58:12.849968+0000","last_active":"2024-03-24T11:59:25.268998+0000","last_peered":"2024-03-24T11:59:25.268998+0000","last_clean":"2024-03-24T11:59:25.268998+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:59:25.268998+0000","last_undegraded":"2024-03-24T11:59:25.268998+0000","last_fullsized":"2024-03-24T11:59:25.268998+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'22","last_scrub_stamp":"2024-03-24T11:58:12.849803+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T11:58:12.849803+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:27.305188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":259,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268759+0000","last_change":"2024-03-24T11:58:58.872999+0000","last_active":"2024-03-24T11:59:25.268759+0000","last_peered":"2024-03-24T11:59:25.268759+0000","last_clean":"2024-03-24T11:59:25.268759+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:59:25.268759+0000","last_undegraded":"2024-03-24T11:59:25.268759+0000","last_fullsized":"2024-03-24T11:59:25.268759+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:58.872877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:58.872877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:12.680882+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":17,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264317+0000","last_change":"2024-03-24T11:59:19.117600+0000","last_active":"2024-03-24T11:59:25.264317+0000","last_peered":"2024-03-24T11:59:25.264317+0000","last_clean":"2024-03-24T11:59:25.264317+0000","last_became_active":"2024-03-24T11:59:19.117244+0000","last_became_peered":"2024-03-24T11:59:19.117244+0000","last_unstale":"2024-03-24T11:59:25.264317+0000","last_undegraded":"2024-03-24T11:59:25.264317+0000","last_fullsized":"2024-03-24T11:59:25.264317+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:01.340280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:01.340280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:21.473542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.271081+0000","last_change":"2024-03-24T11:58:07.728708+0000","last_active":"2024-03-24T11:59:25.271081+0000","last_peered":"2024-03-24T11:59:25.271081+0000","last_clean":"2024-03-24T11:59:25.271081+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:59:25.271081+0000","last_undegraded":"2024-03-24T11:59:25.271081+0000","last_fullsized":"2024-03-24T11:59:25.271081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.728598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.728598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"176'13","reported_seq":287,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.266064+0000","last_change":"2024-03-24T11:59:14.935407+0000","last_active":"2024-03-24T11:59:25.266064+0000","last_peered":"2024-03-24T11:59:25.266064+0000","last_clean":"2024-03-24T11:59:25.266064+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:59:25.266064+0000","last_undegraded":"2024-03-24T11:59:25.266064+0000","last_fullsized":"2024-03-24T11:59:25.266064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'13","last_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.935318+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:32.358216+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":259,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264182+0000","last_change":"2024-03-24T11:58:12.711969+0000","last_active":"2024-03-24T11:59:25.264182+0000","last_peered":"2024-03-24T11:59:25.264182+0000","last_clean":"2024-03-24T11:59:25.264182+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:59:25.264182+0000","last_undegraded":"2024-03-24T11:59:25.264182+0000","last_fullsized":"2024-03-24T11:59:25.264182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:12.711859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:58:12.711859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.266739+0000","last_change":"2024-03-24T11:59:17.067826+0000","last_active":"2024-03-24T11:59:25.266739+0000","last_peered":"2024-03-24T11:59:25.266739+0000","last_clean":"2024-03-24T11:59:25.266739+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:59:25.266739+0000","last_undegraded":"2024-03-24T11:59:25.266739+0000","last_fullsized":"2024-03-24T11:59:25.266739+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T11:59:17.067728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:32.451199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":272,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267921+0000","last_change":"2024-03-24T11:58:06.122994+0000","last_active":"2024-03-24T11:59:25.267921+0000","last_peered":"2024-03-24T11:59:25.267921+0000","last_clean":"2024-03-24T11:59:25.267921+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:59:25.267921+0000","last_undegraded":"2024-03-24T11:59:25.267921+0000","last_fullsized":"2024-03-24T11:59:25.267921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:06.122853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:23.689192+0000","last_clean_scrub_stamp":"2024-03-24T11:58:06.122853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":279,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268920+0000","last_change":"2024-03-24T11:58:53.841225+0000","last_active":"2024-03-24T11:59:25.268920+0000","last_peered":"2024-03-24T11:59:25.268920+0000","last_clean":"2024-03-24T11:59:25.268920+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:59:25.268920+0000","last_undegraded":"2024-03-24T11:59:25.268920+0000","last_fullsized":"2024-03-24T11:59:25.268920+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:58:53.841123+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T11:58:53.841123+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:02.782949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":244,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.273762+0000","last_change":"2024-03-24T11:58:57.644377+0000","last_active":"2024-03-24T11:59:25.273762+0000","last_peered":"2024-03-24T11:59:25.273762+0000","last_clean":"2024-03-24T11:59:25.273762+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:59:25.273762+0000","last_undegraded":"2024-03-24T11:59:25.273762+0000","last_fullsized":"2024-03-24T11:59:25.273762+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:57.644266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.644266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:58.789635+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264334+0000","last_change":"2024-03-24T11:58:03.766621+0000","last_active":"2024-03-24T11:59:25.264334+0000","last_peered":"2024-03-24T11:59:25.264334+0000","last_clean":"2024-03-24T11:59:25.264334+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:59:25.264334+0000","last_undegraded":"2024-03-24T11:59:25.264334+0000","last_fullsized":"2024-03-24T11:59:25.264334+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:03.766504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:03.766504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":275,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267516+0000","last_change":"2024-03-24T11:58:57.909335+0000","last_active":"2024-03-24T11:59:25.267516+0000","last_peered":"2024-03-24T11:59:25.267516+0000","last_clean":"2024-03-24T11:59:25.267516+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:59:25.267516+0000","last_undegraded":"2024-03-24T11:59:25.267516+0000","last_fullsized":"2024-03-24T11:59:25.267516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:57.909246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.909246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:23.726088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":283,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268798+0000","last_change":"2024-03-24T11:58:00.717857+0000","last_active":"2024-03-24T11:59:25.268798+0000","last_peered":"2024-03-24T11:59:25.268798+0000","last_clean":"2024-03-24T11:59:25.268798+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:59:25.268798+0000","last_undegraded":"2024-03-24T11:59:25.268798+0000","last_fullsized":"2024-03-24T11:59:25.268798+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:58:00.717763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:58:00.717763+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":262,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268193+0000","last_change":"2024-03-24T11:59:21.206602+0000","last_active":"2024-03-24T11:59:25.268193+0000","last_peered":"2024-03-24T11:59:25.268193+0000","last_clean":"2024-03-24T11:59:25.268193+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:59:25.268193+0000","last_undegraded":"2024-03-24T11:59:25.268193+0000","last_fullsized":"2024-03-24T11:59:25.268193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.206482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.206482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:42.128353+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269340+0000","last_change":"2024-03-24T11:59:22.719982+0000","last_active":"2024-03-24T11:59:25.269340+0000","last_peered":"2024-03-24T11:59:25.269340+0000","last_clean":"2024-03-24T11:59:25.269340+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:59:25.269340+0000","last_undegraded":"2024-03-24T11:59:25.269340+0000","last_fullsized":"2024-03-24T11:59:25.269340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.719851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.719851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.894655+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":271,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268895+0000","last_change":"2024-03-24T11:57:56.720710+0000","last_active":"2024-03-24T11:59:25.268895+0000","last_peered":"2024-03-24T11:59:25.268895+0000","last_clean":"2024-03-24T11:59:25.268895+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:59:25.268895+0000","last_undegraded":"2024-03-24T11:59:25.268895+0000","last_fullsized":"2024-03-24T11:59:25.268895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:57:56.720566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:57:56.720566+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":307,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268714+0000","last_change":"2024-03-24T11:58:53.063331+0000","last_active":"2024-03-24T11:59:25.268714+0000","last_peered":"2024-03-24T11:59:25.268714+0000","last_clean":"2024-03-24T11:59:25.268714+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:59:25.268714+0000","last_undegraded":"2024-03-24T11:59:25.268714+0000","last_fullsized":"2024-03-24T11:59:25.268714+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:58:53.063227+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:58:53.063227+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:16.650747+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":260,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269153+0000","last_change":"2024-03-24T11:58:57.872490+0000","last_active":"2024-03-24T11:59:25.269153+0000","last_peered":"2024-03-24T11:59:25.269153+0000","last_clean":"2024-03-24T11:59:25.269153+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:59:25.269153+0000","last_undegraded":"2024-03-24T11:59:25.269153+0000","last_fullsized":"2024-03-24T11:59:25.269153+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:57.872392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.872392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:25.514528+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269225+0000","last_change":"2024-03-24T11:59:21.674710+0000","last_active":"2024-03-24T11:59:25.269225+0000","last_peered":"2024-03-24T11:59:25.269225+0000","last_clean":"2024-03-24T11:59:25.269225+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T11:59:25.269225+0000","last_undegraded":"2024-03-24T11:59:25.269225+0000","last_fullsized":"2024-03-24T11:59:25.269225+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.674636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.674636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:27.485678+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274869+0000","last_change":"2024-03-24T11:59:20.890264+0000","last_active":"2024-03-24T11:59:25.274869+0000","last_peered":"2024-03-24T11:59:25.274869+0000","last_clean":"2024-03-24T11:59:25.274869+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:59:25.274869+0000","last_undegraded":"2024-03-24T11:59:25.274869+0000","last_fullsized":"2024-03-24T11:59:25.274869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_clean_scrub_stamp":"2024-03-24T11:59:20.890106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:48.280238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"115'22","reported_seq":301,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264577+0000","last_change":"2024-03-24T11:58:55.811062+0000","last_active":"2024-03-24T11:59:25.264577+0000","last_peered":"2024-03-24T11:59:25.264577+0000","last_clean":"2024-03-24T11:59:25.264577+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:59:25.264577+0000","last_undegraded":"2024-03-24T11:59:25.264577+0000","last_fullsized":"2024-03-24T11:59:25.264577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'22","last_scrub_stamp":"2024-03-24T11:58:55.810966+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T11:58:55.810966+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:00.810200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264801+0000","last_change":"2024-03-24T11:59:19.099058+0000","last_active":"2024-03-24T11:59:25.264801+0000","last_peered":"2024-03-24T11:59:25.264801+0000","last_clean":"2024-03-24T11:59:25.264801+0000","last_became_active":"2024-03-24T11:59:19.098557+0000","last_became_peered":"2024-03-24T11:59:19.098557+0000","last_unstale":"2024-03-24T11:59:25.264801+0000","last_undegraded":"2024-03-24T11:59:25.264801+0000","last_fullsized":"2024-03-24T11:59:25.264801+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:05.394143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.394143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:24.760554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":257,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275118+0000","last_change":"2024-03-24T11:58:07.457814+0000","last_active":"2024-03-24T11:59:25.275118+0000","last_peered":"2024-03-24T11:59:25.275118+0000","last_clean":"2024-03-24T11:59:25.275118+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:59:25.275118+0000","last_undegraded":"2024-03-24T11:59:25.275118+0000","last_fullsized":"2024-03-24T11:59:25.275118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.457593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":267,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268210+0000","last_change":"2024-03-24T11:59:09.953682+0000","last_active":"2024-03-24T11:59:25.268210+0000","last_peered":"2024-03-24T11:59:25.268210+0000","last_clean":"2024-03-24T11:59:25.268210+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:59:25.268210+0000","last_undegraded":"2024-03-24T11:59:25.268210+0000","last_fullsized":"2024-03-24T11:59:25.268210+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:59:09.953549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.953549+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.935470+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":294,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275872+0000","last_change":"2024-03-24T11:59:01.788586+0000","last_active":"2024-03-24T11:59:25.275872+0000","last_peered":"2024-03-24T11:59:25.275872+0000","last_clean":"2024-03-24T11:59:25.275872+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:59:25.275872+0000","last_undegraded":"2024-03-24T11:59:25.275872+0000","last_fullsized":"2024-03-24T11:59:25.275872+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:01.788481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:01.788481+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:14.227573+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":261,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267908+0000","last_change":"2024-03-24T11:58:10.110600+0000","last_active":"2024-03-24T11:59:25.267908+0000","last_peered":"2024-03-24T11:59:25.267908+0000","last_clean":"2024-03-24T11:59:25.267908+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:59:25.267908+0000","last_undegraded":"2024-03-24T11:59:25.267908+0000","last_fullsized":"2024-03-24T11:59:25.267908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:58:10.110510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:10.110510+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":362,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.276187+0000","last_change":"2024-03-24T11:59:16.869955+0000","last_active":"2024-03-24T11:59:25.276187+0000","last_peered":"2024-03-24T11:59:25.276187+0000","last_clean":"2024-03-24T11:59:25.276187+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:59:25.276187+0000","last_undegraded":"2024-03-24T11:59:25.276187+0000","last_fullsized":"2024-03-24T11:59:25.276187+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:16.869590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.869590+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:23.849397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":271,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264163+0000","last_change":"2024-03-24T11:59:04.929757+0000","last_active":"2024-03-24T11:59:25.264163+0000","last_peered":"2024-03-24T11:59:25.264163+0000","last_clean":"2024-03-24T11:59:25.264163+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:59:25.264163+0000","last_undegraded":"2024-03-24T11:59:25.264163+0000","last_fullsized":"2024-03-24T11:59:25.264163+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:04.929640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:04.929640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:28.037222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":281,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275011+0000","last_change":"2024-03-24T11:59:08.692823+0000","last_active":"2024-03-24T11:59:25.275011+0000","last_peered":"2024-03-24T11:59:25.275011+0000","last_clean":"2024-03-24T11:59:25.275011+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:59:25.275011+0000","last_undegraded":"2024-03-24T11:59:25.275011+0000","last_fullsized":"2024-03-24T11:59:25.275011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.692644+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:09.789540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269571+0000","last_change":"2024-03-24T11:59:06.058871+0000","last_active":"2024-03-24T11:59:25.269571+0000","last_peered":"2024-03-24T11:59:25.269571+0000","last_clean":"2024-03-24T11:59:25.269571+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:59:25.269571+0000","last_undegraded":"2024-03-24T11:59:25.269571+0000","last_fullsized":"2024-03-24T11:59:25.269571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:06.058633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.058633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.404319+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":257,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268147+0000","last_change":"2024-03-24T11:58:00.121569+0000","last_active":"2024-03-24T11:59:25.268147+0000","last_peered":"2024-03-24T11:59:25.268147+0000","last_clean":"2024-03-24T11:59:25.268147+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:59:25.268147+0000","last_undegraded":"2024-03-24T11:59:25.268147+0000","last_fullsized":"2024-03-24T11:59:25.268147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T11:58:00.121445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":271,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275697+0000","last_change":"2024-03-24T11:59:04.798046+0000","last_active":"2024-03-24T11:59:25.275697+0000","last_peered":"2024-03-24T11:59:25.275697+0000","last_clean":"2024-03-24T11:59:25.275697+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:59:25.275697+0000","last_undegraded":"2024-03-24T11:59:25.275697+0000","last_fullsized":"2024-03-24T11:59:25.275697+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:04.797928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:04.797928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.060878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":281,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268574+0000","last_change":"2024-03-24T11:59:07.046859+0000","last_active":"2024-03-24T11:59:25.268574+0000","last_peered":"2024-03-24T11:59:25.268574+0000","last_clean":"2024-03-24T11:59:25.268574+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:59:25.268574+0000","last_undegraded":"2024-03-24T11:59:25.268574+0000","last_fullsized":"2024-03-24T11:59:25.268574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:07.046695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.046695+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.385099+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":253,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.266617+0000","last_change":"2024-03-24T11:58:01.133824+0000","last_active":"2024-03-24T11:59:25.266617+0000","last_peered":"2024-03-24T11:59:25.266617+0000","last_clean":"2024-03-24T11:59:25.266617+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:59:25.266617+0000","last_undegraded":"2024-03-24T11:59:25.266617+0000","last_fullsized":"2024-03-24T11:59:25.266617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T11:58:01.133663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274757+0000","last_change":"2024-03-24T11:58:57.780289+0000","last_active":"2024-03-24T11:59:25.274757+0000","last_peered":"2024-03-24T11:59:25.274757+0000","last_clean":"2024-03-24T11:59:25.274757+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:59:25.274757+0000","last_undegraded":"2024-03-24T11:59:25.274757+0000","last_fullsized":"2024-03-24T11:59:25.274757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:57.780180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.780180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:01.258944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264851+0000","last_change":"2024-03-24T11:58:01.734332+0000","last_active":"2024-03-24T11:59:25.264851+0000","last_peered":"2024-03-24T11:59:25.264851+0000","last_clean":"2024-03-24T11:59:25.264851+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:59:25.264851+0000","last_undegraded":"2024-03-24T11:59:25.264851+0000","last_fullsized":"2024-03-24T11:59:25.264851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:01.734097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:01.734097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:34.994396+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"180'6","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.273537+0000","last_change":"2024-03-24T11:59:25.157079+0000","last_active":"2024-03-24T11:59:25.273537+0000","last_peered":"2024-03-24T11:59:25.273537+0000","last_clean":"2024-03-24T11:59:25.273537+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:59:25.273537+0000","last_undegraded":"2024-03-24T11:59:25.273537+0000","last_fullsized":"2024-03-24T11:59:25.273537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'6","last_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T11:59:25.156553+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:29.459760+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267982+0000","last_change":"2024-03-24T11:59:08.970434+0000","last_active":"2024-03-24T11:59:25.267982+0000","last_peered":"2024-03-24T11:59:25.267982+0000","last_clean":"2024-03-24T11:59:25.267982+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:59:25.267982+0000","last_undegraded":"2024-03-24T11:59:25.267982+0000","last_fullsized":"2024-03-24T11:59:25.267982+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:08.970296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.970296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.288013+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":17,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.265096+0000","last_change":"2024-03-24T11:59:19.100130+0000","last_active":"2024-03-24T11:59:25.265096+0000","last_peered":"2024-03-24T11:59:25.265096+0000","last_clean":"2024-03-24T11:59:25.265096+0000","last_became_active":"2024-03-24T11:59:19.099779+0000","last_became_peered":"2024-03-24T11:59:19.099779+0000","last_unstale":"2024-03-24T11:59:25.265096+0000","last_undegraded":"2024-03-24T11:59:25.265096+0000","last_fullsized":"2024-03-24T11:59:25.265096+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:53.764112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:57:53.764112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":273,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269451+0000","last_change":"2024-03-24T11:58:53.199843+0000","last_active":"2024-03-24T11:59:25.269451+0000","last_peered":"2024-03-24T11:59:25.269451+0000","last_clean":"2024-03-24T11:59:25.269451+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:59:25.269451+0000","last_undegraded":"2024-03-24T11:59:25.269451+0000","last_fullsized":"2024-03-24T11:59:25.269451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:53.199764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:53.199764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:17.646838+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":279,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268832+0000","last_change":"2024-03-24T11:57:44.723085+0000","last_active":"2024-03-24T11:59:25.268832+0000","last_peered":"2024-03-24T11:59:25.268832+0000","last_clean":"2024-03-24T11:59:25.268832+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:59:25.268832+0000","last_undegraded":"2024-03-24T11:59:25.268832+0000","last_fullsized":"2024-03-24T11:59:25.268832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:57:44.722990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:57:44.722990+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":253,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.273242+0000","last_change":"2024-03-24T11:58:54.590753+0000","last_active":"2024-03-24T11:59:25.273242+0000","last_peered":"2024-03-24T11:59:25.273242+0000","last_clean":"2024-03-24T11:59:25.273242+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:59:25.273242+0000","last_undegraded":"2024-03-24T11:59:25.273242+0000","last_fullsized":"2024-03-24T11:59:25.273242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:54.590601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T11:58:54.590601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:55.902942+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269394+0000","last_change":"2024-03-24T11:57:59.752607+0000","last_active":"2024-03-24T11:59:25.269394+0000","last_peered":"2024-03-24T11:59:25.269394+0000","last_clean":"2024-03-24T11:59:25.269394+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:59:25.269394+0000","last_undegraded":"2024-03-24T11:59:25.269394+0000","last_fullsized":"2024-03-24T11:59:25.269394+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:59.752484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:59.752484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":279,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274543+0000","last_change":"2024-03-24T11:59:24.868399+0000","last_active":"2024-03-24T11:59:25.274543+0000","last_peered":"2024-03-24T11:59:25.274543+0000","last_clean":"2024-03-24T11:59:25.274543+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:59:25.274543+0000","last_undegraded":"2024-03-24T11:59:25.274543+0000","last_fullsized":"2024-03-24T11:59:25.274543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.868290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.868290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:28.549760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":51,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274369+0000","last_change":"2024-03-24T11:59:19.097992+0000","last_active":"2024-03-24T11:59:25.274369+0000","last_peered":"2024-03-24T11:59:25.274369+0000","last_clean":"2024-03-24T11:59:25.274369+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T11:59:25.274369+0000","last_undegraded":"2024-03-24T11:59:25.274369+0000","last_fullsized":"2024-03-24T11:59:25.274369+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:59:02.301956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:02.301956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:17.685924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":259,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269264+0000","last_change":"2024-03-24T11:59:12.776833+0000","last_active":"2024-03-24T11:59:25.269264+0000","last_peered":"2024-03-24T11:59:25.269264+0000","last_clean":"2024-03-24T11:59:25.269264+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:59:25.269264+0000","last_undegraded":"2024-03-24T11:59:25.269264+0000","last_fullsized":"2024-03-24T11:59:25.269264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T11:59:12.776686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:19.703719+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":362,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.276151+0000","last_change":"2024-03-24T11:59:24.105607+0000","last_active":"2024-03-24T11:59:25.276151+0000","last_peered":"2024-03-24T11:59:25.276151+0000","last_clean":"2024-03-24T11:59:25.276151+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:59:25.276151+0000","last_undegraded":"2024-03-24T11:59:25.276151+0000","last_fullsized":"2024-03-24T11:59:25.276151+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:24.105308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.105308+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.462310+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274641+0000","last_change":"2024-03-24T11:59:22.532876+0000","last_active":"2024-03-24T11:59:25.274641+0000","last_peered":"2024-03-24T11:59:25.274641+0000","last_clean":"2024-03-24T11:59:25.274641+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:59:25.274641+0000","last_undegraded":"2024-03-24T11:59:25.274641+0000","last_fullsized":"2024-03-24T11:59:25.274641+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.532751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.532751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:51.840534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":284,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264614+0000","last_change":"2024-03-24T11:59:19.016347+0000","last_active":"2024-03-24T11:59:25.264614+0000","last_peered":"2024-03-24T11:59:25.264614+0000","last_clean":"2024-03-24T11:59:25.264614+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:59:25.264614+0000","last_undegraded":"2024-03-24T11:59:25.264614+0000","last_fullsized":"2024-03-24T11:59:25.264614+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:59:19.016233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:19.016233+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:39.365965+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":252,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274566+0000","last_change":"2024-03-24T11:59:16.675122+0000","last_active":"2024-03-24T11:59:25.274566+0000","last_peered":"2024-03-24T11:59:25.274566+0000","last_clean":"2024-03-24T11:59:25.274566+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:59:25.274566+0000","last_undegraded":"2024-03-24T11:59:25.274566+0000","last_fullsized":"2024-03-24T11:59:25.274566+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:16.673746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.673746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:20.701023+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":360,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:30.277405+0000","last_change":"2024-03-24T11:58:13.687062+0000","last_active":"2024-03-24T11:59:30.277405+0000","last_peered":"2024-03-24T11:59:30.277405+0000","last_clean":"2024-03-24T11:59:30.277405+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:59:30.277405+0000","last_undegraded":"2024-03-24T11:59:30.277405+0000","last_fullsized":"2024-03-24T11:59:30.277405+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:58:13.686957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:13.686957+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:34.994370+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":272,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264803+0000","last_change":"2024-03-24T11:59:22.001137+0000","last_active":"2024-03-24T11:59:25.264803+0000","last_peered":"2024-03-24T11:59:25.264803+0000","last_clean":"2024-03-24T11:59:25.264803+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:59:25.264803+0000","last_undegraded":"2024-03-24T11:59:25.264803+0000","last_fullsized":"2024-03-24T11:59:25.264803+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.000913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.000913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:25.798172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":282,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269051+0000","last_change":"2024-03-24T11:57:40.662287+0000","last_active":"2024-03-24T11:59:25.269051+0000","last_peered":"2024-03-24T11:59:25.269051+0000","last_clean":"2024-03-24T11:59:25.269051+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:59:25.269051+0000","last_undegraded":"2024-03-24T11:59:25.269051+0000","last_fullsized":"2024-03-24T11:59:25.269051+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:57:40.662163+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:57:40.662163+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":260,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267812+0000","last_change":"2024-03-24T11:59:12.991274+0000","last_active":"2024-03-24T11:59:25.267812+0000","last_peered":"2024-03-24T11:59:25.267812+0000","last_clean":"2024-03-24T11:59:25.267812+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:59:25.267812+0000","last_undegraded":"2024-03-24T11:59:25.267812+0000","last_fullsized":"2024-03-24T11:59:25.267812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:12.991112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:12.991112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.564791+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":256,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264667+0000","last_change":"2024-03-24T11:58:06.748927+0000","last_active":"2024-03-24T11:59:25.264667+0000","last_peered":"2024-03-24T11:59:25.264667+0000","last_clean":"2024-03-24T11:59:25.264667+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:59:25.264667+0000","last_undegraded":"2024-03-24T11:59:25.264667+0000","last_fullsized":"2024-03-24T11:59:25.264667+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:06.748834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:06.748834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:34.994447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":271,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274803+0000","last_change":"2024-03-24T11:58:04.519887+0000","last_active":"2024-03-24T11:59:25.274803+0000","last_peered":"2024-03-24T11:59:25.274803+0000","last_clean":"2024-03-24T11:59:25.274803+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:59:25.274803+0000","last_undegraded":"2024-03-24T11:59:25.274803+0000","last_fullsized":"2024-03-24T11:59:25.274803+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:04.519727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T11:58:04.519727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:29.123627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268283+0000","last_change":"2024-03-24T11:59:23.086796+0000","last_active":"2024-03-24T11:59:25.268283+0000","last_peered":"2024-03-24T11:59:25.268283+0000","last_clean":"2024-03-24T11:59:25.268283+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:59:25.268283+0000","last_undegraded":"2024-03-24T11:59:25.268283+0000","last_fullsized":"2024-03-24T11:59:25.268283+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:59:23.086667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:23.086667+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:41.979853+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":262,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.263289+0000","last_change":"2024-03-24T11:59:23.956872+0000","last_active":"2024-03-24T11:59:25.263289+0000","last_peered":"2024-03-24T11:59:25.263289+0000","last_clean":"2024-03-24T11:59:25.263289+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:59:25.263289+0000","last_undegraded":"2024-03-24T11:59:25.263289+0000","last_fullsized":"2024-03-24T11:59:25.263289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:23.956595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:23.956595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:43.862735+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":257,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268807+0000","last_change":"2024-03-24T11:59:14.960274+0000","last_active":"2024-03-24T11:59:25.268807+0000","last_peered":"2024-03-24T11:59:25.268807+0000","last_clean":"2024-03-24T11:59:25.268807+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:59:25.268807+0000","last_undegraded":"2024-03-24T11:59:25.268807+0000","last_fullsized":"2024-03-24T11:59:25.268807+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:14.960108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.960108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:41.859382+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":274,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.262453+0000","last_change":"2024-03-24T11:58:59.882590+0000","last_active":"2024-03-24T11:59:25.262453+0000","last_peered":"2024-03-24T11:59:25.262453+0000","last_clean":"2024-03-24T11:59:25.262453+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:59:25.262453+0000","last_undegraded":"2024-03-24T11:59:25.262453+0000","last_fullsized":"2024-03-24T11:59:25.262453+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:59.882508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:59.882508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:04.582653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"115'13","reported_seq":281,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275201+0000","last_change":"2024-03-24T11:58:55.576054+0000","last_active":"2024-03-24T11:59:25.275201+0000","last_peered":"2024-03-24T11:59:25.275201+0000","last_clean":"2024-03-24T11:59:25.275201+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:59:25.275201+0000","last_undegraded":"2024-03-24T11:59:25.275201+0000","last_fullsized":"2024-03-24T11:59:25.275201+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'13","last_scrub_stamp":"2024-03-24T11:58:55.575897+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T11:58:55.575897+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:19.716606+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"181'3734","reported_seq":3765,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.184824+0000","last_change":"2024-03-24T11:58:42.913612+0000","last_active":"2024-03-24T11:59:31.184824+0000","last_peered":"2024-03-24T11:59:31.184824+0000","last_clean":"2024-03-24T11:59:31.184824+0000","last_became_active":"2024-03-24T11:58:42.913076+0000","last_became_peered":"2024-03-24T11:58:42.913076+0000","last_unstale":"2024-03-24T11:59:31.184824+0000","last_undegraded":"2024-03-24T11:59:31.184824+0000","last_fullsized":"2024-03-24T11:59:31.184824+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3734,"log_dups_size":0,"ondisk_log_size":3734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:50.673786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3734,"num_write_kb":238976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269479+0000","last_change":"2024-03-24T11:59:03.131422+0000","last_active":"2024-03-24T11:59:25.269479+0000","last_peered":"2024-03-24T11:59:25.269479+0000","last_clean":"2024-03-24T11:59:25.269479+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:59:25.269479+0000","last_undegraded":"2024-03-24T11:59:25.269479+0000","last_fullsized":"2024-03-24T11:59:25.269479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:03.131309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:03.131309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:03.652291+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":284,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269147+0000","last_change":"2024-03-24T11:59:07.635339+0000","last_active":"2024-03-24T11:59:25.269147+0000","last_peered":"2024-03-24T11:59:25.269147+0000","last_clean":"2024-03-24T11:59:25.269147+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:59:25.269147+0000","last_undegraded":"2024-03-24T11:59:25.269147+0000","last_fullsized":"2024-03-24T11:59:25.269147+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:59:07.635222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.635222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:11.202763+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":317,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.266745+0000","last_change":"2024-03-24T11:59:16.735826+0000","last_active":"2024-03-24T11:59:25.266745+0000","last_peered":"2024-03-24T11:59:25.266745+0000","last_clean":"2024-03-24T11:59:25.266745+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:59:25.266745+0000","last_undegraded":"2024-03-24T11:59:25.266745+0000","last_fullsized":"2024-03-24T11:59:25.266745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:59:16.735674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.735674+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.779276+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.262544+0000","last_change":"2024-03-24T11:58:58.833812+0000","last_active":"2024-03-24T11:59:25.262544+0000","last_peered":"2024-03-24T11:59:25.262544+0000","last_clean":"2024-03-24T11:59:25.262544+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:59:25.262544+0000","last_undegraded":"2024-03-24T11:59:25.262544+0000","last_fullsized":"2024-03-24T11:59:25.262544+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:58.833700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:58.833700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:15.607455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"181'3735","reported_seq":3767,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.719539+0000","last_change":"2024-03-24T11:58:42.973572+0000","last_active":"2024-03-24T11:59:31.719539+0000","last_peered":"2024-03-24T11:59:31.719539+0000","last_clean":"2024-03-24T11:59:31.719539+0000","last_became_active":"2024-03-24T11:58:42.972967+0000","last_became_peered":"2024-03-24T11:58:42.972967+0000","last_unstale":"2024-03-24T11:59:31.719539+0000","last_undegraded":"2024-03-24T11:59:31.719539+0000","last_fullsized":"2024-03-24T11:59:31.719539+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:58.005646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244776960,"num_objects":3735,"num_object_clones":0,"num_object_copies":11205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":239040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":253,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.273350+0000","last_change":"2024-03-24T11:59:18.622093+0000","last_active":"2024-03-24T11:59:25.273350+0000","last_peered":"2024-03-24T11:59:25.273350+0000","last_clean":"2024-03-24T11:59:25.273350+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:59:25.273350+0000","last_undegraded":"2024-03-24T11:59:25.273350+0000","last_fullsized":"2024-03-24T11:59:25.273350+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T11:59:18.621992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:44.645312+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274978+0000","last_change":"2024-03-24T11:59:19.117186+0000","last_active":"2024-03-24T11:59:25.274978+0000","last_peered":"2024-03-24T11:59:25.274978+0000","last_clean":"2024-03-24T11:59:25.274978+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T11:59:25.274978+0000","last_undegraded":"2024-03-24T11:59:25.274978+0000","last_fullsized":"2024-03-24T11:59:25.274978+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:13.313625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:59:13.313625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:23.114451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":289,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269591+0000","last_change":"2024-03-24T11:59:22.170397+0000","last_active":"2024-03-24T11:59:25.269591+0000","last_peered":"2024-03-24T11:59:25.269591+0000","last_clean":"2024-03-24T11:59:25.269591+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:59:25.269591+0000","last_undegraded":"2024-03-24T11:59:25.269591+0000","last_fullsized":"2024-03-24T11:59:25.269591+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T11:59:22.170130+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.170130+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:46.149755+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":353,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275780+0000","last_change":"2024-03-24T11:57:37.531779+0000","last_active":"2024-03-24T11:59:25.275780+0000","last_peered":"2024-03-24T11:59:25.275780+0000","last_clean":"2024-03-24T11:59:25.275780+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:59:25.275780+0000","last_undegraded":"2024-03-24T11:59:25.275780+0000","last_fullsized":"2024-03-24T11:59:25.275780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:57:37.531503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:37.531503+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:30.602566+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"181'3780","reported_seq":3812,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.195407+0000","last_change":"2024-03-24T11:58:42.913905+0000","last_active":"2024-03-24T11:59:31.195407+0000","last_peered":"2024-03-24T11:59:31.195407+0000","last_clean":"2024-03-24T11:59:31.195407+0000","last_became_active":"2024-03-24T11:58:42.913442+0000","last_became_peered":"2024-03-24T11:58:42.913442+0000","last_unstale":"2024-03-24T11:59:31.195407+0000","last_undegraded":"2024-03-24T11:59:31.195407+0000","last_fullsized":"2024-03-24T11:59:31.195407+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:11.729965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247726080,"num_objects":3780,"num_object_clones":0,"num_object_copies":11340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":241920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.271162+0000","last_change":"2024-03-24T11:58:10.828344+0000","last_active":"2024-03-24T11:59:25.271162+0000","last_peered":"2024-03-24T11:59:25.271162+0000","last_clean":"2024-03-24T11:59:25.271162+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:59:25.271162+0000","last_undegraded":"2024-03-24T11:59:25.271162+0000","last_fullsized":"2024-03-24T11:59:25.271162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:10.828263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:10.828263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:31.566020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":272,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264902+0000","last_change":"2024-03-24T11:59:09.914029+0000","last_active":"2024-03-24T11:59:25.264902+0000","last_peered":"2024-03-24T11:59:25.264902+0000","last_clean":"2024-03-24T11:59:25.264902+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:59:25.264902+0000","last_undegraded":"2024-03-24T11:59:25.264902+0000","last_fullsized":"2024-03-24T11:59:25.264902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.913946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:13.920799+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":348,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.276965+0000","last_change":"2024-03-24T11:58:11.704697+0000","last_active":"2024-03-24T11:59:25.276965+0000","last_peered":"2024-03-24T11:59:25.276965+0000","last_clean":"2024-03-24T11:59:25.276965+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:59:25.276965+0000","last_undegraded":"2024-03-24T11:59:25.276965+0000","last_fullsized":"2024-03-24T11:59:25.276965+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-03-24T11:58:11.704574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:58:11.704574+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:40.131989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":257,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275370+0000","last_change":"2024-03-24T11:59:14.667905+0000","last_active":"2024-03-24T11:59:25.275370+0000","last_peered":"2024-03-24T11:59:25.275370+0000","last_clean":"2024-03-24T11:59:25.275370+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:59:25.275370+0000","last_undegraded":"2024-03-24T11:59:25.275370+0000","last_fullsized":"2024-03-24T11:59:25.275370+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:14.667523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.667523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:15.199214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"181'3681","reported_seq":3713,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.950142+0000","last_change":"2024-03-24T11:58:42.817551+0000","last_active":"2024-03-24T11:59:31.950142+0000","last_peered":"2024-03-24T11:59:31.950142+0000","last_clean":"2024-03-24T11:59:31.950142+0000","last_became_active":"2024-03-24T11:58:42.817154+0000","last_became_peered":"2024-03-24T11:58:42.817154+0000","last_unstale":"2024-03-24T11:59:31.950142+0000","last_undegraded":"2024-03-24T11:59:31.950142+0000","last_fullsized":"2024-03-24T11:59:31.950142+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3681,"log_dups_size":0,"ondisk_log_size":3681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.457603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241238016,"num_objects":3681,"num_object_clones":0,"num_object_copies":11043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":235584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":253,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268078+0000","last_change":"2024-03-24T11:57:59.157498+0000","last_active":"2024-03-24T11:59:25.268078+0000","last_peered":"2024-03-24T11:59:25.268078+0000","last_clean":"2024-03-24T11:59:25.268078+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:59:25.268078+0000","last_undegraded":"2024-03-24T11:59:25.268078+0000","last_fullsized":"2024-03-24T11:59:25.268078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:59.157368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:59.157368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":78,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:30.111318+0000","last_change":"2024-03-24T11:59:22.979714+0000","last_active":"2024-03-24T11:59:30.111318+0000","last_peered":"2024-03-24T11:59:30.111318+0000","last_clean":"2024-03-24T11:59:30.111318+0000","last_became_active":"2024-03-24T11:59:19.095838+0000","last_became_peered":"2024-03-24T11:59:19.095838+0000","last_unstale":"2024-03-24T11:59:30.111318+0000","last_undegraded":"2024-03-24T11:59:30.111318+0000","last_fullsized":"2024-03-24T11:59:30.111318+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:59:22.978824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.978824+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:47.966247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":282,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.262650+0000","last_change":"2024-03-24T11:59:07.080907+0000","last_active":"2024-03-24T11:59:25.262650+0000","last_peered":"2024-03-24T11:59:25.262650+0000","last_clean":"2024-03-24T11:59:25.262650+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:59:25.262650+0000","last_undegraded":"2024-03-24T11:59:25.262650+0000","last_fullsized":"2024-03-24T11:59:25.262650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:07.080763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.080763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.882906+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":288,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269633+0000","last_change":"2024-03-24T11:58:54.237004+0000","last_active":"2024-03-24T11:59:25.269633+0000","last_peered":"2024-03-24T11:59:25.269633+0000","last_clean":"2024-03-24T11:59:25.269633+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:59:25.269633+0000","last_undegraded":"2024-03-24T11:59:25.269633+0000","last_fullsized":"2024-03-24T11:59:25.269633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-03-24T11:58:54.236909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:58:54.236909+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:07.217796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":274,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.265834+0000","last_change":"2024-03-24T11:59:12.041065+0000","last_active":"2024-03-24T11:59:25.265834+0000","last_peered":"2024-03-24T11:59:25.265834+0000","last_clean":"2024-03-24T11:59:25.265834+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:59:25.265834+0000","last_undegraded":"2024-03-24T11:59:25.265834+0000","last_fullsized":"2024-03-24T11:59:25.265834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:12.040754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T11:59:12.040754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.895458+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"181'3718","reported_seq":3749,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.184600+0000","last_change":"2024-03-24T11:58:42.819958+0000","last_active":"2024-03-24T11:59:31.184600+0000","last_peered":"2024-03-24T11:59:31.184600+0000","last_clean":"2024-03-24T11:59:31.184600+0000","last_became_active":"2024-03-24T11:58:42.819563+0000","last_became_peered":"2024-03-24T11:58:42.819563+0000","last_unstale":"2024-03-24T11:59:31.184600+0000","last_undegraded":"2024-03-24T11:59:31.184600+0000","last_fullsized":"2024-03-24T11:59:31.184600+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3718,"log_dups_size":0,"ondisk_log_size":3718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:47.144387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243662848,"num_objects":3718,"num_object_clones":0,"num_object_copies":11154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":237952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":260,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.265062+0000","last_change":"2024-03-24T11:59:24.028954+0000","last_active":"2024-03-24T11:59:25.265062+0000","last_peered":"2024-03-24T11:59:25.265062+0000","last_clean":"2024-03-24T11:59:25.265062+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:59:25.265062+0000","last_undegraded":"2024-03-24T11:59:25.265062+0000","last_fullsized":"2024-03-24T11:59:25.265062+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.028868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.028868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:41.479995+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":275,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268533+0000","last_change":"2024-03-24T11:58:57.101127+0000","last_active":"2024-03-24T11:59:25.268533+0000","last_peered":"2024-03-24T11:59:25.268533+0000","last_clean":"2024-03-24T11:59:25.268533+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:59:25.268533+0000","last_undegraded":"2024-03-24T11:59:25.268533+0000","last_fullsized":"2024-03-24T11:59:25.268533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T11:58:57.101013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.101013+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:05.969198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264480+0000","last_change":"2024-03-24T11:57:45.806415+0000","last_active":"2024-03-24T11:59:25.264480+0000","last_peered":"2024-03-24T11:59:25.264480+0000","last_clean":"2024-03-24T11:59:25.264480+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:59:25.264480+0000","last_undegraded":"2024-03-24T11:59:25.264480+0000","last_fullsized":"2024-03-24T11:59:25.264480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:45.806321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:57:45.806321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":520,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267848+0000","last_change":"2024-03-24T11:58:09.128398+0000","last_active":"2024-03-24T11:59:25.267848+0000","last_peered":"2024-03-24T11:59:25.267848+0000","last_clean":"2024-03-24T11:59:25.267848+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:59:25.267848+0000","last_undegraded":"2024-03-24T11:59:25.267848+0000","last_fullsized":"2024-03-24T11:59:25.267848+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":385,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275779+0000","last_change":"2024-03-24T11:59:21.898682+0000","last_active":"2024-03-24T11:59:25.275779+0000","last_peered":"2024-03-24T11:59:25.275779+0000","last_clean":"2024-03-24T11:59:25.275779+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:59:25.275779+0000","last_undegraded":"2024-03-24T11:59:25.275779+0000","last_fullsized":"2024-03-24T11:59:25.275779+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:59:21.898325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.898325+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:22.738734+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267736+0000","last_change":"2024-03-24T11:59:13.979671+0000","last_active":"2024-03-24T11:59:25.267736+0000","last_peered":"2024-03-24T11:59:25.267736+0000","last_clean":"2024-03-24T11:59:25.267736+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:59:25.267736+0000","last_undegraded":"2024-03-24T11:59:25.267736+0000","last_fullsized":"2024-03-24T11:59:25.267736+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T11:59:13.979581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:20.284311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"181'3795","reported_seq":3832,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.484136+0000","last_change":"2024-03-24T11:59:19.104062+0000","last_active":"2024-03-24T11:59:31.484136+0000","last_peered":"2024-03-24T11:59:31.484136+0000","last_clean":"2024-03-24T11:59:31.484136+0000","last_became_active":"2024-03-24T11:59:19.098023+0000","last_became_peered":"2024-03-24T11:59:19.098023+0000","last_unstale":"2024-03-24T11:59:31.484136+0000","last_undegraded":"2024-03-24T11:59:31.484136+0000","last_fullsized":"2024-03-24T11:59:31.484136+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:04.358454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248709120,"num_objects":3795,"num_object_clones":0,"num_object_copies":11385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":242880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":261,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.271448+0000","last_change":"2024-03-24T11:59:25.199926+0000","last_active":"2024-03-24T11:59:25.271448+0000","last_peered":"2024-03-24T11:59:25.271448+0000","last_clean":"2024-03-24T11:59:25.271448+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:59:25.271448+0000","last_undegraded":"2024-03-24T11:59:25.271448+0000","last_fullsized":"2024-03-24T11:59:25.271448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:25.199804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:25.199804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:31.158743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":272,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274522+0000","last_change":"2024-03-24T11:59:21.570352+0000","last_active":"2024-03-24T11:59:25.274522+0000","last_peered":"2024-03-24T11:59:25.274522+0000","last_clean":"2024-03-24T11:59:25.274522+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:59:25.274522+0000","last_undegraded":"2024-03-24T11:59:25.274522+0000","last_fullsized":"2024-03-24T11:59:25.274522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.570115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.570115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:25.514405+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.273673+0000","last_change":"2024-03-24T11:59:20.570521+0000","last_active":"2024-03-24T11:59:25.273673+0000","last_peered":"2024-03-24T11:59:25.273673+0000","last_clean":"2024-03-24T11:59:25.273673+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T11:59:25.273673+0000","last_undegraded":"2024-03-24T11:59:25.273673+0000","last_fullsized":"2024-03-24T11:59:25.273673+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:20.570220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:20.570220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:30.373801+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":279,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274676+0000","last_change":"2024-03-24T11:59:03.726421+0000","last_active":"2024-03-24T11:59:25.274676+0000","last_peered":"2024-03-24T11:59:25.274676+0000","last_clean":"2024-03-24T11:59:25.274676+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:59:25.274676+0000","last_undegraded":"2024-03-24T11:59:25.274676+0000","last_fullsized":"2024-03-24T11:59:25.274676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:59:03.726034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:03.726034+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:08.722175+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":256,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.265033+0000","last_change":"2024-03-24T11:57:36.748767+0000","last_active":"2024-03-24T11:59:25.265033+0000","last_peered":"2024-03-24T11:59:25.265033+0000","last_clean":"2024-03-24T11:59:25.265033+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:59:25.265033+0000","last_undegraded":"2024-03-24T11:59:25.265033+0000","last_fullsized":"2024-03-24T11:59:25.265033+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:36.748684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:36.748684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"181'3808","reported_seq":3839,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.952833+0000","last_change":"2024-03-24T11:58:42.970585+0000","last_active":"2024-03-24T11:59:31.952833+0000","last_peered":"2024-03-24T11:59:31.952833+0000","last_clean":"2024-03-24T11:59:31.952833+0000","last_became_active":"2024-03-24T11:58:42.970288+0000","last_became_peered":"2024-03-24T11:58:42.970288+0000","last_unstale":"2024-03-24T11:59:31.952833+0000","last_undegraded":"2024-03-24T11:59:31.952833+0000","last_fullsized":"2024-03-24T11:59:31.952833+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:09.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249561088,"num_objects":3808,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":243712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274591+0000","last_change":"2024-03-24T11:59:19.096886+0000","last_active":"2024-03-24T11:59:25.274591+0000","last_peered":"2024-03-24T11:59:25.274591+0000","last_clean":"2024-03-24T11:59:25.274591+0000","last_became_active":"2024-03-24T11:59:19.096322+0000","last_became_peered":"2024-03-24T11:59:19.096322+0000","last_unstale":"2024-03-24T11:59:25.274591+0000","last_undegraded":"2024-03-24T11:59:25.274591+0000","last_fullsized":"2024-03-24T11:59:25.274591+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:56.287993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:56.287993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:25.635939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268179+0000","last_change":"2024-03-24T11:58:14.079445+0000","last_active":"2024-03-24T11:59:25.268179+0000","last_peered":"2024-03-24T11:59:25.268179+0000","last_clean":"2024-03-24T11:59:25.268179+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:59:25.268179+0000","last_undegraded":"2024-03-24T11:59:25.268179+0000","last_fullsized":"2024-03-24T11:59:25.268179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:14.079300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:14.079300+0000","last_clean_scrub_stamp":"2024-03-24T11:58:14.079300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":297,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274539+0000","last_change":"2024-03-24T11:59:06.717918+0000","last_active":"2024-03-24T11:59:25.274539+0000","last_peered":"2024-03-24T11:59:25.274539+0000","last_clean":"2024-03-24T11:59:25.274539+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:59:25.274539+0000","last_undegraded":"2024-03-24T11:59:25.274539+0000","last_fullsized":"2024-03-24T11:59:25.274539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:59:06.717709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.717709+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:26.508160+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274955+0000","last_change":"2024-03-24T11:59:09.760242+0000","last_active":"2024-03-24T11:59:25.274955+0000","last_peered":"2024-03-24T11:59:25.274955+0000","last_clean":"2024-03-24T11:59:25.274955+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:59:25.274955+0000","last_undegraded":"2024-03-24T11:59:25.274955+0000","last_fullsized":"2024-03-24T11:59:25.274955+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:09.760120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.760120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:11.704260+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"181'3794","reported_seq":3826,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.716079+0000","last_change":"2024-03-24T11:58:42.911757+0000","last_active":"2024-03-24T11:59:31.716079+0000","last_peered":"2024-03-24T11:59:31.716079+0000","last_clean":"2024-03-24T11:59:31.716079+0000","last_became_active":"2024-03-24T11:58:42.911298+0000","last_became_peered":"2024-03-24T11:58:42.911298+0000","last_unstale":"2024-03-24T11:59:31.716079+0000","last_undegraded":"2024-03-24T11:59:31.716079+0000","last_fullsized":"2024-03-24T11:59:31.716079+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:06.815917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248643584,"num_objects":3794,"num_object_clones":0,"num_object_copies":11382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3794,"num_write_kb":242816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":260,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268989+0000","last_change":"2024-03-24T11:58:07.866715+0000","last_active":"2024-03-24T11:59:25.268989+0000","last_peered":"2024-03-24T11:59:25.268989+0000","last_clean":"2024-03-24T11:59:25.268989+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:59:25.268989+0000","last_undegraded":"2024-03-24T11:59:25.268989+0000","last_fullsized":"2024-03-24T11:59:25.268989+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.866581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.866581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":24,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275506+0000","last_change":"2024-03-24T11:59:19.098307+0000","last_active":"2024-03-24T11:59:25.275506+0000","last_peered":"2024-03-24T11:59:25.275506+0000","last_clean":"2024-03-24T11:59:25.275506+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T11:59:25.275506+0000","last_undegraded":"2024-03-24T11:59:25.275506+0000","last_fullsized":"2024-03-24T11:59:25.275506+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:59:10.305433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:10.305433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:20.709439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":294,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.273755+0000","last_change":"2024-03-24T11:58:08.776444+0000","last_active":"2024-03-24T11:59:25.273755+0000","last_peered":"2024-03-24T11:59:25.273755+0000","last_clean":"2024-03-24T11:59:25.273755+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:59:25.273755+0000","last_undegraded":"2024-03-24T11:59:25.273755+0000","last_fullsized":"2024-03-24T11:59:25.273755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:58:08.776286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:58:08.776286+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.266852+0000","last_change":"2024-03-24T11:59:24.679291+0000","last_active":"2024-03-24T11:59:25.266852+0000","last_peered":"2024-03-24T11:59:25.266852+0000","last_clean":"2024-03-24T11:59:25.266852+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:59:25.266852+0000","last_undegraded":"2024-03-24T11:59:25.266852+0000","last_fullsized":"2024-03-24T11:59:25.266852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.679110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:52.794373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"181'3693","reported_seq":3721,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.210332+0000","last_change":"2024-03-24T11:58:42.970705+0000","last_active":"2024-03-24T11:59:31.210332+0000","last_peered":"2024-03-24T11:59:31.210332+0000","last_clean":"2024-03-24T11:59:31.210332+0000","last_became_active":"2024-03-24T11:58:42.970405+0000","last_became_peered":"2024-03-24T11:58:42.970405+0000","last_unstale":"2024-03-24T11:59:31.210332+0000","last_undegraded":"2024-03-24T11:59:31.210332+0000","last_fullsized":"2024-03-24T11:59:31.210332+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3693,"log_dups_size":0,"ondisk_log_size":3693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:49.691670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242024448,"num_objects":3693,"num_object_clones":0,"num_object_copies":11079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":236352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":261,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267193+0000","last_change":"2024-03-24T11:59:05.901391+0000","last_active":"2024-03-24T11:59:25.267193+0000","last_peered":"2024-03-24T11:59:25.267193+0000","last_clean":"2024-03-24T11:59:25.267193+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:59:25.267193+0000","last_undegraded":"2024-03-24T11:59:25.267193+0000","last_fullsized":"2024-03-24T11:59:25.267193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:05.901308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.901308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:14.214403+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":271,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275705+0000","last_change":"2024-03-24T11:58:56.778928+0000","last_active":"2024-03-24T11:59:25.275705+0000","last_peered":"2024-03-24T11:59:25.275705+0000","last_clean":"2024-03-24T11:59:25.275705+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:59:25.275705+0000","last_undegraded":"2024-03-24T11:59:25.275705+0000","last_fullsized":"2024-03-24T11:59:25.275705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:56.778822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:58:56.778822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:04.084039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":296,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.266649+0000","last_change":"2024-03-24T11:59:05.708676+0000","last_active":"2024-03-24T11:59:25.266649+0000","last_peered":"2024-03-24T11:59:25.266649+0000","last_clean":"2024-03-24T11:59:25.266649+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:59:25.266649+0000","last_undegraded":"2024-03-24T11:59:25.266649+0000","last_fullsized":"2024-03-24T11:59:25.266649+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:59:05.708532+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.708532+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.259042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":354,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275978+0000","last_change":"2024-03-24T11:59:00.089695+0000","last_active":"2024-03-24T11:59:25.275978+0000","last_peered":"2024-03-24T11:59:25.275978+0000","last_clean":"2024-03-24T11:59:25.275978+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:59:25.275978+0000","last_undegraded":"2024-03-24T11:59:25.275978+0000","last_fullsized":"2024-03-24T11:59:25.275978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:00.089617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:00.089617+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:26.813217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"181'3766","reported_seq":3798,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.174240+0000","last_change":"2024-03-24T11:58:42.820321+0000","last_active":"2024-03-24T11:59:31.174240+0000","last_peered":"2024-03-24T11:59:31.174240+0000","last_clean":"2024-03-24T11:59:31.174240+0000","last_became_active":"2024-03-24T11:58:42.819941+0000","last_became_peered":"2024-03-24T11:58:42.819941+0000","last_unstale":"2024-03-24T11:59:31.174240+0000","last_undegraded":"2024-03-24T11:59:31.174240+0000","last_fullsized":"2024-03-24T11:59:31.174240+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3766,"log_dups_size":0,"ondisk_log_size":3766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:53.735422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246808576,"num_objects":3766,"num_object_clones":0,"num_object_copies":11298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3766,"num_write_kb":241024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268948+0000","last_change":"2024-03-24T11:58:55.061715+0000","last_active":"2024-03-24T11:59:25.268948+0000","last_peered":"2024-03-24T11:59:25.268948+0000","last_clean":"2024-03-24T11:59:25.268948+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:59:25.268948+0000","last_undegraded":"2024-03-24T11:59:25.268948+0000","last_fullsized":"2024-03-24T11:59:25.268948+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:55.061633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:55.061633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:59.437964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":256,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264950+0000","last_change":"2024-03-24T11:57:40.772815+0000","last_active":"2024-03-24T11:59:25.264950+0000","last_peered":"2024-03-24T11:59:25.264950+0000","last_clean":"2024-03-24T11:59:25.264950+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:59:25.264950+0000","last_undegraded":"2024-03-24T11:59:25.264950+0000","last_fullsized":"2024-03-24T11:59:25.264950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:40.772729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:40.772729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":273,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267813+0000","last_change":"2024-03-24T11:58:07.158615+0000","last_active":"2024-03-24T11:59:25.267813+0000","last_peered":"2024-03-24T11:59:25.267813+0000","last_clean":"2024-03-24T11:59:25.267813+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:59:25.267813+0000","last_undegraded":"2024-03-24T11:59:25.267813+0000","last_fullsized":"2024-03-24T11:59:25.267813+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.158533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.158533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"178'16","reported_seq":292,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269044+0000","last_change":"2024-03-24T11:59:18.653135+0000","last_active":"2024-03-24T11:59:25.269044+0000","last_peered":"2024-03-24T11:59:25.269044+0000","last_clean":"2024-03-24T11:59:25.269044+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:59:25.269044+0000","last_undegraded":"2024-03-24T11:59:25.269044+0000","last_fullsized":"2024-03-24T11:59:25.269044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'16","last_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T11:59:18.653047+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:38.949524+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"181'3754","reported_seq":3792,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.169576+0000","last_change":"2024-03-24T11:59:19.102218+0000","last_active":"2024-03-24T11:59:31.169576+0000","last_peered":"2024-03-24T11:59:31.169576+0000","last_clean":"2024-03-24T11:59:31.169576+0000","last_became_active":"2024-03-24T11:59:19.096884+0000","last_became_peered":"2024-03-24T11:59:19.096884+0000","last_unstale":"2024-03-24T11:59:31.169576+0000","last_undegraded":"2024-03-24T11:59:31.169576+0000","last_fullsized":"2024-03-24T11:59:31.169576+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:58.719085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246022144,"num_objects":3754,"num_object_clones":0,"num_object_copies":11262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3754,"num_write_kb":240256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":260,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264722+0000","last_change":"2024-03-24T11:59:07.943563+0000","last_active":"2024-03-24T11:59:25.264722+0000","last_peered":"2024-03-24T11:59:25.264722+0000","last_clean":"2024-03-24T11:59:25.264722+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:59:25.264722+0000","last_undegraded":"2024-03-24T11:59:25.264722+0000","last_fullsized":"2024-03-24T11:59:25.264722+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.943432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:08.936009+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":363,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275862+0000","last_change":"2024-03-24T11:58:59.829483+0000","last_active":"2024-03-24T11:59:25.275862+0000","last_peered":"2024-03-24T11:59:25.275862+0000","last_clean":"2024-03-24T11:59:25.275862+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:59:25.275862+0000","last_undegraded":"2024-03-24T11:59:25.275862+0000","last_fullsized":"2024-03-24T11:59:25.275862+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:58:59.829360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:59.829360+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:24.491446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.270976+0000","last_change":"2024-03-24T11:58:00.806590+0000","last_active":"2024-03-24T11:59:25.270976+0000","last_peered":"2024-03-24T11:59:25.270976+0000","last_clean":"2024-03-24T11:59:25.270976+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:59:25.270976+0000","last_undegraded":"2024-03-24T11:59:25.270976+0000","last_fullsized":"2024-03-24T11:59:25.270976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:00.806476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:00.806476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:28.589516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"130'22","reported_seq":301,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269733+0000","last_change":"2024-03-24T11:59:09.356719+0000","last_active":"2024-03-24T11:59:25.269733+0000","last_peered":"2024-03-24T11:59:25.269733+0000","last_clean":"2024-03-24T11:59:25.269733+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:59:25.269733+0000","last_undegraded":"2024-03-24T11:59:25.269733+0000","last_fullsized":"2024-03-24T11:59:25.269733+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'22","last_scrub_stamp":"2024-03-24T11:59:09.356251+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.356251+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:36.899318+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"181'3645","reported_seq":3677,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.918140+0000","last_change":"2024-03-24T11:58:42.822374+0000","last_active":"2024-03-24T11:59:31.918140+0000","last_peered":"2024-03-24T11:59:31.918140+0000","last_clean":"2024-03-24T11:59:31.918140+0000","last_became_active":"2024-03-24T11:58:42.821264+0000","last_became_peered":"2024-03-24T11:58:42.821264+0000","last_unstale":"2024-03-24T11:59:31.918140+0000","last_undegraded":"2024-03-24T11:59:31.918140+0000","last_fullsized":"2024-03-24T11:59:31.918140+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3645,"log_dups_size":0,"ondisk_log_size":3645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:51.838517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238878720,"num_objects":3645,"num_object_clones":0,"num_object_copies":10935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3645,"num_write_kb":233280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":261,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269274+0000","last_change":"2024-03-24T11:57:57.650759+0000","last_active":"2024-03-24T11:59:25.269274+0000","last_peered":"2024-03-24T11:59:25.269274+0000","last_clean":"2024-03-24T11:59:25.269274+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:59:25.269274+0000","last_undegraded":"2024-03-24T11:59:25.269274+0000","last_fullsized":"2024-03-24T11:59:25.269274+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:57:57.650631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:57.650631+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":362,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.276673+0000","last_change":"2024-03-24T11:58:12.887923+0000","last_active":"2024-03-24T11:59:25.276673+0000","last_peered":"2024-03-24T11:59:25.276673+0000","last_clean":"2024-03-24T11:59:25.276673+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:59:25.276673+0000","last_undegraded":"2024-03-24T11:59:25.276673+0000","last_fullsized":"2024-03-24T11:59:25.276673+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:58:12.887826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:12.887826+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:30.954382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.268649+0000","last_change":"2024-03-24T11:59:19.116929+0000","last_active":"2024-03-24T11:59:25.268649+0000","last_peered":"2024-03-24T11:59:25.268649+0000","last_clean":"2024-03-24T11:59:25.268649+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T11:59:25.268649+0000","last_undegraded":"2024-03-24T11:59:25.268649+0000","last_fullsized":"2024-03-24T11:59:25.268649+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:10.932134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_clean_scrub_stamp":"2024-03-24T11:58:10.932134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:35.554155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"130'15","reported_seq":288,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269148+0000","last_change":"2024-03-24T11:58:54.798236+0000","last_active":"2024-03-24T11:59:25.269148+0000","last_peered":"2024-03-24T11:59:25.269148+0000","last_clean":"2024-03-24T11:59:25.269148+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:59:25.269148+0000","last_undegraded":"2024-03-24T11:59:25.269148+0000","last_fullsized":"2024-03-24T11:59:25.269148+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'15","last_scrub_stamp":"2024-03-24T11:58:54.798112+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T11:58:54.798112+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:14.552204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"181'3754","reported_seq":3792,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.718657+0000","last_change":"2024-03-24T11:59:19.103634+0000","last_active":"2024-03-24T11:59:31.718657+0000","last_peered":"2024-03-24T11:59:31.718657+0000","last_clean":"2024-03-24T11:59:31.718657+0000","last_became_active":"2024-03-24T11:59:19.098207+0000","last_became_peered":"2024-03-24T11:59:19.098207+0000","last_unstale":"2024-03-24T11:59:31.718657+0000","last_undegraded":"2024-03-24T11:59:31.718657+0000","last_fullsized":"2024-03-24T11:59:31.718657+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.248098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246022144,"num_objects":3754,"num_object_clones":0,"num_object_copies":11262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3754,"num_write_kb":240256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":262,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.262726+0000","last_change":"2024-03-24T11:59:05.050526+0000","last_active":"2024-03-24T11:59:25.262726+0000","last_peered":"2024-03-24T11:59:25.262726+0000","last_clean":"2024-03-24T11:59:25.262726+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:59:25.262726+0000","last_undegraded":"2024-03-24T11:59:25.262726+0000","last_fullsized":"2024-03-24T11:59:25.262726+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:05.050396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.050396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:15.630214+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":17,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269020+0000","last_change":"2024-03-24T11:59:19.100322+0000","last_active":"2024-03-24T11:59:25.269020+0000","last_peered":"2024-03-24T11:59:25.269020+0000","last_clean":"2024-03-24T11:59:25.269020+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T11:59:25.269020+0000","last_undegraded":"2024-03-24T11:59:25.269020+0000","last_fullsized":"2024-03-24T11:59:25.269020+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:08.349033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.349033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:18.291266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.262742+0000","last_change":"2024-03-24T11:59:16.062490+0000","last_active":"2024-03-24T11:59:25.262742+0000","last_peered":"2024-03-24T11:59:25.262742+0000","last_clean":"2024-03-24T11:59:25.262742+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:59:25.262742+0000","last_undegraded":"2024-03-24T11:59:25.262742+0000","last_fullsized":"2024-03-24T11:59:25.262742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:59:16.062361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.062361+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:28.948471+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"116'25","reported_seq":304,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269108+0000","last_change":"2024-03-24T11:59:06.683554+0000","last_active":"2024-03-24T11:59:25.269108+0000","last_peered":"2024-03-24T11:59:25.269108+0000","last_clean":"2024-03-24T11:59:25.269108+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:59:25.269108+0000","last_undegraded":"2024-03-24T11:59:25.269108+0000","last_fullsized":"2024-03-24T11:59:25.269108+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'25","last_scrub_stamp":"2024-03-24T11:59:06.683322+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T11:57:11.800708+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.683322+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:07.986959+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"181'3678","reported_seq":3710,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.726736+0000","last_change":"2024-03-24T11:58:42.913010+0000","last_active":"2024-03-24T11:59:31.726736+0000","last_peered":"2024-03-24T11:59:31.726736+0000","last_clean":"2024-03-24T11:59:31.726736+0000","last_became_active":"2024-03-24T11:58:42.912707+0000","last_became_peered":"2024-03-24T11:58:42.912707+0000","last_unstale":"2024-03-24T11:59:31.726736+0000","last_undegraded":"2024-03-24T11:59:31.726736+0000","last_fullsized":"2024-03-24T11:59:31.726736+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:54.222840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241041408,"num_objects":3678,"num_object_clones":0,"num_object_copies":11034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3678,"num_write_kb":235392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":261,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.267547+0000","last_change":"2024-03-24T11:59:19.138247+0000","last_active":"2024-03-24T11:59:25.267547+0000","last_peered":"2024-03-24T11:59:25.267547+0000","last_clean":"2024-03-24T11:59:25.267547+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:59:25.267547+0000","last_undegraded":"2024-03-24T11:59:25.267547+0000","last_fullsized":"2024-03-24T11:59:25.267547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:19.138097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T11:59:19.138097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:42.801466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.262954+0000","last_change":"2024-03-24T11:58:09.829306+0000","last_active":"2024-03-24T11:59:25.262954+0000","last_peered":"2024-03-24T11:59:25.262954+0000","last_clean":"2024-03-24T11:59:25.262954+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:59:25.262954+0000","last_undegraded":"2024-03-24T11:59:25.262954+0000","last_fullsized":"2024-03-24T11:59:25.262954+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:09.829226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.829226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:35.677878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":274,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264358+0000","last_change":"2024-03-24T11:57:57.819681+0000","last_active":"2024-03-24T11:59:25.264358+0000","last_peered":"2024-03-24T11:59:25.264358+0000","last_clean":"2024-03-24T11:59:25.264358+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:59:25.264358+0000","last_undegraded":"2024-03-24T11:59:25.264358+0000","last_fullsized":"2024-03-24T11:59:25.264358+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:57.819565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T11:57:57.819565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:27.190556+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":298,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274755+0000","last_change":"2024-03-24T11:59:00.822687+0000","last_active":"2024-03-24T11:59:25.274755+0000","last_peered":"2024-03-24T11:59:25.274755+0000","last_clean":"2024-03-24T11:59:25.274755+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:59:25.274755+0000","last_undegraded":"2024-03-24T11:59:25.274755+0000","last_fullsized":"2024-03-24T11:59:25.274755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:59:00.822562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:00.822562+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:04.641919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"181'3757","reported_seq":3795,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.955803+0000","last_change":"2024-03-24T11:59:19.102975+0000","last_active":"2024-03-24T11:59:31.955803+0000","last_peered":"2024-03-24T11:59:31.955803+0000","last_clean":"2024-03-24T11:59:31.955803+0000","last_became_active":"2024-03-24T11:59:19.097645+0000","last_became_peered":"2024-03-24T11:59:19.097645+0000","last_unstale":"2024-03-24T11:59:31.955803+0000","last_undegraded":"2024-03-24T11:59:31.955803+0000","last_fullsized":"2024-03-24T11:59:31.955803+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:47.635477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246218752,"num_objects":3757,"num_object_clones":0,"num_object_copies":11271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3757,"num_write_kb":240448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":245,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.274804+0000","last_change":"2024-03-24T11:58:05.499256+0000","last_active":"2024-03-24T11:59:25.274804+0000","last_peered":"2024-03-24T11:59:25.274804+0000","last_clean":"2024-03-24T11:59:25.274804+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:59:25.274804+0000","last_undegraded":"2024-03-24T11:59:25.274804+0000","last_fullsized":"2024-03-24T11:59:25.274804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:05.498978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:05.498978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.262906+0000","last_change":"2024-03-24T11:58:10.853378+0000","last_active":"2024-03-24T11:59:25.262906+0000","last_peered":"2024-03-24T11:59:25.262906+0000","last_clean":"2024-03-24T11:59:25.262906+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:59:25.262906+0000","last_undegraded":"2024-03-24T11:59:25.262906+0000","last_fullsized":"2024-03-24T11:59:25.262906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:10.853268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:10.853268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":273,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.269567+0000","last_change":"2024-03-24T11:59:19.233121+0000","last_active":"2024-03-24T11:59:25.269567+0000","last_peered":"2024-03-24T11:59:25.269567+0000","last_clean":"2024-03-24T11:59:25.269567+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:59:25.269567+0000","last_undegraded":"2024-03-24T11:59:25.269567+0000","last_fullsized":"2024-03-24T11:59:25.269567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:19.233002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:59:19.233002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:29.859027+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":277,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275091+0000","last_change":"2024-03-24T11:59:05.755260+0000","last_active":"2024-03-24T11:59:25.275091+0000","last_peered":"2024-03-24T11:59:25.275091+0000","last_clean":"2024-03-24T11:59:25.275091+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:59:25.275091+0000","last_undegraded":"2024-03-24T11:59:25.275091+0000","last_fullsized":"2024-03-24T11:59:25.275091+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:59:05.755110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.755110+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:07.357503+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"181'3645","reported_seq":3677,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:31.194939+0000","last_change":"2024-03-24T11:58:42.972556+0000","last_active":"2024-03-24T11:59:31.194939+0000","last_peered":"2024-03-24T11:59:31.194939+0000","last_clean":"2024-03-24T11:59:31.194939+0000","last_became_active":"2024-03-24T11:58:42.972210+0000","last_became_peered":"2024-03-24T11:58:42.972210+0000","last_unstale":"2024-03-24T11:59:31.194939+0000","last_undegraded":"2024-03-24T11:59:31.194939+0000","last_fullsized":"2024-03-24T11:59:31.194939+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":3645,"log_dups_size":0,"ondisk_log_size":3645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:57.436309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238878720,"num_objects":3645,"num_object_clones":0,"num_object_copies":10935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3645,"num_write_kb":233280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":252,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264922+0000","last_change":"2024-03-24T11:57:58.742125+0000","last_active":"2024-03-24T11:59:25.264922+0000","last_peered":"2024-03-24T11:59:25.264922+0000","last_clean":"2024-03-24T11:59:25.264922+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:59:25.264922+0000","last_undegraded":"2024-03-24T11:59:25.264922+0000","last_fullsized":"2024-03-24T11:59:25.264922+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:58.741990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T11:57:58.741990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:25.518760+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":274,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.265653+0000","last_change":"2024-03-24T11:58:07.919554+0000","last_active":"2024-03-24T11:59:25.265653+0000","last_peered":"2024-03-24T11:59:25.265653+0000","last_clean":"2024-03-24T11:59:25.265653+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:59:25.265653+0000","last_undegraded":"2024-03-24T11:59:25.265653+0000","last_fullsized":"2024-03-24T11:59:25.265653+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.919445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.919445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:31.725508+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":285,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.264239+0000","last_change":"2024-03-24T11:57:56.751966+0000","last_active":"2024-03-24T11:59:25.264239+0000","last_peered":"2024-03-24T11:59:25.264239+0000","last_clean":"2024-03-24T11:59:25.264239+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:59:25.264239+0000","last_undegraded":"2024-03-24T11:59:25.264239+0000","last_fullsized":"2024-03-24T11:59:25.264239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:57:56.751874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:57:56.751874+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":362,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-24T11:59:25.275833+0000","last_change":"2024-03-24T11:59:08.941579+0000","last_active":"2024-03-24T11:59:25.275833+0000","last_peered":"2024-03-24T11:59:25.275833+0000","last_clean":"2024-03-24T11:59:25.275833+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:59:25.275833+0000","last_undegraded":"2024-03-24T11:59:25.275833+0000","last_fullsized":"2024-03-24T11:59:25.275833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T11:59:08.941456+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T11:59:08.941456+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.941456+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:13.592522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3914924032,"num_objects":59737,"num_object_clones":0,"num_object_copies":179211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59737,"num_write_kb":3823168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11713642496,"data_stored":11713642496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59737,"ondisk_log_size":59737,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005147,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1327432,"kb_used_data":1187968,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92417720,"statfs":{"total":95995035648,"available":94635745280,"internally_reserved":0,"allocated":1216479232,"data_stored":1216069787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142801354},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332194,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1337284,"kb_used_data":1175612,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92407868,"statfs":{"total":95995035648,"available":94625656832,"internally_reserved":0,"allocated":1203826688,"data_stored":1203421157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1832012,"kb_used_data":1656452,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91913140,"statfs":{"total":95995035648,"available":94119055360,"internally_reserved":0,"allocated":1696206848,"data_stored":1695792160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1695880,"kb_used_data":1439808,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92049272,"statfs":{"total":95995035648,"available":94258454528,"internally_reserved":0,"allocated":1474363392,"data_stored":1473952814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1610504,"kb_used_data":1416896,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92134648,"statfs":{"total":95995035648,"available":94345879552,"internally_reserved":0,"allocated":1450901504,"data_stored":1450491332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117104,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1675932,"kb_used_data":1419860,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92069220,"statfs":{"total":95995035648,"available":94278881280,"internally_reserved":0,"allocated":1453936640,"data_stored":1453506418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":262207943},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920875,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326928,"kb_used_data":1165256,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92418224,"statfs":{"total":95995035648,"available":94636261376,"internally_reserved":0,"allocated":1193222144,"data_stored":1192803395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659238,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2036360,"kb_used_data":1892736,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91708792,"statfs":{"total":95995035648,"available":93909803008,"internally_reserved":0,"allocated":1938161664,"data_stored":1937751219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1477640192,"data_stored":1477640192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1700724736,"data_stored":1700724736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1469120512,"data_stored":1469120512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1477771264,"data_stored":1477771264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1209139200,"data_stored":1209139200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1947664384,"data_stored":1947664384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1213267968,"data_stored":1213267968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1218314240,"data_stored":1218314240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:59:39.181 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T11:59:39.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-24T11:59:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:39.183 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:39.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:39.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:39.322 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:39.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:39.423 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:39.458 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:39.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:39.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:39.602 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:39.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:39.704 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:39.748 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:39.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:39.822 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 39 16 70425 70409 112.818 76 0.0129711 0.00885834 2024-03-24T11:59:39.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:39.889 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:39.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:39.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:39 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:39.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:40.033 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:40.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:40.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:39 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:40.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:39 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:40.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:40.173 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:40.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:40.275 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:40.308 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:40.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:40.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:40.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:40.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:40.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:40.579 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:40.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:40.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:40.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:40.746 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:40.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:40.789 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T11:59:40.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T11:59:40.826872+0000 min lat: 0.00128588 max lat: 1.02708 avg lat: 0.00894276 2024-03-24T11:59:40.830 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:59:40.830 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 40 16 71549 71533 111.744 70.25 0.018056 0.00894276 2024-03-24T11:59:40.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:40.901 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:40.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:40.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:41.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:41.049 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:41.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:41.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:40 smithi064 ceph-mon[42243]: pgmap v305: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:59:41.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:40 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3291759730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:41.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:40 smithi064 ceph-mon[42243]: from='client.25096 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:40 smithi064 ceph-mon[30223]: pgmap v305: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:59:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:40 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3291759730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:41.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:40 smithi064 ceph-mon[30223]: from='client.25096 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:41.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:40 smithi193 ceph-mon[33593]: pgmap v305: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:59:41.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:40 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3291759730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:41.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:40 smithi193 ceph-mon[33593]: from='client.25096 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:59:41.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:41.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:41.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:41.329 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:41.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:41.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:41.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:41.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:41.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:41.558 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:41.560 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:41.589 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:41.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:41.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:41.734 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:41.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:41.825 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 41 16 72734 72718 110.825 74.0625 0.0167936 0.00901809 2024-03-24T11:59:41.835 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:41.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:41.871 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:41.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:41.972 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:42.009 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:42.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:42.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: from='client.25096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:42.071 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:59:42.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:59:42.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:42.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: from='client.25096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:59:42.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:59:42.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:42.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:42.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:42.073 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:42.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:59:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:59:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: from='client.25096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:59:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:59:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:42.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:42.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:42.136 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:42.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:42.237 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:42.249 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:59:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:42.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:42.298 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:59:42.341 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:59:42.371 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:42.381 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:59:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:42.411 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:42.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:42.416 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:59:42.459 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:59:42.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:59:42.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:42.547 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:42.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:42.547 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:59:42.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:59:42.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:59:42.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:59:42.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:59:42.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:59:42.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:59:42.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:59:42.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:59:42.648 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:42.677 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:42.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:42.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:42.823 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:42.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:42.825 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 42 15 73744 73729 109.691 63.1875 0.00721694 0.00911179 2024-03-24T11:59:42.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:42.961 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:42.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:43.063 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:43.098 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:43.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:43.200 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:43.225 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:43.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:43 smithi064 ceph-mon[30223]: pgmap v307: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-24T11:59:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:43 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/764172380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:43 smithi064 ceph-mon[42243]: pgmap v307: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-24T11:59:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:43 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/764172380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:43.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:43 smithi193 ceph-mon[33593]: pgmap v307: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-24T11:59:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:43 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/764172380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:43.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:43.454 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:43.489 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:43.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:43.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:43.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:59:43.622 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:43.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:43.725 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:43.750 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:43.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:43.825 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 43 16 74668 74652 108.481 57.6875 0.00688004 0.00921315 2024-03-24T11:59:43.851 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:43.886 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:43.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:43.987 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:44.024 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:44.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:44.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:44.153 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:44.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:44.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:44.287 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:44.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:44.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:44.414 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:44.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:44.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:44.540 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:44.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:44.641 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:44.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:44.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:44.686 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:59:44.686 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:59:44.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:44.822 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:44.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:44.825 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 44 16 75513 75497 107.216 52.8125 0.00849409 0.00932181 2024-03-24T11:59:44.862 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:44.925 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:44.954 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:44.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:45.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:45.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:45.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:45 smithi064 ceph-mon[30223]: pgmap v308: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:59:45.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:45 smithi064 ceph-mon[42243]: pgmap v308: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:59:45.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:45.247 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:45.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:45 smithi193 ceph-mon[33593]: pgmap v308: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:59:45.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:45.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:45.461 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T11:59:45.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:45.523 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:45.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:45.625 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:45.670 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:45.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:45.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:45.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:45.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:59:45.810 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:45.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:45.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 45 16 76707 76691 106.492 74.625 0.015213 0.0093858 2024-03-24T11:59:45.848 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:59:45.912 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:45.953 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:45.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:46.033 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 1 pg_num to 11 2024-03-24T11:59:46.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:46.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:46.115 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:46.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:46.187 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:46 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/443742551' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:59:46.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:46 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/443742551' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:59:46.217 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:46.250 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:46.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:46.313 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:46 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/443742551' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:59:46.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:46.393 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:46.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:46.495 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:46.523 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:46.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:46.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:46.669 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:46.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:46.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:46.801 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:46.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:46.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 46 16 77802 77786 105.665 68.4375 0.0165377 0.00945863 2024-03-24T11:59:46.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:46.930 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:46.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:47.033 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:47.042 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:47 smithi064 ceph-mon[30223]: pgmap v309: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:59:47.042 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:47 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/443742551' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:59:47.042 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:47 smithi064 ceph-mon[30223]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:59:47.043 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:47 smithi064 ceph-mon[42243]: pgmap v309: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:59:47.043 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:47 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/443742551' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:59:47.043 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:47 smithi064 ceph-mon[42243]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:59:47.064 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:47.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:47.124 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:47 smithi193 ceph-mon[33593]: pgmap v309: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-24T11:59:47.124 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:47 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/443742551' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:59:47.124 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:47 smithi193 ceph-mon[33593]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:59:47.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:47.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:47.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:47.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:47.354 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:47.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:47.378 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:59:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:59:47.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:47.482 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:47.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:47.525 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:47.584 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:47.613 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:47.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:47.714 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:59:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:59:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:47.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:47.761 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:47.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:47.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 47 16 78829 78813 104.782 64.1875 0.00389307 0.00953803 2024-03-24T11:59:47.851 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:59:47.851 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:59:47.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:47.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:47.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:47.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:48.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:48.054 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:48.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:48.075 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T11:59:48.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:48.193 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:48.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:48.295 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:48.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:59:48.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:48 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1886582537' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:59:48.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:59:48.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:48 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1886582537' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:59:48.331 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:48.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:48.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:59:48.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:48 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1886582537' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:59:48.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:48.473 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:48.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:48.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:48.575 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:48.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T11:59:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:48.616 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:48.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:48.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:48.757 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:48.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:48.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 48 15 79964 79949 104.078 71 0.0100346 0.0096022 2024-03-24T11:59:48.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:48.906 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:48.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:49.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:49.035 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:59:49.047 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:49.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:49.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:49.187 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:49.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:49.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:49.298 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:49.298 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:49 smithi193 ceph-mon[33593]: pgmap v311: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:59:49.298 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:49 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:49.299 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:49 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:59:49.299 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:49 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1886582537' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:59:49.299 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:49 smithi193 ceph-mon[33593]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:59:49.299 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:49 smithi193 ceph-mon[33593]: 5.7 scrub starts 2024-03-24T11:59:49.299 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:49 smithi193 ceph-mon[33593]: 5.7 scrub ok 2024-03-24T11:59:49.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:49.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[30223]: pgmap v311: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:59:49.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1886582537' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[30223]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[30223]: 5.7 scrub starts 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[30223]: 5.7 scrub ok 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[42243]: pgmap v311: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:59:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:59:49.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1886582537' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:59:49.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[42243]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:59:49.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[42243]: 5.7 scrub starts 2024-03-24T11:59:49.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:49 smithi064 ceph-mon[42243]: 5.7 scrub ok 2024-03-24T11:59:49.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:49.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:49.427 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:49.464 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:49.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:49.566 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:49.601 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:49.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:49.703 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:49.737 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:49.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:49.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 49 16 81105 81089 103.408 71.25 0.00532732 0.00966474 2024-03-24T11:59:49.839 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:49.875 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:49.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:49.977 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:50.019 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:50.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:50.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:50.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:50.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:50.248 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:50.280 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:50.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:50.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: 8.7 scrub starts 2024-03-24T11:59:50.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T11:59:50.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T11:59:50.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T11:59:50.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T11:59:50.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:59:50.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T11:59:50.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T11:59:50.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:50 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:59:50.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: 8.7 scrub starts 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T11:59:50.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:59:50.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: 8.7 scrub starts 2024-03-24T11:59:50.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T11:59:50.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T11:59:50.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T11:59:50.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T11:59:50.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:59:50.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T11:59:50.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T11:59:50.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:50 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:59:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:50.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:50.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:50.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:50.574 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:50.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:50.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:50.712 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:50.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:50.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:50.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 50 16 82276 82260 102.804 73.1875 0.0121927 0.00972206 2024-03-24T11:59:50.853 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:50.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:50.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:50.992 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:50.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:51.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:51.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:51.140 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:51.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:51.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:51.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:51.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[30223]: pgmap v314: 158 pgs: 10 unknown, 148 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 854 op/s 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[30223]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[42243]: pgmap v314: 158 pgs: 10 unknown, 148 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 854 op/s 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T11:59:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T11:59:51.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T11:59:51.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T11:59:51.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:59:51.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:51 smithi064 ceph-mon[42243]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:59:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:51 smithi193 ceph-mon[33593]: pgmap v314: 158 pgs: 10 unknown, 148 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 854 op/s 2024-03-24T11:59:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:51 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T11:59:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:51 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T11:59:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:51 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T11:59:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:51 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T11:59:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:59:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:51 smithi193 ceph-mon[33593]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:59:51.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:51.424 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:51.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:51.526 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:51.571 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:51.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:51.577 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T11:59:51.577 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:59:51.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T11:59:51.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T11:59:51.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T11:59:51.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T11:59:51.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T11:59:51.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T11:59:51.633 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T11:59:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T11:59:51.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:51.675 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:59:51.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:59:51.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:59:51.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:59:51.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:59:51.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:59:51.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:59:51.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:59:51.713 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:51.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:51.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:51.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 51 15 83390 83375 102.155 69.6875 0.0236451 0.0097841 2024-03-24T11:59:51.848 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:51.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:51.950 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:59:51.950 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:59:51.950 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-03-24T11:59:51.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:59:51.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:52.001 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:52.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:52.102 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:52.140 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:52.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:52.242 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:52.276 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:52.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:52.282 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: 8.7 scrub ok 2024-03-24T11:59:52.282 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T11:59:52.282 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T11:59:52.282 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T11:59:52.282 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T11:59:52.282 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:59:52.282 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: pgmap v317: 158 pgs: 5 remapped+peering, 5 unknown, 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 590 op/s 2024-03-24T11:59:52.282 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1406656804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:52.283 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:52 smithi193 ceph-mon[33593]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:59:52.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: 8.7 scrub ok 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: pgmap v317: 158 pgs: 5 remapped+peering, 5 unknown, 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 590 op/s 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1406656804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[30223]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: 8.7 scrub ok 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T11:59:52.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T11:59:52.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T11:59:52.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T11:59:52.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:59:52.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: pgmap v317: 158 pgs: 5 remapped+peering, 5 unknown, 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 590 op/s 2024-03-24T11:59:52.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1406656804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:59:52.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:52 smithi064 ceph-mon[42243]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:59:52.326 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T11:59:52.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:52.440 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:52.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:52.541 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:52.583 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:52.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:52.602 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:52.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:52.726 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:52.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:52.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:59:52.792 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:59:52.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 52 16 84525 84509 101.553 70.875 0.0141885 0.00984148 2024-03-24T11:59:52.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:52.863 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:52.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:52.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:52.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:53.012 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:53.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:53.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:53.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:53.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:53.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:53.281 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:53 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T11:59:53.281 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:53 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T11:59:53.281 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:53 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T11:59:53.281 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:53 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T11:59:53.281 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:53 smithi193 ceph-mon[33593]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-24T11:59:53.281 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:53 smithi193 ceph-mon[33593]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:59:53.310 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:53.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:53.412 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:53.444 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:53.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:53.546 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:53.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[42243]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[42243]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[30223]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-24T11:59:53.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:53 smithi064 ceph-mon[30223]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:59:53.577 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:53.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:53.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:53.722 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:53.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:53.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:53.827 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 53 16 85743 85727 101.073 76.125 0.0123311 0.00988942 2024-03-24T11:59:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:53.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:53.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:53.993 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:53.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:54.098 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:54.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:54.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:54.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:54.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:54.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:54.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:54 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T11:59:54.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:54 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T11:59:54.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:54 smithi193 ceph-mon[33593]: pgmap v320: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 remapped+peering, 3 active+undersized+remapped, 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 3/249906 objects degraded (0.001%); 2/249906 objects misplaced (0.001%) 2024-03-24T11:59:54.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:54 smithi193 ceph-mon[33593]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:59:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:54.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:54.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:59:54.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:54.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:54 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T11:59:54.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:54 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T11:59:54.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:54 smithi064 ceph-mon[42243]: pgmap v320: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 remapped+peering, 3 active+undersized+remapped, 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 3/249906 objects degraded (0.001%); 2/249906 objects misplaced (0.001%) 2024-03-24T11:59:54.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:54 smithi064 ceph-mon[42243]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:59:54.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:54 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T11:59:54.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:54 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T11:59:54.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:54 smithi064 ceph-mon[30223]: pgmap v320: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 remapped+peering, 3 active+undersized+remapped, 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 3/249906 objects degraded (0.001%); 2/249906 objects misplaced (0.001%) 2024-03-24T11:59:54.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:54 smithi064 ceph-mon[30223]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:59:54.570 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:54.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:54.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:54.707 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:54.741 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:54.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:54.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 54 16 86907 86891 100.549 72.75 0.0124378 0.00994077 2024-03-24T11:59:54.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:54.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:54.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:54.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:55.024 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:55.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:55.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:55.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:55.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:55.266 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:55.299 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:55.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:55.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:55.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:55.429 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:55.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:55.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:55.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T11:59:55.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: 1.a scrub starts 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: 1.a scrub ok 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: Health check failed: Degraded data redundancy: 3/249906 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: 1.9 scrub starts 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: 1.9 scrub ok 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: 1.a scrub starts 2024-03-24T11:59:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: 1.a scrub ok 2024-03-24T11:59:55.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: Health check failed: Degraded data redundancy: 3/249906 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:55.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T11:59:55.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: 1.9 scrub starts 2024-03-24T11:59:55.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: 1.9 scrub ok 2024-03-24T11:59:55.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:55 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T11:59:55.567 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:55.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: 1.a scrub starts 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: 1.a scrub ok 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: Health check failed: Degraded data redundancy: 3/249906 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: 1.9 scrub starts 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: 1.9 scrub ok 2024-03-24T11:59:55.601 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:55 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T11:59:55.611 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24379.0:926 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94338803872128 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:52:36.783837+0000", 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 438.82546920200002, 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.016555906, 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24379", 2024-03-24T11:59:55.612 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 926 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:36.783837+0000", 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:36.783837+0000", 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:36.783841+0000", 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1499999999999999e-06 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:36.783848+0000", 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3370000000000002e-06 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:36.783866+0000", 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8233999999999999e-05 2024-03-24T11:59:55.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:36.783875+0000", 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2629999999999992e-06 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:36.786561+0000", 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026857349999999999 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:36.786571+0000", 2024-03-24T11:59:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9369999999999999e-06 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:37.791685+0000", 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.005114217 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for missing object", 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:37.791842+0000", 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000156651 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:37.794446+0000", 2024-03-24T11:59:55.616 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002604145 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:37.794518+0000", 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2329000000000005e-05 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:37.794906+0000", 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000387814 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:37.794962+0000", 2024-03-24T11:59:55.617 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5911000000000001e-05 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:52:37.800393+0000", 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.005431183 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28171 7.0 7:09019b55:::benchmark_data_smithi064_87013_object28170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.179233+0000", 2024-03-24T11:59:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 308.43007417299998, 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16348571100000001, 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28171 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.179233+0000", 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.179233+0000", 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.179233+0000", 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1999999999999999e-07 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.179264+0000", 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1130999999999997e-05 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.179272+0000", 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7449999999999995e-06 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.179278+0000", 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9959999999999999e-06 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.181185+0000", 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001906732 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.181292+0000", 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107447 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.181417+0000", 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000124578 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.182444+0000", 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001027041 2024-03-24T11:59:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.184130+0000", 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016864709999999999 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.342595+0000", 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.158464253 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.342670+0000", 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4950999999999994e-05 2024-03-24T11:59:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.342718+0000", 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8646000000000001e-05 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28194 7.0 7:083e1c7e:::benchmark_data_smithi064_87013_object28193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.188834+0000", 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 308.42047290099998, 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15409931900000001, 2024-03-24T11:59:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T11:59:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T11:59:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28194 2024-03-24T11:59:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188834+0000", 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188834+0000", 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188836+0000", 2024-03-24T11:59:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8250000000000001e-06 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188994+0000", 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000157956 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.189006+0000", 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2276000000000001e-05 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.189011+0000", 2024-03-24T11:59:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5260000000000001e-06 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.189062+0000", 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0433999999999999e-05 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.189180+0000", 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000118447 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:59:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.189270+0000", 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9371999999999997e-05 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.190214+0000", 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00094480300000000003 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.190601+0000", 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000386757 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.342776+0000", 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15217497799999999 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.342886+0000", 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011025499999999999 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.342933+0000", 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6690000000000002e-05 2024-03-24T11:59:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:34137 8.8 8:1cf1448d:::benchmark_data_smithi064_120708_object34136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:14.765597+0000", 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 40.843709836999999, 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16523170500000001, 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 34137 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765597+0000", 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765597+0000", 2024-03-24T11:59:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765598+0000", 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5200000000000001e-06 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765655+0000", 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6402e-05 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765666+0000", 2024-03-24T11:59:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1191e-05 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765674+0000", 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.763e-06 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765725+0000", 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0933000000000002e-05 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765816+0000", 2024-03-24T11:59:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1565999999999997e-05 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.765899+0000", 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2738999999999995e-05 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.767864+0000", 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0019649709999999998 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.769702+0000", 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018376829999999999 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.930712+0000", 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16101064100000001 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.930774+0000", 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1588999999999996e-05 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.930829+0000", 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4707000000000003e-05 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:34141 8.8 8:1620effa:::benchmark_data_smithi064_120708_object34140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:14.767739+0000", 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 40.84156806, 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16801923599999999, 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 34141 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.767739+0000", 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.767739+0000", 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.767740+0000", 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4300000000000001e-06 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.767777+0000", 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7064000000000003e-05 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.767795+0000", 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7734e-05 2024-03-24T11:59:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.767802+0000", 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.435e-06 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.767894+0000", 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1266000000000003e-05 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.768022+0000", 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012866599999999999 2024-03-24T11:59:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.768140+0000", 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000117768 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.770023+0000", 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001883476 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.770129+0000", 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000105802 2024-03-24T11:59:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.935659+0000", 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16552958000000001 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.935724+0000", 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4659000000000006e-05 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:14.935758+0000", 2024-03-24T11:59:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4356000000000001e-05 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:35792 8.8 8:17c285b4:::benchmark_data_smithi064_120708_object35791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:15.840956+0000", 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 39.768350323999996, 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15408485199999999, 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 35792 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.840956+0000", 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.840956+0000", 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.840957+0000", 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5799999999999998e-07 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.840990+0000", 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2660000000000002e-05 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.840999+0000", 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5109999999999992e-06 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.841007+0000", 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3100000000000003e-06 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.841059+0000", 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2101999999999997e-05 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.841141+0000", 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2089999999999995e-05 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.841270+0000", 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000128758 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.843584+0000", 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023147559999999998 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844129+0000", 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00054450600000000005 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.994977+0000", 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15084798999999999 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995013+0000", 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6131999999999999e-05 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995041+0000", 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8178999999999999e-05 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:35802 8.8 8:184b8d4d:::benchmark_data_smithi064_120708_object35801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:15.844266+0000", 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 39.765040270999997, 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15092953200000001, 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 35802 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844266+0000", 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844266+0000", 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844268+0000", 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7659999999999999e-06 2024-03-24T11:59:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844312+0000", 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3784e-05 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844326+0000", 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3689000000000001e-05 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844347+0000", 2024-03-24T11:59:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1319999999999999e-05 2024-03-24T11:59:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844407+0000", 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0041000000000003e-05 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844530+0000", 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012334600000000001 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.844655+0000", 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000124141 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.847992+0000", 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0033378090000000002 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.848287+0000", 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00029469300000000001 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995129+0000", 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14684191399999999 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995172+0000", 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2648000000000002e-05 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995196+0000", 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4380999999999999e-05 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:35815 8.8 8:13d3f657:::benchmark_data_smithi064_120708_object35814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:15.851761+0000", 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 39.757545831999998, 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14352247400000001, 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 35815 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851761+0000", 2024-03-24T11:59:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851761+0000", 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851762+0000", 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1030000000000001e-06 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851824+0000", 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2444e-05 2024-03-24T11:59:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851833+0000", 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1839999999999999e-06 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851838+0000", 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1880000000000001e-06 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851878+0000", 2024-03-24T11:59:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0472000000000001e-05 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851974+0000", 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6000000000000002e-05 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852180+0000", 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000205885 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.662 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.854355+0000", 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002175265 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.854702+0000", 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00034682499999999999 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995232+0000", 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.140529717 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995260+0000", 2024-03-24T11:59:55.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8201999999999999e-05 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995283+0000", 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3189000000000001e-05 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:35817 8.8 8:1efb2d01:::benchmark_data_smithi064_120708_object35816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:15.852045+0000", 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 39.757261978000003, 2024-03-24T11:59:55.664 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.143331337, 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 35817 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852045+0000", 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852045+0000", 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852046+0000", 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1030000000000001e-06 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852111+0000", 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5667000000000007e-05 2024-03-24T11:59:55.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852120+0000", 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0530000000000007e-06 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852130+0000", 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0888e-05 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.853274+0000", 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0011433470000000001 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.853370+0000", 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6606000000000003e-05 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.853522+0000", 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0001515 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.854478+0000", 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000956273 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.854814+0000", 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033581000000000003 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995309+0000", 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.140495496 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995347+0000", 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7905999999999999e-05 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.669 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.995376+0000", 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8688e-05 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35033.0:72990 8.a e182/178 8:5a5d6a33:::benchmark_data_smithi064_120708_object72989:head v 182'4645, mlcod=182'4643)", 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.014751+0000", 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.594555565, 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14555330799999999, 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.014751+0000", 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.014751+0000", 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.014752+0000", 2024-03-24T11:59:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2399999999999998e-07 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.014798+0000", 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6143000000000002e-05 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.014805+0000", 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8859999999999998e-06 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.014811+0000", 2024-03-24T11:59:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.823e-06 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.014954+0000", 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000143733 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.014972+0000", 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.783e-05 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160259+0000", 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14528725100000001 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160304+0000", 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5018000000000002e-05 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35033.0:72995 8.2 e182/170 8:402b8c35:::benchmark_data_smithi064_120708_object72994:head v 182'4634, mlcod=182'4632)", 2024-03-24T11:59:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.018048+0000", 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.591258781000001, 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14540161800000001, 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018048+0000", 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018048+0000", 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018049+0000", 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3090000000000001e-06 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018075+0000", 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5364999999999999e-05 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018085+0000", 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0025e-05 2024-03-24T11:59:55.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018092+0000", 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9999999999999999e-06 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018158+0000", 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6801e-05 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018177+0000", 2024-03-24T11:59:55.677 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8393000000000002e-05 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163404+0000", 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14522712199999999 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163450+0000", 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5602999999999997e-05 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.678 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35033.0:72992 8.a e182/178 8:558bd69d:::benchmark_data_smithi064_120708_object72991:head v 182'4646, mlcod=182'4643)", 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.018507+0000", 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.590799651999999, 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14498569, 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018507+0000", 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.679 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018507+0000", 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018507+0000", 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1800000000000001e-07 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018539+0000", 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1118999999999998e-05 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.680 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018546+0000", 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9770000000000003e-06 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018550+0000", 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8740000000000003e-06 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019503+0000", 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00095284500000000002 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.681 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019521+0000", 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8221000000000001e-05 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163453+0000", 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.143931315 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163493+0000", 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9920999999999999e-05 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.682 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:72996 8.8 8:1c19f408:::benchmark_data_smithi064_120708_object72995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.018982+0000", 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.590324429000001, 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14122016200000001, 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 72996 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.683 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018982+0000", 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018982+0000", 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.018984+0000", 2024-03-24T11:59:55.684 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.305e-06 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019011+0000", 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7192999999999999e-05 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019018+0000", 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3339999999999996e-06 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.685 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019024+0000", 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8649999999999998e-06 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019126+0000", 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000101591 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019252+0000", 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000126718 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.686 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019394+0000", 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014207299999999999 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022670+0000", 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032753140000000001 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024009+0000", 2024-03-24T11:59:55.687 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013388880000000001 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026832+0000", 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002823819 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160146+0000", 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13331358700000001 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.688 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160202+0000", 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6474999999999997e-05 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35033.0:73001 8.a e182/178 8:54eaffae:::benchmark_data_smithi064_120708_object73000:head v 182'4647, mlcod=182'4643)", 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.021526+0000", 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.587780558, 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14201475, 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.689 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.021526+0000", 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.021526+0000", 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.021527+0000", 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1969999999999999e-06 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.690 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.021559+0000", 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.171e-05 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.021569+0000", 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6090000000000007e-06 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.021576+0000", 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4510000000000003e-06 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.691 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024920+0000", 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003344082 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024938+0000", 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7731e-05 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163520+0000", 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13858166499999999 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.692 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163541+0000", 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1305e-05 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35033.0:72999 8.2 e182/170 8:46ada980:::benchmark_data_smithi064_120708_object72998:head v 182'4635, mlcod=182'4632)", 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.022326+0000", 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.586980198999999, 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14119126500000001, 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.693 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022326+0000", 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022326+0000", 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022328+0000", 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.291e-06 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.694 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.695 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:55.696 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.696 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022355+0000", 2024-03-24T11:59:55.696 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7684000000000001e-05 2024-03-24T11:59:55.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.696 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.696 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.696 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022366+0000", 2024-03-24T11:59:55.696 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0190000000000001e-05 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022373+0000", 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2180000000000002e-06 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024798+0000", 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024254200000000002 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.697 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024817+0000", 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9049999999999999e-05 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163495+0000", 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.138677569 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163518+0000", 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2843000000000001e-05 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35033.0:73003 8.0 e182/170 8:0035f76e:::benchmark_data_smithi064_120708_object73002:head v 182'4460, mlcod=182'4457)", 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.022773+0000", 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.586534160999999, 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13711520299999999, 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022773+0000", 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.699 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022773+0000", 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022774+0000", 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0100000000000001e-06 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022817+0000", 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3136000000000002e-05 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.700 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022826+0000", 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8580000000000006e-06 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022832+0000", 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4489999999999996e-06 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022867+0000", 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5238999999999999e-05 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.701 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.022884+0000", 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7037e-05 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026826+0000", 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003941567 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.159888+0000", 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13306190700000001 2024-03-24T11:59:55.702 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:73006 8.3 8:cefef0d5:::benchmark_data_smithi064_120708_object73005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.023759+0000", 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.585548053, 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13689489299999999, 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 73006 2024-03-24T11:59:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023759+0000", 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023759+0000", 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023761+0000", 2024-03-24T11:59:55.704 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8780000000000001e-06 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023810+0000", 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9060000000000001e-05 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023821+0000", 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1415e-05 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.705 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023829+0000", 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4579999999999993e-06 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023886+0000", 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6195e-05 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024013+0000", 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012767799999999999 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:59:55.706 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024155+0000", 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014138000000000001 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160382+0000", 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13622708 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160424+0000", 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2268000000000003e-05 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.707 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160554+0000", 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012989100000000001 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160608+0000", 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3677999999999997e-05 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160654+0000", 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5911999999999997e-05 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.708 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:73007 8.d 8:b36e4137:::benchmark_data_smithi064_120708_object73006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.023855+0000", 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.585451614, 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13690118000000001, 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 73007 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.709 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023855+0000", 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023855+0000", 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.710 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023856+0000", 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9899999999999999e-07 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023942+0000", 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6598e-05 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023950+0000", 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4599999999999997e-06 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.711 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023956+0000", 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8130000000000001e-06 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024009+0000", 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3028000000000001e-05 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024087+0000", 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8467999999999995e-05 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.712 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.024212+0000", 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012490799999999999 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026827+0000", 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026150919999999999 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160553+0000", 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13372542900000001 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.713 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160666+0000", 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000113565 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160717+0000", 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0930000000000002e-05 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160756+0000", 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.909e-05 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.714 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35033.0:73008 8.a e182/178 8:545958b9:::benchmark_data_smithi064_120708_object73007:head v 182'4648, mlcod=182'4643)", 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.025198+0000", 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.584108871, 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14032218599999999, 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.025198+0000", 2024-03-24T11:59:55.715 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.025198+0000", 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.025199+0000", 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.226e-06 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.716 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.025232+0000", 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2614999999999999e-05 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.025240+0000", 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4200000000000007e-06 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.025247+0000", 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4330000000000001e-06 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.717 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160553+0000", 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13530642700000001 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160572+0000", 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9298000000000001e-05 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.165490+0000", 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0049180250000000003 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.718 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.165520+0000", 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9742000000000001e-05 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop_reply(client.35033.0:72996 8.8 e182/170 ondisk, result = 0)", 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.026012+0000", 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.583294355, 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.134199648, 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.719 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026012+0000", 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026012+0000", 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026014+0000", 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2330000000000001e-06 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T11:59:55.720 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026018+0000", 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8489999999999997e-06 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026028+0000", 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3749999999999992e-06 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026035+0000", 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7410000000000003e-06 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026820+0000", 2024-03-24T11:59:55.721 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00078532299999999999 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026827+0000", 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0999999999999998e-06 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.160212+0000", 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13338502699999999 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T11:59:55.722 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T11:59:55.723 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:59:55.746 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:55.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:55.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 55 16 88037 88021 100.005 70.625 0.00468303 0.00999489 2024-03-24T11:59:55.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:55.878 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:55.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:55.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:56.025 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:56.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:56.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:56.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:56.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:56.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:56.305 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:56.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.4 scrub starts 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.4 scrub ok 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.5 scrub starts 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.5 scrub ok 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.7 scrub starts 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.7 scrub ok 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.1 scrub starts 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.1 scrub ok 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:59:56.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: pgmap v322: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 remapped+peering, 3 active+undersized+remapped, 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 3/249906 objects degraded (0.001%); 2/249906 objects misplaced (0.001%) 2024-03-24T11:59:56.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:59:56.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:56 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:59:56.407 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:56.441 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:56.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:56.486 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T11:59:56.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.4 scrub starts 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.4 scrub ok 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.5 scrub starts 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.5 scrub ok 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.7 scrub starts 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.7 scrub ok 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.1 scrub starts 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.1 scrub ok 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: pgmap v322: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 remapped+peering, 3 active+undersized+remapped, 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 3/249906 objects degraded (0.001%); 2/249906 objects misplaced (0.001%) 2024-03-24T11:59:56.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.4 scrub starts 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.4 scrub ok 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.5 scrub starts 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.5 scrub ok 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.7 scrub starts 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.7 scrub ok 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.1 scrub starts 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.1 scrub ok 2024-03-24T11:59:56.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T11:59:56.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T11:59:56.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:59:56.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:59:56.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: pgmap v322: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 remapped+peering, 3 active+undersized+remapped, 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 3/249906 objects degraded (0.001%); 2/249906 objects misplaced (0.001%) 2024-03-24T11:59:56.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:59:56.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:56 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:59:56.498 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T11:59:56.533 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T11:59:56.542 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:56.621 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:56.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:56.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:56.761 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:56.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:59:56.826 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 56 16 89166 89150 99.4788 70.5625 0.0196273 0.0100475 2024-03-24T11:59:56.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:56.915 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:56.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:56.927 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:59:57.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:57.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:57.064 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:57.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:57.165 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:57.198 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:57.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:57.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:57.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: 1.6 scrub starts 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: 1.6 scrub ok 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: 1.8 scrub starts 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: 1.8 scrub ok 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: from='client.15249 v1:172.21.15.64:0/2654016741' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:59:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:57 smithi193 ceph-mon[33593]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:59:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:57.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T11:59:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T11:59:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: 1.6 scrub starts 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: 1.6 scrub ok 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: 1.8 scrub starts 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: 1.8 scrub ok 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: from='client.15249 v1:172.21.15.64:0/2654016741' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[30223]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: 1.6 scrub starts 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: 1.6 scrub ok 2024-03-24T11:59:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: 1.8 scrub starts 2024-03-24T11:59:57.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: 1.8 scrub ok 2024-03-24T11:59:57.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:59:57.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T11:59:57.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:57.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: from='client.15249 v1:172.21.15.64:0/2654016741' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:59:57.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-mon[42243]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:59:57.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:57.477 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:57.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:57.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:57.620 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:57.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:57.691 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 11:59:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:11:59:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:57.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:57.778 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:57.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:57.789 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:59:57.830 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 57 16 90372 90356 99.0524 75.375 0.00905753 0.0100907 2024-03-24T11:59:57.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:57.929 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:57.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:58.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:58.068 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:58.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:58.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:58 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:58.142 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:58 smithi193 ceph-mon[33593]: pgmap v325: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 80 MiB/s wr, 1.25k op/s 2024-03-24T11:59:58.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:58.212 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:58.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:58.295 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-03-24T11:59:56.128850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5446846393,"num_objects":83302,"num_object_clones":0,"num_object_copies":249906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":83302,"num_whiteouts":0,"num_read":936,"num_read_kb":776,"num_write":83703,"num_write_kb":5320898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1611291,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80388,"ondisk_log_size":80388,"up":474,"acting":464,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17478240,"kb_used_data":15865440,"kb_used_omap":12,"kb_used_meta":1612659,"kb_avail":732482976,"statfs":{"total":767960285184,"available":750062567424,"internally_reserved":0,"allocated":16246210560,"data_stored":16242681088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1651363376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":205199421,"num_objects":3128,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":95,"num_read_kb":139,"num_write":3208,"num_write_kb":201226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":805048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.867137"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.752787+0000","last_change":"2024-03-24T11:59:02.144257+0000","last_active":"2024-03-24T11:59:51.752787+0000","last_peered":"2024-03-24T11:59:51.752787+0000","last_clean":"2024-03-24T11:59:51.752787+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T11:59:51.752787+0000","last_undegraded":"2024-03-24T11:59:51.752787+0000","last_fullsized":"2024-03-24T11:59:51.752787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:02.144154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:02.144154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:12.729914+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.095238+0000","last_change":"2024-03-24T11:59:17.681747+0000","last_active":"2024-03-24T11:59:51.095238+0000","last_peered":"2024-03-24T11:59:51.095238+0000","last_clean":"2024-03-24T11:59:51.095238+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T11:59:51.095238+0000","last_undegraded":"2024-03-24T11:59:51.095238+0000","last_fullsized":"2024-03-24T11:59:51.095238+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:17.681628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:59:17.681628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.437272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":279,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754540+0000","last_change":"2024-03-24T11:59:50.555909+0000","last_active":"2024-03-24T11:59:51.754540+0000","last_peered":"2024-03-24T11:59:51.754540+0000","last_clean":"2024-03-24T11:59:51.754540+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T11:59:51.754540+0000","last_undegraded":"2024-03-24T11:59:51.754540+0000","last_fullsized":"2024-03-24T11:59:51.754540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:50.555718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T11:59:50.555718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:52.285420+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"117'22","reported_seq":308,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751584+0000","last_change":"2024-03-24T11:58:12.849968+0000","last_active":"2024-03-24T11:59:51.751584+0000","last_peered":"2024-03-24T11:59:51.751584+0000","last_clean":"2024-03-24T11:59:51.751584+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T11:59:51.751584+0000","last_undegraded":"2024-03-24T11:59:51.751584+0000","last_fullsized":"2024-03-24T11:59:51.751584+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'22","last_scrub_stamp":"2024-03-24T11:58:12.849803+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T11:58:12.849803+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.096925+0000","last_change":"2024-03-24T11:58:58.872999+0000","last_active":"2024-03-24T11:59:51.096925+0000","last_peered":"2024-03-24T11:59:51.096925+0000","last_clean":"2024-03-24T11:59:51.096925+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T11:59:51.096925+0000","last_undegraded":"2024-03-24T11:59:51.096925+0000","last_fullsized":"2024-03-24T11:59:51.096925+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:58.872877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:58.872877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:12.680882+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":29,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.098959+0000","last_change":"2024-03-24T11:59:19.117600+0000","last_active":"2024-03-24T11:59:51.098959+0000","last_peered":"2024-03-24T11:59:51.098959+0000","last_clean":"2024-03-24T11:59:51.098959+0000","last_became_active":"2024-03-24T11:59:19.117244+0000","last_became_peered":"2024-03-24T11:59:19.117244+0000","last_unstale":"2024-03-24T11:59:51.098959+0000","last_undegraded":"2024-03-24T11:59:51.098959+0000","last_fullsized":"2024-03-24T11:59:51.098959+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:01.340280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:01.340280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:21.473542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":280,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751248+0000","last_change":"2024-03-24T11:58:07.728708+0000","last_active":"2024-03-24T11:59:51.751248+0000","last_peered":"2024-03-24T11:59:51.751248+0000","last_clean":"2024-03-24T11:59:51.751248+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T11:59:51.751248+0000","last_undegraded":"2024-03-24T11:59:51.751248+0000","last_fullsized":"2024-03-24T11:59:51.751248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.728598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.728598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"176'13","reported_seq":300,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.094928+0000","last_change":"2024-03-24T11:59:14.935407+0000","last_active":"2024-03-24T11:59:51.094928+0000","last_peered":"2024-03-24T11:59:51.094928+0000","last_clean":"2024-03-24T11:59:51.094928+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T11:59:51.094928+0000","last_undegraded":"2024-03-24T11:59:51.094928+0000","last_fullsized":"2024-03-24T11:59:51.094928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'13","last_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.935318+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:32.358216+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.097949+0000","last_change":"2024-03-24T11:58:12.711969+0000","last_active":"2024-03-24T11:59:51.097949+0000","last_peered":"2024-03-24T11:59:51.097949+0000","last_clean":"2024-03-24T11:59:51.097949+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T11:59:51.097949+0000","last_undegraded":"2024-03-24T11:59:51.097949+0000","last_fullsized":"2024-03-24T11:59:51.097949+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:12.711859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T11:58:12.711859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":277,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.103673+0000","last_change":"2024-03-24T11:59:17.067826+0000","last_active":"2024-03-24T11:59:51.103673+0000","last_peered":"2024-03-24T11:59:51.103673+0000","last_clean":"2024-03-24T11:59:51.103673+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T11:59:51.103673+0000","last_undegraded":"2024-03-24T11:59:51.103673+0000","last_fullsized":"2024-03-24T11:59:51.103673+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T11:59:17.067728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:32.451199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105772+0000","last_change":"2024-03-24T11:58:06.122994+0000","last_active":"2024-03-24T11:59:51.105772+0000","last_peered":"2024-03-24T11:59:51.105772+0000","last_clean":"2024-03-24T11:59:51.105772+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T11:59:51.105772+0000","last_undegraded":"2024-03-24T11:59:51.105772+0000","last_fullsized":"2024-03-24T11:59:51.105772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:06.122853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:23.689192+0000","last_clean_scrub_stamp":"2024-03-24T11:58:06.122853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":292,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.097091+0000","last_change":"2024-03-24T11:58:53.841225+0000","last_active":"2024-03-24T11:59:51.097091+0000","last_peered":"2024-03-24T11:59:51.097091+0000","last_clean":"2024-03-24T11:59:51.097091+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T11:59:51.097091+0000","last_undegraded":"2024-03-24T11:59:51.097091+0000","last_fullsized":"2024-03-24T11:59:51.097091+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T11:58:53.841123+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T11:58:53.841123+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:02.782949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":259,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754846+0000","last_change":"2024-03-24T11:58:57.644377+0000","last_active":"2024-03-24T11:59:51.754846+0000","last_peered":"2024-03-24T11:59:51.754846+0000","last_clean":"2024-03-24T11:59:51.754846+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T11:59:51.754846+0000","last_undegraded":"2024-03-24T11:59:51.754846+0000","last_fullsized":"2024-03-24T11:59:51.754846+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:57.644266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.644266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:58.789635+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":268,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.098390+0000","last_change":"2024-03-24T11:58:03.766621+0000","last_active":"2024-03-24T11:59:51.098390+0000","last_peered":"2024-03-24T11:59:51.098390+0000","last_clean":"2024-03-24T11:59:51.098390+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T11:59:51.098390+0000","last_undegraded":"2024-03-24T11:59:51.098390+0000","last_fullsized":"2024-03-24T11:59:51.098390+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:03.766504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:03.766504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":288,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105252+0000","last_change":"2024-03-24T11:58:57.909335+0000","last_active":"2024-03-24T11:59:51.105252+0000","last_peered":"2024-03-24T11:59:51.105252+0000","last_clean":"2024-03-24T11:59:51.105252+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T11:59:51.105252+0000","last_undegraded":"2024-03-24T11:59:51.105252+0000","last_fullsized":"2024-03-24T11:59:51.105252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:57.909246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.909246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:23.726088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.095405+0000","last_change":"2024-03-24T11:58:00.717857+0000","last_active":"2024-03-24T11:59:51.095405+0000","last_peered":"2024-03-24T11:59:51.095405+0000","last_clean":"2024-03-24T11:59:51.095405+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T11:59:51.095405+0000","last_undegraded":"2024-03-24T11:59:51.095405+0000","last_fullsized":"2024-03-24T11:59:51.095405+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:58:00.717763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:58:00.717763+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":276,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751640+0000","last_change":"2024-03-24T11:59:21.206602+0000","last_active":"2024-03-24T11:59:51.751640+0000","last_peered":"2024-03-24T11:59:51.751640+0000","last_clean":"2024-03-24T11:59:51.751640+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T11:59:51.751640+0000","last_undegraded":"2024-03-24T11:59:51.751640+0000","last_fullsized":"2024-03-24T11:59:51.751640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.206482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.206482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:42.128353+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":276,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.095745+0000","last_change":"2024-03-24T11:59:22.719982+0000","last_active":"2024-03-24T11:59:51.095745+0000","last_peered":"2024-03-24T11:59:51.095745+0000","last_clean":"2024-03-24T11:59:51.095745+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T11:59:51.095745+0000","last_undegraded":"2024-03-24T11:59:51.095745+0000","last_fullsized":"2024-03-24T11:59:51.095745+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.719851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.719851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.894655+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":286,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.752956+0000","last_change":"2024-03-24T11:57:56.720710+0000","last_active":"2024-03-24T11:59:51.752956+0000","last_peered":"2024-03-24T11:59:51.752956+0000","last_clean":"2024-03-24T11:59:51.752956+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T11:59:51.752956+0000","last_undegraded":"2024-03-24T11:59:51.752956+0000","last_fullsized":"2024-03-24T11:59:51.752956+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:57:56.720566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:57:56.720566+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":322,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751217+0000","last_change":"2024-03-24T11:58:53.063331+0000","last_active":"2024-03-24T11:59:51.751217+0000","last_peered":"2024-03-24T11:59:51.751217+0000","last_clean":"2024-03-24T11:59:51.751217+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T11:59:51.751217+0000","last_undegraded":"2024-03-24T11:59:51.751217+0000","last_fullsized":"2024-03-24T11:59:51.751217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:58:53.063227+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:58:53.063227+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:16.650747+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.098087+0000","last_change":"2024-03-24T11:58:57.872490+0000","last_active":"2024-03-24T11:59:51.098087+0000","last_peered":"2024-03-24T11:59:51.098087+0000","last_clean":"2024-03-24T11:59:51.098087+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T11:59:51.098087+0000","last_undegraded":"2024-03-24T11:59:51.098087+0000","last_fullsized":"2024-03-24T11:59:51.098087+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:57.872392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.872392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:25.514528+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":37,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.097371+0000","last_change":"2024-03-24T11:59:21.674710+0000","last_active":"2024-03-24T11:59:51.097371+0000","last_peered":"2024-03-24T11:59:51.097371+0000","last_clean":"2024-03-24T11:59:51.097371+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T11:59:51.097371+0000","last_undegraded":"2024-03-24T11:59:51.097371+0000","last_fullsized":"2024-03-24T11:59:51.097371+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.674636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.674636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:27.485678+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":291,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101428+0000","last_change":"2024-03-24T11:59:20.890264+0000","last_active":"2024-03-24T11:59:51.101428+0000","last_peered":"2024-03-24T11:59:51.101428+0000","last_clean":"2024-03-24T11:59:51.101428+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T11:59:51.101428+0000","last_undegraded":"2024-03-24T11:59:51.101428+0000","last_fullsized":"2024-03-24T11:59:51.101428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_clean_scrub_stamp":"2024-03-24T11:59:20.890106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:48.280238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"115'22","reported_seq":314,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.110087+0000","last_change":"2024-03-24T11:58:55.811062+0000","last_active":"2024-03-24T11:59:51.110087+0000","last_peered":"2024-03-24T11:59:51.110087+0000","last_clean":"2024-03-24T11:59:51.110087+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T11:59:51.110087+0000","last_undegraded":"2024-03-24T11:59:51.110087+0000","last_fullsized":"2024-03-24T11:59:51.110087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'22","last_scrub_stamp":"2024-03-24T11:58:55.810966+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T11:58:55.810966+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:00.810200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":28,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.100561+0000","last_change":"2024-03-24T11:59:19.099058+0000","last_active":"2024-03-24T11:59:51.100561+0000","last_peered":"2024-03-24T11:59:51.100561+0000","last_clean":"2024-03-24T11:59:51.100561+0000","last_became_active":"2024-03-24T11:59:19.098557+0000","last_became_peered":"2024-03-24T11:59:19.098557+0000","last_unstale":"2024-03-24T11:59:51.100561+0000","last_undegraded":"2024-03-24T11:59:51.100561+0000","last_fullsized":"2024-03-24T11:59:51.100561+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:05.394143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.394143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:24.760554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":271,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.755941+0000","last_change":"2024-03-24T11:58:07.457814+0000","last_active":"2024-03-24T11:59:51.755941+0000","last_peered":"2024-03-24T11:59:51.755941+0000","last_clean":"2024-03-24T11:59:51.755941+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T11:59:51.755941+0000","last_undegraded":"2024-03-24T11:59:51.755941+0000","last_fullsized":"2024-03-24T11:59:51.755941+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.457593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":281,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751168+0000","last_change":"2024-03-24T11:59:09.953682+0000","last_active":"2024-03-24T11:59:51.751168+0000","last_peered":"2024-03-24T11:59:51.751168+0000","last_clean":"2024-03-24T11:59:51.751168+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T11:59:51.751168+0000","last_undegraded":"2024-03-24T11:59:51.751168+0000","last_fullsized":"2024-03-24T11:59:51.751168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:59:09.953549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.953549+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.935470+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":306,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102612+0000","last_change":"2024-03-24T11:59:01.788586+0000","last_active":"2024-03-24T11:59:51.102612+0000","last_peered":"2024-03-24T11:59:51.102612+0000","last_clean":"2024-03-24T11:59:51.102612+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T11:59:51.102612+0000","last_undegraded":"2024-03-24T11:59:51.102612+0000","last_fullsized":"2024-03-24T11:59:51.102612+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:01.788481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:01.788481+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:14.227573+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105608+0000","last_change":"2024-03-24T11:58:10.110600+0000","last_active":"2024-03-24T11:59:51.105608+0000","last_peered":"2024-03-24T11:59:51.105608+0000","last_clean":"2024-03-24T11:59:51.105608+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T11:59:51.105608+0000","last_undegraded":"2024-03-24T11:59:51.105608+0000","last_fullsized":"2024-03-24T11:59:51.105608+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:58:10.110510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:10.110510+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":378,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102509+0000","last_change":"2024-03-24T11:59:16.869955+0000","last_active":"2024-03-24T11:59:51.102509+0000","last_peered":"2024-03-24T11:59:51.102509+0000","last_clean":"2024-03-24T11:59:51.102509+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T11:59:51.102509+0000","last_undegraded":"2024-03-24T11:59:51.102509+0000","last_fullsized":"2024-03-24T11:59:51.102509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:16.869590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.869590+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:23.849397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.099937+0000","last_change":"2024-03-24T11:59:04.929757+0000","last_active":"2024-03-24T11:59:51.099937+0000","last_peered":"2024-03-24T11:59:51.099937+0000","last_clean":"2024-03-24T11:59:51.099937+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T11:59:51.099937+0000","last_undegraded":"2024-03-24T11:59:51.099937+0000","last_fullsized":"2024-03-24T11:59:51.099937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:04.929640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:04.929640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:28.037222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":295,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.755816+0000","last_change":"2024-03-24T11:59:08.692823+0000","last_active":"2024-03-24T11:59:51.755816+0000","last_peered":"2024-03-24T11:59:51.755816+0000","last_clean":"2024-03-24T11:59:51.755816+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T11:59:51.755816+0000","last_undegraded":"2024-03-24T11:59:51.755816+0000","last_fullsized":"2024-03-24T11:59:51.755816+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.692644+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:09.789540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753401+0000","last_change":"2024-03-24T11:59:06.058871+0000","last_active":"2024-03-24T11:59:51.753401+0000","last_peered":"2024-03-24T11:59:51.753401+0000","last_clean":"2024-03-24T11:59:51.753401+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T11:59:51.753401+0000","last_undegraded":"2024-03-24T11:59:51.753401+0000","last_fullsized":"2024-03-24T11:59:51.753401+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:06.058633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.058633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.404319+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":269,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102991+0000","last_change":"2024-03-24T11:58:00.121569+0000","last_active":"2024-03-24T11:59:51.102991+0000","last_peered":"2024-03-24T11:59:51.102991+0000","last_clean":"2024-03-24T11:59:51.102991+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T11:59:51.102991+0000","last_undegraded":"2024-03-24T11:59:51.102991+0000","last_fullsized":"2024-03-24T11:59:51.102991+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T11:58:00.121445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":283,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102330+0000","last_change":"2024-03-24T11:59:04.798046+0000","last_active":"2024-03-24T11:59:51.102330+0000","last_peered":"2024-03-24T11:59:51.102330+0000","last_clean":"2024-03-24T11:59:51.102330+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T11:59:51.102330+0000","last_undegraded":"2024-03-24T11:59:51.102330+0000","last_fullsized":"2024-03-24T11:59:51.102330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:04.797928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:04.797928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.060878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":295,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751484+0000","last_change":"2024-03-24T11:59:07.046859+0000","last_active":"2024-03-24T11:59:51.751484+0000","last_peered":"2024-03-24T11:59:51.751484+0000","last_clean":"2024-03-24T11:59:51.751484+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T11:59:51.751484+0000","last_undegraded":"2024-03-24T11:59:51.751484+0000","last_fullsized":"2024-03-24T11:59:51.751484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:07.046695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.046695+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.385099+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.103913+0000","last_change":"2024-03-24T11:58:01.133824+0000","last_active":"2024-03-24T11:59:51.103913+0000","last_peered":"2024-03-24T11:59:51.103913+0000","last_clean":"2024-03-24T11:59:51.103913+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T11:59:51.103913+0000","last_undegraded":"2024-03-24T11:59:51.103913+0000","last_fullsized":"2024-03-24T11:59:51.103913+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T11:58:01.133663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":275,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101293+0000","last_change":"2024-03-24T11:58:57.780289+0000","last_active":"2024-03-24T11:59:51.101293+0000","last_peered":"2024-03-24T11:59:51.101293+0000","last_clean":"2024-03-24T11:59:51.101293+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T11:59:51.101293+0000","last_undegraded":"2024-03-24T11:59:51.101293+0000","last_fullsized":"2024-03-24T11:59:51.101293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:57.780180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.780180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:01.258944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":276,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102074+0000","last_change":"2024-03-24T11:58:01.734332+0000","last_active":"2024-03-24T11:59:51.102074+0000","last_peered":"2024-03-24T11:59:51.102074+0000","last_clean":"2024-03-24T11:59:51.102074+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T11:59:51.102074+0000","last_undegraded":"2024-03-24T11:59:51.102074+0000","last_fullsized":"2024-03-24T11:59:51.102074+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:01.734097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:01.734097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"180'6","reported_seq":292,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.752594+0000","last_change":"2024-03-24T11:59:25.157079+0000","last_active":"2024-03-24T11:59:51.752594+0000","last_peered":"2024-03-24T11:59:51.752594+0000","last_clean":"2024-03-24T11:59:51.752594+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T11:59:51.752594+0000","last_undegraded":"2024-03-24T11:59:51.752594+0000","last_fullsized":"2024-03-24T11:59:51.752594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'6","last_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T11:59:25.156553+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:29.459760+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.750991+0000","last_change":"2024-03-24T11:59:08.970434+0000","last_active":"2024-03-24T11:59:51.750991+0000","last_peered":"2024-03-24T11:59:51.750991+0000","last_clean":"2024-03-24T11:59:51.750991+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T11:59:51.750991+0000","last_undegraded":"2024-03-24T11:59:51.750991+0000","last_fullsized":"2024-03-24T11:59:51.750991+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:08.970296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.970296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.288013+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":29,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.099776+0000","last_change":"2024-03-24T11:59:19.100130+0000","last_active":"2024-03-24T11:59:51.099776+0000","last_peered":"2024-03-24T11:59:51.099776+0000","last_clean":"2024-03-24T11:59:51.099776+0000","last_became_active":"2024-03-24T11:59:19.099779+0000","last_became_peered":"2024-03-24T11:59:19.099779+0000","last_unstale":"2024-03-24T11:59:51.099776+0000","last_undegraded":"2024-03-24T11:59:51.099776+0000","last_fullsized":"2024-03-24T11:59:51.099776+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:53.764112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:57:53.764112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.111897+0000","last_change":"2024-03-24T11:58:53.199843+0000","last_active":"2024-03-24T11:59:51.111897+0000","last_peered":"2024-03-24T11:59:51.111897+0000","last_clean":"2024-03-24T11:59:51.111897+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T11:59:51.111897+0000","last_undegraded":"2024-03-24T11:59:51.111897+0000","last_fullsized":"2024-03-24T11:59:51.111897+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:53.199764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:53.199764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:17.646838+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":299,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.096952+0000","last_change":"2024-03-24T11:59:49.721576+0000","last_active":"2024-03-24T11:59:51.096952+0000","last_peered":"2024-03-24T11:59:51.096952+0000","last_clean":"2024-03-24T11:59:51.096952+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T11:59:51.096952+0000","last_undegraded":"2024-03-24T11:59:51.096952+0000","last_fullsized":"2024-03-24T11:59:51.096952+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:49.721488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:49.721488+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:12.583653+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":267,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754234+0000","last_change":"2024-03-24T11:58:54.590753+0000","last_active":"2024-03-24T11:59:51.754234+0000","last_peered":"2024-03-24T11:59:51.754234+0000","last_clean":"2024-03-24T11:59:51.754234+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T11:59:51.754234+0000","last_undegraded":"2024-03-24T11:59:51.754234+0000","last_fullsized":"2024-03-24T11:59:51.754234+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:54.590601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T11:58:54.590601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:55.902942+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":272,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753214+0000","last_change":"2024-03-24T11:57:59.752607+0000","last_active":"2024-03-24T11:59:51.753214+0000","last_peered":"2024-03-24T11:59:51.753214+0000","last_clean":"2024-03-24T11:59:51.753214+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T11:59:51.753214+0000","last_undegraded":"2024-03-24T11:59:51.753214+0000","last_fullsized":"2024-03-24T11:59:51.753214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:59.752484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:59.752484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":291,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101618+0000","last_change":"2024-03-24T11:59:24.868399+0000","last_active":"2024-03-24T11:59:51.101618+0000","last_peered":"2024-03-24T11:59:51.101618+0000","last_clean":"2024-03-24T11:59:51.101618+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T11:59:51.101618+0000","last_undegraded":"2024-03-24T11:59:51.101618+0000","last_fullsized":"2024-03-24T11:59:51.101618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.868290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.868290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:28.549760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":63,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101310+0000","last_change":"2024-03-24T11:59:19.097992+0000","last_active":"2024-03-24T11:59:51.101310+0000","last_peered":"2024-03-24T11:59:51.101310+0000","last_clean":"2024-03-24T11:59:51.101310+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T11:59:51.101310+0000","last_undegraded":"2024-03-24T11:59:51.101310+0000","last_fullsized":"2024-03-24T11:59:51.101310+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:59:02.301956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:02.301956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:17.685924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.096592+0000","last_change":"2024-03-24T11:59:12.776833+0000","last_active":"2024-03-24T11:59:51.096592+0000","last_peered":"2024-03-24T11:59:51.096592+0000","last_clean":"2024-03-24T11:59:51.096592+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T11:59:51.096592+0000","last_undegraded":"2024-03-24T11:59:51.096592+0000","last_fullsized":"2024-03-24T11:59:51.096592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T11:59:12.776686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:19.703719+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":380,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753996+0000","last_change":"2024-03-24T11:59:24.105607+0000","last_active":"2024-03-24T11:59:51.753996+0000","last_peered":"2024-03-24T11:59:51.753996+0000","last_clean":"2024-03-24T11:59:51.753996+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T11:59:51.753996+0000","last_undegraded":"2024-03-24T11:59:51.753996+0000","last_fullsized":"2024-03-24T11:59:51.753996+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:24.105308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.105308+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.462310+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":279,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754612+0000","last_change":"2024-03-24T11:59:22.532876+0000","last_active":"2024-03-24T11:59:51.754612+0000","last_peered":"2024-03-24T11:59:51.754612+0000","last_clean":"2024-03-24T11:59:51.754612+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T11:59:51.754612+0000","last_undegraded":"2024-03-24T11:59:51.754612+0000","last_fullsized":"2024-03-24T11:59:51.754612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.532751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.532751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:51.840534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.100325+0000","last_change":"2024-03-24T11:59:19.016347+0000","last_active":"2024-03-24T11:59:51.100325+0000","last_peered":"2024-03-24T11:59:51.100325+0000","last_clean":"2024-03-24T11:59:51.100325+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T11:59:51.100325+0000","last_undegraded":"2024-03-24T11:59:51.100325+0000","last_fullsized":"2024-03-24T11:59:51.100325+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:59:19.016233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:19.016233+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:39.365965+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":267,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753938+0000","last_change":"2024-03-24T11:59:16.675122+0000","last_active":"2024-03-24T11:59:51.753938+0000","last_peered":"2024-03-24T11:59:51.753938+0000","last_clean":"2024-03-24T11:59:51.753938+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T11:59:51.753938+0000","last_undegraded":"2024-03-24T11:59:51.753938+0000","last_fullsized":"2024-03-24T11:59:51.753938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:16.673746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.673746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:20.701023+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":376,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101539+0000","last_change":"2024-03-24T11:58:13.687062+0000","last_active":"2024-03-24T11:59:51.101539+0000","last_peered":"2024-03-24T11:59:51.101539+0000","last_clean":"2024-03-24T11:59:51.101539+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T11:59:51.101539+0000","last_undegraded":"2024-03-24T11:59:51.101539+0000","last_fullsized":"2024-03-24T11:59:51.101539+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:58:13.686957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:13.686957+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.099574+0000","last_change":"2024-03-24T11:59:22.001137+0000","last_active":"2024-03-24T11:59:51.099574+0000","last_peered":"2024-03-24T11:59:51.099574+0000","last_clean":"2024-03-24T11:59:51.099574+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T11:59:51.099574+0000","last_undegraded":"2024-03-24T11:59:51.099574+0000","last_fullsized":"2024-03-24T11:59:51.099574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.000913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.000913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:25.798172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":296,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753724+0000","last_change":"2024-03-24T11:57:40.662287+0000","last_active":"2024-03-24T11:59:51.753724+0000","last_peered":"2024-03-24T11:59:51.753724+0000","last_clean":"2024-03-24T11:59:51.753724+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T11:59:51.753724+0000","last_undegraded":"2024-03-24T11:59:51.753724+0000","last_fullsized":"2024-03-24T11:59:51.753724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:57:40.662163+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:57:40.662163+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105632+0000","last_change":"2024-03-24T11:59:12.991274+0000","last_active":"2024-03-24T11:59:51.105632+0000","last_peered":"2024-03-24T11:59:51.105632+0000","last_clean":"2024-03-24T11:59:51.105632+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T11:59:51.105632+0000","last_undegraded":"2024-03-24T11:59:51.105632+0000","last_fullsized":"2024-03-24T11:59:51.105632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:12.991112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:12.991112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.564791+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":268,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101709+0000","last_change":"2024-03-24T11:58:06.748927+0000","last_active":"2024-03-24T11:59:51.101709+0000","last_peered":"2024-03-24T11:59:51.101709+0000","last_clean":"2024-03-24T11:59:51.101709+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T11:59:51.101709+0000","last_undegraded":"2024-03-24T11:59:51.101709+0000","last_fullsized":"2024-03-24T11:59:51.101709+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:06.748834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:06.748834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":291,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101259+0000","last_change":"2024-03-24T11:59:48.877101+0000","last_active":"2024-03-24T11:59:51.101259+0000","last_peered":"2024-03-24T11:59:51.101259+0000","last_clean":"2024-03-24T11:59:51.101259+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T11:59:51.101259+0000","last_undegraded":"2024-03-24T11:59:51.101259+0000","last_fullsized":"2024-03-24T11:59:51.101259+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:48.876979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T11:59:48.876979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:17.408575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":293,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751742+0000","last_change":"2024-03-24T11:59:23.086796+0000","last_active":"2024-03-24T11:59:51.751742+0000","last_peered":"2024-03-24T11:59:51.751742+0000","last_clean":"2024-03-24T11:59:51.751742+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T11:59:51.751742+0000","last_undegraded":"2024-03-24T11:59:51.751742+0000","last_fullsized":"2024-03-24T11:59:51.751742+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:59:23.086667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:23.086667+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:41.979853+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.109496+0000","last_change":"2024-03-24T11:59:23.956872+0000","last_active":"2024-03-24T11:59:51.109496+0000","last_peered":"2024-03-24T11:59:51.109496+0000","last_clean":"2024-03-24T11:59:51.109496+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T11:59:51.109496+0000","last_undegraded":"2024-03-24T11:59:51.109496+0000","last_fullsized":"2024-03-24T11:59:51.109496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:23.956595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:23.956595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:43.862735+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":272,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751833+0000","last_change":"2024-03-24T11:59:14.960274+0000","last_active":"2024-03-24T11:59:51.751833+0000","last_peered":"2024-03-24T11:59:51.751833+0000","last_clean":"2024-03-24T11:59:51.751833+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T11:59:51.751833+0000","last_undegraded":"2024-03-24T11:59:51.751833+0000","last_fullsized":"2024-03-24T11:59:51.751833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:14.960108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.960108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:41.859382+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":286,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.104454+0000","last_change":"2024-03-24T11:58:59.882590+0000","last_active":"2024-03-24T11:59:51.104454+0000","last_peered":"2024-03-24T11:59:51.104454+0000","last_clean":"2024-03-24T11:59:51.104454+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T11:59:51.104454+0000","last_undegraded":"2024-03-24T11:59:51.104454+0000","last_fullsized":"2024-03-24T11:59:51.104454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:59.882508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:59.882508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:04.582653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"115'13","reported_seq":295,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754139+0000","last_change":"2024-03-24T11:58:55.576054+0000","last_active":"2024-03-24T11:59:51.754139+0000","last_peered":"2024-03-24T11:59:51.754139+0000","last_clean":"2024-03-24T11:59:51.754139+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:59:51.754139+0000","last_undegraded":"2024-03-24T11:59:51.754139+0000","last_fullsized":"2024-03-24T11:59:51.754139+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'13","last_scrub_stamp":"2024-03-24T11:58:55.575897+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T11:58:55.575897+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:19.716606+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"187'5192","reported_seq":5235,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.202525+0000","last_change":"2024-03-24T11:58:42.913612+0000","last_active":"2024-03-24T11:59:51.202525+0000","last_peered":"2024-03-24T11:59:51.202525+0000","last_clean":"2024-03-24T11:59:51.202525+0000","last_became_active":"2024-03-24T11:58:42.913076+0000","last_became_peered":"2024-03-24T11:58:42.913076+0000","last_unstale":"2024-03-24T11:59:51.202525+0000","last_undegraded":"2024-03-24T11:59:51.202525+0000","last_fullsized":"2024-03-24T11:59:51.202525+0000","mapping_epoch":170,"log_start":"175'100","ondisk_log_start":"175'100","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340262912,"num_objects":5192,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":332288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753308+0000","last_change":"2024-03-24T11:59:03.131422+0000","last_active":"2024-03-24T11:59:51.753308+0000","last_peered":"2024-03-24T11:59:51.753308+0000","last_clean":"2024-03-24T11:59:51.753308+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T11:59:51.753308+0000","last_undegraded":"2024-03-24T11:59:51.753308+0000","last_fullsized":"2024-03-24T11:59:51.753308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:03.131309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:03.131309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:03.652291+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.096357+0000","last_change":"2024-03-24T11:59:07.635339+0000","last_active":"2024-03-24T11:59:51.096357+0000","last_peered":"2024-03-24T11:59:51.096357+0000","last_clean":"2024-03-24T11:59:51.096357+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T11:59:51.096357+0000","last_undegraded":"2024-03-24T11:59:51.096357+0000","last_fullsized":"2024-03-24T11:59:51.096357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T11:59:07.635222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.635222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:11.202763+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":329,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.095399+0000","last_change":"2024-03-24T11:59:16.735826+0000","last_active":"2024-03-24T11:59:51.095399+0000","last_peered":"2024-03-24T11:59:51.095399+0000","last_clean":"2024-03-24T11:59:51.095399+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T11:59:51.095399+0000","last_undegraded":"2024-03-24T11:59:51.095399+0000","last_fullsized":"2024-03-24T11:59:51.095399+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:59:16.735674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.735674+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.779276+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.104546+0000","last_change":"2024-03-24T11:58:58.833812+0000","last_active":"2024-03-24T11:59:51.104546+0000","last_peered":"2024-03-24T11:59:51.104546+0000","last_clean":"2024-03-24T11:59:51.104546+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T11:59:51.104546+0000","last_undegraded":"2024-03-24T11:59:51.104546+0000","last_fullsized":"2024-03-24T11:59:51.104546+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:58.833700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:58.833700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:15.607455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":299,"reported_epoch":187,"state":"remapped+peering","last_fresh":"2024-03-24T11:59:51.103499+0000","last_change":"2024-03-24T11:59:51.103499+0000","last_active":"2024-03-24T11:59:50.079024+0000","last_peered":"2024-03-24T11:52:57.225001+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:59:51.103499+0000","last_undegraded":"2024-03-24T11:59:51.103499+0000","last_fullsized":"2024-03-24T11:59:51.103499+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"188'5175","reported_seq":5221,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753613+0000","last_change":"2024-03-24T11:58:42.973572+0000","last_active":"2024-03-24T11:59:51.753613+0000","last_peered":"2024-03-24T11:59:51.753613+0000","last_clean":"2024-03-24T11:59:51.753613+0000","last_became_active":"2024-03-24T11:58:42.972967+0000","last_became_peered":"2024-03-24T11:58:42.972967+0000","last_unstale":"2024-03-24T11:59:51.753613+0000","last_undegraded":"2024-03-24T11:59:51.753613+0000","last_fullsized":"2024-03-24T11:59:51.753613+0000","mapping_epoch":170,"log_start":"175'343","ondisk_log_start":"175'343","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:58.005646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339148800,"num_objects":5175,"num_object_clones":0,"num_object_copies":15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":331200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":267,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754335+0000","last_change":"2024-03-24T11:59:18.622093+0000","last_active":"2024-03-24T11:59:51.754335+0000","last_peered":"2024-03-24T11:59:51.754335+0000","last_clean":"2024-03-24T11:59:51.754335+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T11:59:51.754335+0000","last_undegraded":"2024-03-24T11:59:51.754335+0000","last_fullsized":"2024-03-24T11:59:51.754335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T11:59:18.621992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:44.645312+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754685+0000","last_change":"2024-03-24T11:59:19.117186+0000","last_active":"2024-03-24T11:59:51.754685+0000","last_peered":"2024-03-24T11:59:51.754685+0000","last_clean":"2024-03-24T11:59:51.754685+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T11:59:51.754685+0000","last_undegraded":"2024-03-24T11:59:51.754685+0000","last_fullsized":"2024-03-24T11:59:51.754685+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:13.313625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:59:13.313625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:23.114451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":301,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.112682+0000","last_change":"2024-03-24T11:59:22.170397+0000","last_active":"2024-03-24T11:59:51.112682+0000","last_peered":"2024-03-24T11:59:51.112682+0000","last_clean":"2024-03-24T11:59:51.112682+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T11:59:51.112682+0000","last_undegraded":"2024-03-24T11:59:51.112682+0000","last_fullsized":"2024-03-24T11:59:51.112682+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T11:59:22.170130+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.170130+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:46.149755+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":379,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754773+0000","last_change":"2024-03-24T11:59:51.517974+0000","last_active":"2024-03-24T11:59:51.754773+0000","last_peered":"2024-03-24T11:59:51.754773+0000","last_clean":"2024-03-24T11:59:51.754773+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T11:59:51.754773+0000","last_undegraded":"2024-03-24T11:59:51.754773+0000","last_fullsized":"2024-03-24T11:59:51.754773+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:51.517833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:51.517833+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:52.842240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":305,"reported_epoch":188,"state":"active+undersized+remapped","last_fresh":"2024-03-24T11:59:51.775973+0000","last_change":"2024-03-24T11:59:51.775652+0000","last_active":"2024-03-24T11:59:51.775973+0000","last_peered":"2024-03-24T11:59:51.775973+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T11:59:51.775652+0000","last_became_peered":"2024-03-24T11:59:51.775652+0000","last_unstale":"2024-03-24T11:59:51.775973+0000","last_undegraded":"2024-03-24T11:59:51.775973+0000","last_fullsized":"2024-03-24T11:59:51.754972+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[0,2],"avail_no_missing":["0","2","5","6","7"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"187'5275","reported_seq":5319,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.219343+0000","last_change":"2024-03-24T11:58:42.913905+0000","last_active":"2024-03-24T11:59:51.219343+0000","last_peered":"2024-03-24T11:59:51.219343+0000","last_clean":"2024-03-24T11:59:51.219343+0000","last_became_active":"2024-03-24T11:58:42.913442+0000","last_became_peered":"2024-03-24T11:58:42.913442+0000","last_unstale":"2024-03-24T11:59:51.219343+0000","last_undegraded":"2024-03-24T11:59:51.219343+0000","last_fullsized":"2024-03-24T11:59:51.219343+0000","mapping_epoch":170,"log_start":"175'196","ondisk_log_start":"175'196","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:11.729965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345702400,"num_objects":5275,"num_object_clones":0,"num_object_copies":15825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":337600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751335+0000","last_change":"2024-03-24T11:58:10.828344+0000","last_active":"2024-03-24T11:59:51.751335+0000","last_peered":"2024-03-24T11:59:51.751335+0000","last_clean":"2024-03-24T11:59:51.751335+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T11:59:51.751335+0000","last_undegraded":"2024-03-24T11:59:51.751335+0000","last_fullsized":"2024-03-24T11:59:51.751335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:10.828263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:10.828263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.099618+0000","last_change":"2024-03-24T11:59:09.914029+0000","last_active":"2024-03-24T11:59:51.099618+0000","last_peered":"2024-03-24T11:59:51.099618+0000","last_clean":"2024-03-24T11:59:51.099618+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T11:59:51.099618+0000","last_undegraded":"2024-03-24T11:59:51.099618+0000","last_fullsized":"2024-03-24T11:59:51.099618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.913946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:13.920799+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":364,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.099502+0000","last_change":"2024-03-24T11:58:11.704697+0000","last_active":"2024-03-24T11:59:51.099502+0000","last_peered":"2024-03-24T11:59:51.099502+0000","last_clean":"2024-03-24T11:59:51.099502+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T11:59:51.099502+0000","last_undegraded":"2024-03-24T11:59:51.099502+0000","last_fullsized":"2024-03-24T11:59:51.099502+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-03-24T11:58:11.704574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:58:11.704574+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":271,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753959+0000","last_change":"2024-03-24T11:59:14.667905+0000","last_active":"2024-03-24T11:59:51.753959+0000","last_peered":"2024-03-24T11:59:51.753959+0000","last_clean":"2024-03-24T11:59:51.753959+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T11:59:51.753959+0000","last_undegraded":"2024-03-24T11:59:51.753959+0000","last_fullsized":"2024-03-24T11:59:51.753959+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:14.667523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.667523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:15.199214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":538,"reported_epoch":187,"state":"remapped+peering","last_fresh":"2024-03-24T11:59:51.105337+0000","last_change":"2024-03-24T11:59:51.105337+0000","last_active":"2024-03-24T11:59:50.083542+0000","last_peered":"2024-03-24T11:59:49.092280+0000","last_clean":"2024-03-24T11:59:49.092280+0000","last_became_active":"2024-03-24T11:59:49.091462+0000","last_became_peered":"2024-03-24T11:59:49.091462+0000","last_unstale":"2024-03-24T11:59:51.105337+0000","last_undegraded":"2024-03-24T11:59:51.105337+0000","last_fullsized":"2024-03-24T11:59:51.105337+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"188'5142","reported_seq":5188,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.941345+0000","last_change":"2024-03-24T11:58:42.817551+0000","last_active":"2024-03-24T11:59:51.941345+0000","last_peered":"2024-03-24T11:59:51.941345+0000","last_clean":"2024-03-24T11:59:51.941345+0000","last_became_active":"2024-03-24T11:58:42.817154+0000","last_became_peered":"2024-03-24T11:58:42.817154+0000","last_unstale":"2024-03-24T11:59:51.941345+0000","last_undegraded":"2024-03-24T11:59:51.941345+0000","last_fullsized":"2024-03-24T11:59:51.941345+0000","mapping_epoch":170,"log_start":"175'584","ondisk_log_start":"175'584","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.457603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336986112,"num_objects":5142,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5142,"num_write_kb":329088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105797+0000","last_change":"2024-03-24T11:59:49.115722+0000","last_active":"2024-03-24T11:59:51.105797+0000","last_peered":"2024-03-24T11:59:51.105797+0000","last_clean":"2024-03-24T11:59:51.105797+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T11:59:51.105797+0000","last_undegraded":"2024-03-24T11:59:51.105797+0000","last_fullsized":"2024-03-24T11:59:51.105797+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:49.115475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:49.115475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:55.296110+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":110,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.103928+0000","last_change":"2024-03-24T11:59:22.979714+0000","last_active":"2024-03-24T11:59:51.103928+0000","last_peered":"2024-03-24T11:59:51.103928+0000","last_clean":"2024-03-24T11:59:51.103928+0000","last_became_active":"2024-03-24T11:59:19.095838+0000","last_became_peered":"2024-03-24T11:59:19.095838+0000","last_unstale":"2024-03-24T11:59:51.103928+0000","last_undegraded":"2024-03-24T11:59:51.103928+0000","last_fullsized":"2024-03-24T11:59:51.103928+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:59:22.978824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.978824+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:47.966247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":294,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105338+0000","last_change":"2024-03-24T11:59:07.080907+0000","last_active":"2024-03-24T11:59:51.105338+0000","last_peered":"2024-03-24T11:59:51.105338+0000","last_clean":"2024-03-24T11:59:51.105338+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T11:59:51.105338+0000","last_undegraded":"2024-03-24T11:59:51.105338+0000","last_fullsized":"2024-03-24T11:59:51.105338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:07.080763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.080763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:35.882906+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":300,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.111241+0000","last_change":"2024-03-24T11:58:54.237004+0000","last_active":"2024-03-24T11:59:51.111241+0000","last_peered":"2024-03-24T11:59:51.111241+0000","last_clean":"2024-03-24T11:59:51.111241+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T11:59:51.111241+0000","last_undegraded":"2024-03-24T11:59:51.111241+0000","last_fullsized":"2024-03-24T11:59:51.111241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-03-24T11:58:54.236909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:58:54.236909+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:07.217796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":286,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105693+0000","last_change":"2024-03-24T11:59:12.041065+0000","last_active":"2024-03-24T11:59:51.105693+0000","last_peered":"2024-03-24T11:59:51.105693+0000","last_clean":"2024-03-24T11:59:51.105693+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T11:59:51.105693+0000","last_undegraded":"2024-03-24T11:59:51.105693+0000","last_fullsized":"2024-03-24T11:59:51.105693+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:12.040754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T11:59:12.040754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.895458+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"187'5220","reported_seq":5263,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.168568+0000","last_change":"2024-03-24T11:58:42.819958+0000","last_active":"2024-03-24T11:59:51.168568+0000","last_peered":"2024-03-24T11:59:51.168568+0000","last_clean":"2024-03-24T11:59:51.168568+0000","last_became_active":"2024-03-24T11:58:42.819563+0000","last_became_peered":"2024-03-24T11:58:42.819563+0000","last_unstale":"2024-03-24T11:59:51.168568+0000","last_undegraded":"2024-03-24T11:59:51.168568+0000","last_fullsized":"2024-03-24T11:59:51.168568+0000","mapping_epoch":170,"log_start":"175'100","ondisk_log_start":"175'100","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":5120,"log_dups_size":0,"ondisk_log_size":5120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342097920,"num_objects":5220,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":334080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102845+0000","last_change":"2024-03-24T11:59:24.028954+0000","last_active":"2024-03-24T11:59:51.102845+0000","last_peered":"2024-03-24T11:59:51.102845+0000","last_clean":"2024-03-24T11:59:51.102845+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T11:59:51.102845+0000","last_undegraded":"2024-03-24T11:59:51.102845+0000","last_fullsized":"2024-03-24T11:59:51.102845+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.028868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.028868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:41.479995+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":289,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751761+0000","last_change":"2024-03-24T11:58:57.101127+0000","last_active":"2024-03-24T11:59:51.751761+0000","last_peered":"2024-03-24T11:59:51.751761+0000","last_clean":"2024-03-24T11:59:51.751761+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T11:59:51.751761+0000","last_undegraded":"2024-03-24T11:59:51.751761+0000","last_fullsized":"2024-03-24T11:59:51.751761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T11:58:57.101013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.101013+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:05.969198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.099341+0000","last_change":"2024-03-24T11:59:51.078512+0000","last_active":"2024-03-24T11:59:51.099341+0000","last_peered":"2024-03-24T11:59:51.099341+0000","last_clean":"2024-03-24T11:59:51.099341+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T11:59:51.099341+0000","last_undegraded":"2024-03-24T11:59:51.099341+0000","last_fullsized":"2024-03-24T11:59:51.099341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:51.078392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:51.078392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:17.393683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":549,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102200+0000","last_change":"2024-03-24T11:59:50.052718+0000","last_active":"2024-03-24T11:59:51.102200+0000","last_peered":"2024-03-24T11:59:51.102200+0000","last_clean":"2024-03-24T11:59:51.102200+0000","last_became_active":"2024-03-24T11:59:49.085087+0000","last_became_peered":"2024-03-24T11:59:49.085087+0000","last_unstale":"2024-03-24T11:59:51.102200+0000","last_undegraded":"2024-03-24T11:59:51.102200+0000","last_fullsized":"2024-03-24T11:59:51.102200+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:50.052605+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:50.052605+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:15.833750+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":401,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102130+0000","last_change":"2024-03-24T11:59:21.898682+0000","last_active":"2024-03-24T11:59:51.102130+0000","last_peered":"2024-03-24T11:59:51.102130+0000","last_clean":"2024-03-24T11:59:51.102130+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T11:59:51.102130+0000","last_undegraded":"2024-03-24T11:59:51.102130+0000","last_fullsized":"2024-03-24T11:59:51.102130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:59:21.898325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.898325+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:22.738734+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":277,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105453+0000","last_change":"2024-03-24T11:59:13.979671+0000","last_active":"2024-03-24T11:59:51.105453+0000","last_peered":"2024-03-24T11:59:51.105453+0000","last_clean":"2024-03-24T11:59:51.105453+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T11:59:51.105453+0000","last_undegraded":"2024-03-24T11:59:51.105453+0000","last_fullsized":"2024-03-24T11:59:51.105453+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T11:59:13.979581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:20.284311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":299,"reported_epoch":187,"state":"remapped+peering","last_fresh":"2024-03-24T11:59:51.102333+0000","last_change":"2024-03-24T11:59:51.102333+0000","last_active":"2024-03-24T11:59:50.082895+0000","last_peered":"2024-03-24T11:52:57.225001+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:59:51.102333+0000","last_undegraded":"2024-03-24T11:59:51.102333+0000","last_fullsized":"2024-03-24T11:59:51.102333+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"187'5256","reported_seq":5305,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.514006+0000","last_change":"2024-03-24T11:59:19.104062+0000","last_active":"2024-03-24T11:59:51.514006+0000","last_peered":"2024-03-24T11:59:51.514006+0000","last_clean":"2024-03-24T11:59:51.514006+0000","last_became_active":"2024-03-24T11:59:19.098023+0000","last_became_peered":"2024-03-24T11:59:19.098023+0000","last_unstale":"2024-03-24T11:59:51.514006+0000","last_undegraded":"2024-03-24T11:59:51.514006+0000","last_fullsized":"2024-03-24T11:59:51.514006+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:04.358454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344457216,"num_objects":5256,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5256,"num_write_kb":336384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.111336+0000","last_change":"2024-03-24T11:59:25.199926+0000","last_active":"2024-03-24T11:59:51.111336+0000","last_peered":"2024-03-24T11:59:51.111336+0000","last_clean":"2024-03-24T11:59:51.111336+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T11:59:51.111336+0000","last_undegraded":"2024-03-24T11:59:51.111336+0000","last_fullsized":"2024-03-24T11:59:51.111336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:25.199804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:25.199804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:31.158743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":286,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.760418+0000","last_change":"2024-03-24T11:59:21.570352+0000","last_active":"2024-03-24T11:59:51.760418+0000","last_peered":"2024-03-24T11:59:51.760418+0000","last_clean":"2024-03-24T11:59:51.760418+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T11:59:51.760418+0000","last_undegraded":"2024-03-24T11:59:51.760418+0000","last_fullsized":"2024-03-24T11:59:51.760418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.570115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.570115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:25.514405+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":123,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754756+0000","last_change":"2024-03-24T11:59:20.570521+0000","last_active":"2024-03-24T11:59:51.754756+0000","last_peered":"2024-03-24T11:59:51.754756+0000","last_clean":"2024-03-24T11:59:51.754756+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T11:59:51.754756+0000","last_undegraded":"2024-03-24T11:59:51.754756+0000","last_fullsized":"2024-03-24T11:59:51.754756+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:20.570220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:20.570220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:30.373801+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":293,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754038+0000","last_change":"2024-03-24T11:59:03.726421+0000","last_active":"2024-03-24T11:59:51.754038+0000","last_peered":"2024-03-24T11:59:51.754038+0000","last_clean":"2024-03-24T11:59:51.754038+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T11:59:51.754038+0000","last_undegraded":"2024-03-24T11:59:51.754038+0000","last_fullsized":"2024-03-24T11:59:51.754038+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:59:03.726034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:03.726034+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:08.722175+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":276,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102745+0000","last_change":"2024-03-24T11:59:48.136534+0000","last_active":"2024-03-24T11:59:51.102745+0000","last_peered":"2024-03-24T11:59:51.102745+0000","last_clean":"2024-03-24T11:59:51.102745+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T11:59:51.102745+0000","last_undegraded":"2024-03-24T11:59:51.102745+0000","last_fullsized":"2024-03-24T11:59:51.102745+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:48.136407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:48.136407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:48.341122+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":299,"reported_epoch":187,"state":"remapped+peering","last_fresh":"2024-03-24T11:59:51.102661+0000","last_change":"2024-03-24T11:59:51.102661+0000","last_active":"2024-03-24T11:59:50.079905+0000","last_peered":"2024-03-24T11:52:57.225001+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T11:52:05.619150+0000","last_became_peered":"2024-03-24T11:52:05.619150+0000","last_unstale":"2024-03-24T11:59:51.102661+0000","last_undegraded":"2024-03-24T11:59:51.102661+0000","last_fullsized":"2024-03-24T11:59:51.102661+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"188'5247","reported_seq":5293,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.984385+0000","last_change":"2024-03-24T11:58:42.970585+0000","last_active":"2024-03-24T11:59:51.984385+0000","last_peered":"2024-03-24T11:59:51.984385+0000","last_clean":"2024-03-24T11:59:51.984385+0000","last_became_active":"2024-03-24T11:58:42.970288+0000","last_became_peered":"2024-03-24T11:58:42.970288+0000","last_unstale":"2024-03-24T11:59:51.984385+0000","last_undegraded":"2024-03-24T11:59:51.984385+0000","last_fullsized":"2024-03-24T11:59:51.984385+0000","mapping_epoch":170,"log_start":"175'683","ondisk_log_start":"175'683","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:09.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343867392,"num_objects":5247,"num_object_clones":0,"num_object_copies":15741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5247,"num_write_kb":335808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":28,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.100828+0000","last_change":"2024-03-24T11:59:19.096886+0000","last_active":"2024-03-24T11:59:51.100828+0000","last_peered":"2024-03-24T11:59:51.100828+0000","last_clean":"2024-03-24T11:59:51.100828+0000","last_became_active":"2024-03-24T11:59:19.096322+0000","last_became_peered":"2024-03-24T11:59:19.096322+0000","last_unstale":"2024-03-24T11:59:51.100828+0000","last_undegraded":"2024-03-24T11:59:51.100828+0000","last_fullsized":"2024-03-24T11:59:51.100828+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:56.287993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:56.287993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:25.635939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105844+0000","last_change":"2024-03-24T11:58:14.079445+0000","last_active":"2024-03-24T11:59:51.105844+0000","last_peered":"2024-03-24T11:59:51.105844+0000","last_clean":"2024-03-24T11:59:51.105844+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T11:59:51.105844+0000","last_undegraded":"2024-03-24T11:59:51.105844+0000","last_fullsized":"2024-03-24T11:59:51.105844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:14.079300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:14.079300+0000","last_clean_scrub_stamp":"2024-03-24T11:58:14.079300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":311,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754502+0000","last_change":"2024-03-24T11:59:06.717918+0000","last_active":"2024-03-24T11:59:51.754502+0000","last_peered":"2024-03-24T11:59:51.754502+0000","last_clean":"2024-03-24T11:59:51.754502+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T11:59:51.754502+0000","last_undegraded":"2024-03-24T11:59:51.754502+0000","last_fullsized":"2024-03-24T11:59:51.754502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:59:06.717709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.717709+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:26.508160+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":275,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101527+0000","last_change":"2024-03-24T11:59:09.760242+0000","last_active":"2024-03-24T11:59:51.101527+0000","last_peered":"2024-03-24T11:59:51.101527+0000","last_clean":"2024-03-24T11:59:51.101527+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T11:59:51.101527+0000","last_undegraded":"2024-03-24T11:59:51.101527+0000","last_fullsized":"2024-03-24T11:59:51.101527+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:09.760120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.760120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:11.704260+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":306,"reported_epoch":188,"state":"active+undersized+remapped","last_fresh":"2024-03-24T11:59:51.773113+0000","last_change":"2024-03-24T11:59:51.772822+0000","last_active":"2024-03-24T11:59:51.773113+0000","last_peered":"2024-03-24T11:59:51.773113+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T11:59:51.772822+0000","last_became_peered":"2024-03-24T11:59:51.772822+0000","last_unstale":"2024-03-24T11:59:51.773113+0000","last_undegraded":"2024-03-24T11:59:51.773113+0000","last_fullsized":"2024-03-24T11:59:51.753084+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[0,6],"avail_no_missing":["0","2","4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"187'5291","reported_seq":5337,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754510+0000","last_change":"2024-03-24T11:58:42.911757+0000","last_active":"2024-03-24T11:59:51.754510+0000","last_peered":"2024-03-24T11:59:51.754510+0000","last_clean":"2024-03-24T11:59:51.754510+0000","last_became_active":"2024-03-24T11:58:42.911298+0000","last_became_peered":"2024-03-24T11:58:42.911298+0000","last_unstale":"2024-03-24T11:59:51.754510+0000","last_undegraded":"2024-03-24T11:59:51.754510+0000","last_fullsized":"2024-03-24T11:59:51.754510+0000","mapping_epoch":170,"log_start":"175'440","ondisk_log_start":"175'440","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:06.815917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346750976,"num_objects":5291,"num_object_clones":0,"num_object_copies":15873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":338624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.097177+0000","last_change":"2024-03-24T11:58:07.866715+0000","last_active":"2024-03-24T11:59:51.097177+0000","last_peered":"2024-03-24T11:59:51.097177+0000","last_clean":"2024-03-24T11:59:51.097177+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T11:59:51.097177+0000","last_undegraded":"2024-03-24T11:59:51.097177+0000","last_fullsized":"2024-03-24T11:59:51.097177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.866581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.866581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":36,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.100268+0000","last_change":"2024-03-24T11:59:19.098307+0000","last_active":"2024-03-24T11:59:51.100268+0000","last_peered":"2024-03-24T11:59:51.100268+0000","last_clean":"2024-03-24T11:59:51.100268+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T11:59:51.100268+0000","last_undegraded":"2024-03-24T11:59:51.100268+0000","last_fullsized":"2024-03-24T11:59:51.100268+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:59:10.305433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:10.305433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:20.709439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":308,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.752767+0000","last_change":"2024-03-24T11:58:08.776444+0000","last_active":"2024-03-24T11:59:51.752767+0000","last_peered":"2024-03-24T11:59:51.752767+0000","last_clean":"2024-03-24T11:59:51.752767+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T11:59:51.752767+0000","last_undegraded":"2024-03-24T11:59:51.752767+0000","last_fullsized":"2024-03-24T11:59:51.752767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T11:58:08.776286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:58:08.776286+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":276,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.095445+0000","last_change":"2024-03-24T11:59:24.679291+0000","last_active":"2024-03-24T11:59:51.095445+0000","last_peered":"2024-03-24T11:59:51.095445+0000","last_clean":"2024-03-24T11:59:51.095445+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T11:59:51.095445+0000","last_undegraded":"2024-03-24T11:59:51.095445+0000","last_fullsized":"2024-03-24T11:59:51.095445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.679110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:52.794373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":306,"reported_epoch":188,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-24T11:59:51.772112+0000","last_change":"2024-03-24T11:59:51.772112+0000","last_active":"2024-03-24T11:59:51.772112+0000","last_peered":"2024-03-24T11:59:51.772112+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T11:59:51.769594+0000","last_became_peered":"2024-03-24T11:59:51.769594+0000","last_unstale":"2024-03-24T11:59:51.772112+0000","last_undegraded":"2024-03-24T11:59:51.755415+0000","last_fullsized":"2024-03-24T11:59:51.754305+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[0,7],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,6,7","objects":1},{"shards":"0,7","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"187'5166","reported_seq":5210,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.206961+0000","last_change":"2024-03-24T11:58:42.970705+0000","last_active":"2024-03-24T11:59:51.206961+0000","last_peered":"2024-03-24T11:59:51.206961+0000","last_clean":"2024-03-24T11:59:51.206961+0000","last_became_active":"2024-03-24T11:58:42.970405+0000","last_became_peered":"2024-03-24T11:58:42.970405+0000","last_unstale":"2024-03-24T11:59:51.206961+0000","last_undegraded":"2024-03-24T11:59:51.206961+0000","last_fullsized":"2024-03-24T11:59:51.206961+0000","mapping_epoch":170,"log_start":"175'100","ondisk_log_start":"175'100","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338558976,"num_objects":5166,"num_object_clones":0,"num_object_copies":15498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5166,"num_write_kb":330624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.104225+0000","last_change":"2024-03-24T11:59:05.901391+0000","last_active":"2024-03-24T11:59:51.104225+0000","last_peered":"2024-03-24T11:59:51.104225+0000","last_clean":"2024-03-24T11:59:51.104225+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T11:59:51.104225+0000","last_undegraded":"2024-03-24T11:59:51.104225+0000","last_fullsized":"2024-03-24T11:59:51.104225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:05.901308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.901308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:14.214403+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":283,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102418+0000","last_change":"2024-03-24T11:58:56.778928+0000","last_active":"2024-03-24T11:59:51.102418+0000","last_peered":"2024-03-24T11:59:51.102418+0000","last_clean":"2024-03-24T11:59:51.102418+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T11:59:51.102418+0000","last_undegraded":"2024-03-24T11:59:51.102418+0000","last_fullsized":"2024-03-24T11:59:51.102418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:56.778822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T11:58:56.778822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:04.084039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":308,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.095247+0000","last_change":"2024-03-24T11:59:05.708676+0000","last_active":"2024-03-24T11:59:51.095247+0000","last_peered":"2024-03-24T11:59:51.095247+0000","last_clean":"2024-03-24T11:59:51.095247+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T11:59:51.095247+0000","last_undegraded":"2024-03-24T11:59:51.095247+0000","last_fullsized":"2024-03-24T11:59:51.095247+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:59:05.708532+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.708532+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:33.259042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":372,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754473+0000","last_change":"2024-03-24T11:59:00.089695+0000","last_active":"2024-03-24T11:59:51.754473+0000","last_peered":"2024-03-24T11:59:51.754473+0000","last_clean":"2024-03-24T11:59:51.754473+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T11:59:51.754473+0000","last_undegraded":"2024-03-24T11:59:51.754473+0000","last_fullsized":"2024-03-24T11:59:51.754473+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:00.089617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:00.089617+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:26.813217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"187'5279","reported_seq":5323,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.198769+0000","last_change":"2024-03-24T11:58:42.820321+0000","last_active":"2024-03-24T11:59:51.198769+0000","last_peered":"2024-03-24T11:59:51.198769+0000","last_clean":"2024-03-24T11:59:51.198769+0000","last_became_active":"2024-03-24T11:58:42.819941+0000","last_became_peered":"2024-03-24T11:58:42.819941+0000","last_unstale":"2024-03-24T11:59:51.198769+0000","last_undegraded":"2024-03-24T11:59:51.198769+0000","last_fullsized":"2024-03-24T11:59:51.198769+0000","mapping_epoch":170,"log_start":"175'100","ondisk_log_start":"175'100","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:53.735422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345964544,"num_objects":5279,"num_object_clones":0,"num_object_copies":15837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":337856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753631+0000","last_change":"2024-03-24T11:58:55.061715+0000","last_active":"2024-03-24T11:59:51.753631+0000","last_peered":"2024-03-24T11:59:51.753631+0000","last_clean":"2024-03-24T11:59:51.753631+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T11:59:51.753631+0000","last_undegraded":"2024-03-24T11:59:51.753631+0000","last_fullsized":"2024-03-24T11:59:51.753631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:55.061633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:55.061633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:59.437964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":268,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102608+0000","last_change":"2024-03-24T11:57:40.772815+0000","last_active":"2024-03-24T11:59:51.102608+0000","last_peered":"2024-03-24T11:59:51.102608+0000","last_clean":"2024-03-24T11:59:51.102608+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T11:59:51.102608+0000","last_undegraded":"2024-03-24T11:59:51.102608+0000","last_fullsized":"2024-03-24T11:59:51.102608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:40.772729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:57:40.772729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102316+0000","last_change":"2024-03-24T11:58:07.158615+0000","last_active":"2024-03-24T11:59:51.102316+0000","last_peered":"2024-03-24T11:59:51.102316+0000","last_clean":"2024-03-24T11:59:51.102316+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T11:59:51.102316+0000","last_undegraded":"2024-03-24T11:59:51.102316+0000","last_fullsized":"2024-03-24T11:59:51.102316+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.158533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.158533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"178'16","reported_seq":304,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.097994+0000","last_change":"2024-03-24T11:59:18.653135+0000","last_active":"2024-03-24T11:59:51.097994+0000","last_peered":"2024-03-24T11:59:51.097994+0000","last_clean":"2024-03-24T11:59:51.097994+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T11:59:51.097994+0000","last_undegraded":"2024-03-24T11:59:51.097994+0000","last_fullsized":"2024-03-24T11:59:51.097994+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'16","last_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T11:59:18.653047+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:38.949524+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":306,"reported_epoch":188,"state":"active+undersized+remapped","last_fresh":"2024-03-24T11:59:51.770850+0000","last_change":"2024-03-24T11:59:51.770426+0000","last_active":"2024-03-24T11:59:51.770850+0000","last_peered":"2024-03-24T11:59:51.770850+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T11:59:51.770426+0000","last_became_peered":"2024-03-24T11:59:51.770426+0000","last_unstale":"2024-03-24T11:59:51.770850+0000","last_undegraded":"2024-03-24T11:59:51.770850+0000","last_fullsized":"2024-03-24T11:59:51.754939+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[0,6],"avail_no_missing":["0","2","5","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"187'5160","reported_seq":5210,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.225518+0000","last_change":"2024-03-24T11:59:19.102218+0000","last_active":"2024-03-24T11:59:51.225518+0000","last_peered":"2024-03-24T11:59:51.225518+0000","last_clean":"2024-03-24T11:59:51.225518+0000","last_became_active":"2024-03-24T11:59:19.096884+0000","last_became_peered":"2024-03-24T11:59:19.096884+0000","last_unstale":"2024-03-24T11:59:51.225518+0000","last_undegraded":"2024-03-24T11:59:51.225518+0000","last_fullsized":"2024-03-24T11:59:51.225518+0000","mapping_epoch":178,"log_start":"175'100","ondisk_log_start":"175'100","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:58.719085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338165760,"num_objects":5160,"num_object_clones":0,"num_object_copies":15480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":330240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101755+0000","last_change":"2024-03-24T11:59:07.943563+0000","last_active":"2024-03-24T11:59:51.101755+0000","last_peered":"2024-03-24T11:59:51.101755+0000","last_clean":"2024-03-24T11:59:51.101755+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T11:59:51.101755+0000","last_undegraded":"2024-03-24T11:59:51.101755+0000","last_fullsized":"2024-03-24T11:59:51.101755+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.943432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:08.936009+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":379,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.098222+0000","last_change":"2024-03-24T11:58:59.829483+0000","last_active":"2024-03-24T11:59:51.098222+0000","last_peered":"2024-03-24T11:59:51.098222+0000","last_clean":"2024-03-24T11:59:51.098222+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T11:59:51.098222+0000","last_undegraded":"2024-03-24T11:59:51.098222+0000","last_fullsized":"2024-03-24T11:59:51.098222+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:58:59.829360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:59.829360+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:24.491446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.111100+0000","last_change":"2024-03-24T11:59:49.021795+0000","last_active":"2024-03-24T11:59:51.111100+0000","last_peered":"2024-03-24T11:59:51.111100+0000","last_clean":"2024-03-24T11:59:51.111100+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T11:59:51.111100+0000","last_undegraded":"2024-03-24T11:59:51.111100+0000","last_fullsized":"2024-03-24T11:59:51.111100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:49.021673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:49.021673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:50.973017+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"130'22","reported_seq":313,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.111347+0000","last_change":"2024-03-24T11:59:09.356719+0000","last_active":"2024-03-24T11:59:51.111347+0000","last_peered":"2024-03-24T11:59:51.111347+0000","last_clean":"2024-03-24T11:59:51.111347+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T11:59:51.111347+0000","last_undegraded":"2024-03-24T11:59:51.111347+0000","last_fullsized":"2024-03-24T11:59:51.111347+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'22","last_scrub_stamp":"2024-03-24T11:59:09.356251+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.356251+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:36.899318+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":538,"reported_epoch":187,"state":"remapped+peering","last_fresh":"2024-03-24T11:59:51.102340+0000","last_change":"2024-03-24T11:59:51.102340+0000","last_active":"2024-03-24T11:59:50.078251+0000","last_peered":"2024-03-24T11:59:49.091128+0000","last_clean":"2024-03-24T11:59:49.091128+0000","last_became_active":"2024-03-24T11:59:49.089372+0000","last_became_peered":"2024-03-24T11:59:49.089372+0000","last_unstale":"2024-03-24T11:59:51.102340+0000","last_undegraded":"2024-03-24T11:59:51.102340+0000","last_fullsized":"2024-03-24T11:59:51.102340+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"188'5094","reported_seq":5140,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.974693+0000","last_change":"2024-03-24T11:58:42.822374+0000","last_active":"2024-03-24T11:59:51.974693+0000","last_peered":"2024-03-24T11:59:51.974693+0000","last_clean":"2024-03-24T11:59:51.974693+0000","last_became_active":"2024-03-24T11:58:42.821264+0000","last_became_peered":"2024-03-24T11:58:42.821264+0000","last_unstale":"2024-03-24T11:59:51.974693+0000","last_undegraded":"2024-03-24T11:59:51.974693+0000","last_fullsized":"2024-03-24T11:59:51.974693+0000","mapping_epoch":170,"log_start":"175'534","ondisk_log_start":"175'534","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333840384,"num_objects":5094,"num_object_clones":0,"num_object_copies":15282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":326016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.098321+0000","last_change":"2024-03-24T11:57:57.650759+0000","last_active":"2024-03-24T11:59:51.098321+0000","last_peered":"2024-03-24T11:59:51.098321+0000","last_clean":"2024-03-24T11:59:51.098321+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T11:59:51.098321+0000","last_undegraded":"2024-03-24T11:59:51.098321+0000","last_fullsized":"2024-03-24T11:59:51.098321+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T11:57:57.650631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:57:57.650631+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":378,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.108380+0000","last_change":"2024-03-24T11:58:12.887923+0000","last_active":"2024-03-24T11:59:51.108380+0000","last_peered":"2024-03-24T11:59:51.108380+0000","last_clean":"2024-03-24T11:59:51.108380+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T11:59:51.108380+0000","last_undegraded":"2024-03-24T11:59:51.108380+0000","last_fullsized":"2024-03-24T11:59:51.108380+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:58:12.887826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:12.887826+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.101711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":30,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.751858+0000","last_change":"2024-03-24T11:59:19.116929+0000","last_active":"2024-03-24T11:59:51.751858+0000","last_peered":"2024-03-24T11:59:51.751858+0000","last_clean":"2024-03-24T11:59:51.751858+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T11:59:51.751858+0000","last_undegraded":"2024-03-24T11:59:51.751858+0000","last_fullsized":"2024-03-24T11:59:51.751858+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:10.932134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_clean_scrub_stamp":"2024-03-24T11:58:10.932134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"130'15","reported_seq":300,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.095539+0000","last_change":"2024-03-24T11:58:54.798236+0000","last_active":"2024-03-24T11:59:51.095539+0000","last_peered":"2024-03-24T11:59:51.095539+0000","last_clean":"2024-03-24T11:59:51.095539+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T11:59:51.095539+0000","last_undegraded":"2024-03-24T11:59:51.095539+0000","last_fullsized":"2024-03-24T11:59:51.095539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'15","last_scrub_stamp":"2024-03-24T11:58:54.798112+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T11:58:54.798112+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:14.552204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":306,"reported_epoch":188,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-24T11:59:51.771750+0000","last_change":"2024-03-24T11:59:51.771750+0000","last_active":"2024-03-24T11:59:51.771750+0000","last_peered":"2024-03-24T11:59:51.771750+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T11:59:51.771400+0000","last_became_peered":"2024-03-24T11:59:51.771400+0000","last_unstale":"2024-03-24T11:59:51.771750+0000","last_undegraded":"2024-03-24T11:59:51.754019+0000","last_fullsized":"2024-03-24T11:59:51.753123+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.128250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[0,2],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,2","objects":-1},{"shards":"0,6,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"187'5196","reported_seq":5248,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.760234+0000","last_change":"2024-03-24T11:59:19.103634+0000","last_active":"2024-03-24T11:59:51.760234+0000","last_peered":"2024-03-24T11:59:51.760234+0000","last_clean":"2024-03-24T11:59:51.760234+0000","last_became_active":"2024-03-24T11:59:19.098207+0000","last_became_peered":"2024-03-24T11:59:19.098207+0000","last_unstale":"2024-03-24T11:59:51.760234+0000","last_undegraded":"2024-03-24T11:59:51.760234+0000","last_fullsized":"2024-03-24T11:59:51.760234+0000","mapping_epoch":178,"log_start":"175'383","ondisk_log_start":"175'383","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.248098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340525056,"num_objects":5196,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":332544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.104875+0000","last_change":"2024-03-24T11:59:05.050526+0000","last_active":"2024-03-24T11:59:51.104875+0000","last_peered":"2024-03-24T11:59:51.104875+0000","last_clean":"2024-03-24T11:59:51.104875+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T11:59:51.104875+0000","last_undegraded":"2024-03-24T11:59:51.104875+0000","last_fullsized":"2024-03-24T11:59:51.104875+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:05.050396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.050396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:15.630214+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":29,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.095567+0000","last_change":"2024-03-24T11:59:19.100322+0000","last_active":"2024-03-24T11:59:51.095567+0000","last_peered":"2024-03-24T11:59:51.095567+0000","last_clean":"2024-03-24T11:59:51.095567+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T11:59:51.095567+0000","last_undegraded":"2024-03-24T11:59:51.095567+0000","last_fullsized":"2024-03-24T11:59:51.095567+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:08.349033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.349033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:18.291266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":276,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.104887+0000","last_change":"2024-03-24T11:59:16.062490+0000","last_active":"2024-03-24T11:59:51.104887+0000","last_peered":"2024-03-24T11:59:51.104887+0000","last_clean":"2024-03-24T11:59:51.104887+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T11:59:51.104887+0000","last_undegraded":"2024-03-24T11:59:51.104887+0000","last_fullsized":"2024-03-24T11:59:51.104887+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:59:16.062361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.062361+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:28.948471+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"116'25","reported_seq":316,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.097147+0000","last_change":"2024-03-24T11:59:06.683554+0000","last_active":"2024-03-24T11:59:51.097147+0000","last_peered":"2024-03-24T11:59:51.097147+0000","last_clean":"2024-03-24T11:59:51.097147+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T11:59:51.097147+0000","last_undegraded":"2024-03-24T11:59:51.097147+0000","last_fullsized":"2024-03-24T11:59:51.097147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'25","last_scrub_stamp":"2024-03-24T11:59:06.683322+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T11:57:11.800708+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.683322+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:07.986959+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"187'5101","reported_seq":5147,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.753773+0000","last_change":"2024-03-24T11:58:42.913010+0000","last_active":"2024-03-24T11:59:51.753773+0000","last_peered":"2024-03-24T11:59:51.753773+0000","last_clean":"2024-03-24T11:59:51.753773+0000","last_became_active":"2024-03-24T11:58:42.912707+0000","last_became_peered":"2024-03-24T11:58:42.912707+0000","last_unstale":"2024-03-24T11:59:51.753773+0000","last_undegraded":"2024-03-24T11:59:51.753773+0000","last_fullsized":"2024-03-24T11:59:51.753773+0000","mapping_epoch":170,"log_start":"175'300","ondisk_log_start":"175'300","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4801,"log_dups_size":0,"ondisk_log_size":4801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:54.222840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334299136,"num_objects":5101,"num_object_clones":0,"num_object_copies":15303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5101,"num_write_kb":326464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.102046+0000","last_change":"2024-03-24T11:59:19.138247+0000","last_active":"2024-03-24T11:59:51.102046+0000","last_peered":"2024-03-24T11:59:51.102046+0000","last_clean":"2024-03-24T11:59:51.102046+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T11:59:51.102046+0000","last_undegraded":"2024-03-24T11:59:51.102046+0000","last_fullsized":"2024-03-24T11:59:51.102046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:19.138097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T11:59:19.138097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:42.801466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105117+0000","last_change":"2024-03-24T11:58:09.829306+0000","last_active":"2024-03-24T11:59:51.105117+0000","last_peered":"2024-03-24T11:59:51.105117+0000","last_clean":"2024-03-24T11:59:51.105117+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T11:59:51.105117+0000","last_undegraded":"2024-03-24T11:59:51.105117+0000","last_fullsized":"2024-03-24T11:59:51.105117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:09.829226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:09.829226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.102183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":286,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.109816+0000","last_change":"2024-03-24T11:57:57.819681+0000","last_active":"2024-03-24T11:59:51.109816+0000","last_peered":"2024-03-24T11:59:51.109816+0000","last_clean":"2024-03-24T11:59:51.109816+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T11:59:51.109816+0000","last_undegraded":"2024-03-24T11:59:51.109816+0000","last_fullsized":"2024-03-24T11:59:51.109816+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:57.819565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T11:57:57.819565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.101743+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":310,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.101424+0000","last_change":"2024-03-24T11:59:00.822687+0000","last_active":"2024-03-24T11:59:51.101424+0000","last_peered":"2024-03-24T11:59:51.101424+0000","last_clean":"2024-03-24T11:59:51.101424+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T11:59:51.101424+0000","last_undegraded":"2024-03-24T11:59:51.101424+0000","last_fullsized":"2024-03-24T11:59:51.101424+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:59:00.822562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:00.822562+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:04.641919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"188'5197","reported_seq":5308,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.975574+0000","last_change":"2024-03-24T11:59:50.493313+0000","last_active":"2024-03-24T11:59:51.975574+0000","last_peered":"2024-03-24T11:59:51.975574+0000","last_clean":"2024-03-24T11:59:51.975574+0000","last_became_active":"2024-03-24T11:59:19.097645+0000","last_became_peered":"2024-03-24T11:59:19.097645+0000","last_unstale":"2024-03-24T11:59:51.975574+0000","last_undegraded":"2024-03-24T11:59:51.975574+0000","last_fullsized":"2024-03-24T11:59:51.975574+0000","mapping_epoch":178,"log_start":"175'635","ondisk_log_start":"175'635","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'5091","last_scrub_stamp":"2024-03-24T11:59:50.492994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:59:50.492994+0000","objects_scrubbed":4998,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:08.884073+0000","scrub_duration":2205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340590592,"num_objects":5197,"num_object_clones":0,"num_object_copies":15591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":332608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":259,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.760679+0000","last_change":"2024-03-24T11:58:05.499256+0000","last_active":"2024-03-24T11:59:51.760679+0000","last_peered":"2024-03-24T11:59:51.760679+0000","last_clean":"2024-03-24T11:59:51.760679+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T11:59:51.760679+0000","last_undegraded":"2024-03-24T11:59:51.760679+0000","last_fullsized":"2024-03-24T11:59:51.760679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:05.498978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:05.498978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105924+0000","last_change":"2024-03-24T11:58:10.853378+0000","last_active":"2024-03-24T11:59:51.105924+0000","last_peered":"2024-03-24T11:59:51.105924+0000","last_clean":"2024-03-24T11:59:51.105924+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T11:59:51.105924+0000","last_undegraded":"2024-03-24T11:59:51.105924+0000","last_fullsized":"2024-03-24T11:59:51.105924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:10.853268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:58:10.853268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.101673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.112051+0000","last_change":"2024-03-24T11:59:19.233121+0000","last_active":"2024-03-24T11:59:51.112051+0000","last_peered":"2024-03-24T11:59:51.112051+0000","last_clean":"2024-03-24T11:59:51.112051+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T11:59:51.112051+0000","last_undegraded":"2024-03-24T11:59:51.112051+0000","last_fullsized":"2024-03-24T11:59:51.112051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:19.233002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T11:59:19.233002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:29.859027+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":291,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.754716+0000","last_change":"2024-03-24T11:59:05.755260+0000","last_active":"2024-03-24T11:59:51.754716+0000","last_peered":"2024-03-24T11:59:51.754716+0000","last_clean":"2024-03-24T11:59:51.754716+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T11:59:51.754716+0000","last_undegraded":"2024-03-24T11:59:51.754716+0000","last_fullsized":"2024-03-24T11:59:51.754716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:59:05.755110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.755110+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:07.357503+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"187'5113","reported_seq":5157,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.207094+0000","last_change":"2024-03-24T11:58:42.972556+0000","last_active":"2024-03-24T11:59:51.207094+0000","last_peered":"2024-03-24T11:59:51.207094+0000","last_clean":"2024-03-24T11:59:51.207094+0000","last_became_active":"2024-03-24T11:58:42.972210+0000","last_became_peered":"2024-03-24T11:58:42.972210+0000","last_unstale":"2024-03-24T11:59:51.207094+0000","last_undegraded":"2024-03-24T11:59:51.207094+0000","last_fullsized":"2024-03-24T11:59:51.207094+0000","mapping_epoch":170,"log_start":"175'100","ondisk_log_start":"175'100","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:57.436309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335085568,"num_objects":5113,"num_object_clones":0,"num_object_copies":15339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5113,"num_write_kb":327232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":264,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.099524+0000","last_change":"2024-03-24T11:57:58.742125+0000","last_active":"2024-03-24T11:59:51.099524+0000","last_peered":"2024-03-24T11:59:51.099524+0000","last_clean":"2024-03-24T11:59:51.099524+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T11:59:51.099524+0000","last_undegraded":"2024-03-24T11:59:51.099524+0000","last_fullsized":"2024-03-24T11:59:51.099524+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:57:58.741990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T11:57:58.741990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":286,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.105476+0000","last_change":"2024-03-24T11:58:07.919554+0000","last_active":"2024-03-24T11:59:51.105476+0000","last_peered":"2024-03-24T11:59:51.105476+0000","last_clean":"2024-03-24T11:59:51.105476+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T11:59:51.105476+0000","last_undegraded":"2024-03-24T11:59:51.105476+0000","last_fullsized":"2024-03-24T11:59:51.105476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:07.919445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:07.919445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.101773+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":297,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.099972+0000","last_change":"2024-03-24T11:57:56.751966+0000","last_active":"2024-03-24T11:59:51.099972+0000","last_peered":"2024-03-24T11:59:51.099972+0000","last_clean":"2024-03-24T11:59:51.099972+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T11:59:51.099972+0000","last_undegraded":"2024-03-24T11:59:51.099972+0000","last_fullsized":"2024-03-24T11:59:51.099972+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:57:56.751874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:57:56.751874+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":378,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-24T11:59:51.104928+0000","last_change":"2024-03-24T11:59:08.941579+0000","last_active":"2024-03-24T11:59:51.104928+0000","last_peered":"2024-03-24T11:59:51.104928+0000","last_clean":"2024-03-24T11:59:51.104928+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T11:59:51.104928+0000","last_undegraded":"2024-03-24T11:59:51.104928+0000","last_fullsized":"2024-03-24T11:59:51.104928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T11:59:08.941456+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T11:59:08.941456+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.941456+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:00:13.592522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5446303744,"num_objects":83104,"num_object_clones":0,"num_object_copies":249312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83104,"num_write_kb":5318656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16317808640,"data_stored":16317808640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78406,"ondisk_log_size":78406,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":536699,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":183,"num_read_kb":271,"num_write":163,"num_write_kb":2197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1610095,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":23,"num_store_stats":6},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005151,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1810700,"kb_used_data":1657340,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91934452,"statfs":{"total":95995035648,"available":94140878848,"internally_reserved":0,"allocated":1697116160,"data_stored":1696681041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":157022666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332198,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797832,"kb_used_data":1644472,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91947320,"statfs":{"total":95995035648,"available":94154055680,"internally_reserved":0,"allocated":1683939328,"data_stored":1683508123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574974,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2549712,"kb_used_data":2301952,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91195440,"statfs":{"total":95995035648,"available":93384130560,"internally_reserved":0,"allocated":2357198848,"data_stored":2356758486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2253080,"kb_used_data":2001160,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91492072,"statfs":{"total":95995035648,"available":93687881728,"internally_reserved":0,"allocated":2049187840,"data_stored":2048746838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411450,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2207516,"kb_used_data":1981960,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":91537636,"statfs":{"total":95995035648,"available":93734539264,"internally_reserved":0,"allocated":2029527040,"data_stored":2029086444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117108,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2257712,"kb_used_data":1987740,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91487440,"statfs":{"total":95995035648,"available":93683138560,"internally_reserved":0,"allocated":2035445760,"data_stored":2034985114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276429255},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920879,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797456,"kb_used_data":1639940,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91947696,"statfs":{"total":95995035648,"available":94154440704,"internally_reserved":0,"allocated":1679298560,"data_stored":1678854137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659242,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2804232,"kb_used_data":2650876,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":90940920,"statfs":{"total":95995035648,"available":93123502080,"internally_reserved":0,"allocated":2714497024,"data_stored":2714060905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2054619136,"data_stored":2054619136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2366504960,"data_stored":2366504960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2040922112,"data_stored":2040922112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2052587520,"data_stored":2052587520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1691746304,"data_stored":1691746304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2720071680,"data_stored":2720071680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1690304512,"data_stored":1690304512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1701052416,"data_stored":1701052416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:59:58.305 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T11:59:58.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_1 pg_num 26 2024-03-24T11:59:58.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:59:58.403 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:58.404 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:58 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:58.404 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:58 smithi064 ceph-mon[30223]: pgmap v325: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 80 MiB/s wr, 1.25k op/s 2024-03-24T11:59:58.405 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:58 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:58.405 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:58 smithi064 ceph-mon[42243]: pgmap v325: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 80 MiB/s wr, 1.25k op/s 2024-03-24T11:59:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:58.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:59:58.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:58.538 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:58.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:58.640 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:58.678 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:58.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:58.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:59:58.822 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:58.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:59:58.828 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 58 16 91562 91546 98.6269 74.375 0.0191979 0.010134 2024-03-24T11:59:58.922 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T11:59:58.923 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T11:59:58.923 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-03-24T11:59:58.923 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T11:59:58.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:58.971 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:58.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:59.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:59:59.109 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:59.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:59:59.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:59.247 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:59.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[30223]: 4.14 scrub starts 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[30223]: 4.14 scrub ok 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[30223]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[30223]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/249906 objects degraded (0.001%), 2 pgs degraded) 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[30223]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[42243]: 4.14 scrub starts 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[42243]: 4.14 scrub ok 2024-03-24T11:59:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T11:59:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T11:59:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[42243]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-24T11:59:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[42243]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/249906 objects degraded (0.001%), 2 pgs degraded) 2024-03-24T11:59:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 11:59:59 smithi064 ceph-mon[42243]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:59:59.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:59:59.389 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:59.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:59:59.491 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:59:59.519 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:59.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:59:59.530 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:59 smithi193 ceph-mon[33593]: 4.14 scrub starts 2024-03-24T11:59:59.530 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:59 smithi193 ceph-mon[33593]: 4.14 scrub ok 2024-03-24T11:59:59.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:59 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T11:59:59.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:59 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T11:59:59.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:59 smithi193 ceph-mon[33593]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-24T11:59:59.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:59 smithi193 ceph-mon[33593]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/249906 objects degraded (0.001%), 2 pgs degraded) 2024-03-24T11:59:59.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 11:59:59 smithi193 ceph-mon[33593]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:59:59.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:59:59.654 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:59.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:59:59.756 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:59:59.796 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T11:59:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:59.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:59:59.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 59 16 92691 92675 98.1512 70.5625 0.0127095 0.0101837 2024-03-24T11:59:59.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:59:59.903 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T11:59:59.942 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T11:59:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:59:59.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:00.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:00.087 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:00.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:00.160 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:00:00.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:00.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:00.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:00.261 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:00:00.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:00.379 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:00.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:00.438 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:00 smithi193 ceph-mon[33593]: 5.12 scrub starts 2024-03-24T12:00:00.438 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:00 smithi193 ceph-mon[33593]: 8.9 scrub starts 2024-03-24T12:00:00.438 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:00 smithi193 ceph-mon[33593]: 5.12 scrub ok 2024-03-24T12:00:00.438 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:00 smithi193 ceph-mon[33593]: pgmap v327: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.02k op/s 2024-03-24T12:00:00.438 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:00 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T12:00:00.438 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:00 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2279704278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:00.439 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:00 smithi193 ceph-mon[33593]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-24T12:00:00.439 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:00 smithi193 ceph-mon[33593]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-24T12:00:00.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:00.510 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:00.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:00.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[30223]: 5.12 scrub starts 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[30223]: 8.9 scrub starts 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[30223]: 5.12 scrub ok 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[30223]: pgmap v327: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.02k op/s 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2279704278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[30223]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[30223]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[42243]: 5.12 scrub starts 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[42243]: 8.9 scrub starts 2024-03-24T12:00:00.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[42243]: 5.12 scrub ok 2024-03-24T12:00:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[42243]: pgmap v327: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.02k op/s 2024-03-24T12:00:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T12:00:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2279704278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[42243]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-24T12:00:00.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:00 smithi064 ceph-mon[42243]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-24T12:00:00.612 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:00.649 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:00.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:00.751 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:00.788 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:00.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:00.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:00:00.830544+0000 min lat: 0.00128588 max lat: 1.02708 avg lat: 0.0102406 2024-03-24T12:00:00.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:00:00.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 60 16 93736 93720 97.6038 65.3125 0.0100961 0.0102406 2024-03-24T12:00:00.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:00.928 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:00.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:01.030 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:01.063 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:01.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:01.166 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:01.198 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:01.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:01.300 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:01.338 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:01.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:01.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:01.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:01.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T12:00:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[30223]: 4.8 scrub starts 2024-03-24T12:00:01.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[30223]: 4.8 scrub ok 2024-03-24T12:00:01.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:00:01.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:00:01.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2279704278' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:00:01.555 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[30223]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T12:00:01.556 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T12:00:01.556 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[42243]: 4.8 scrub starts 2024-03-24T12:00:01.556 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[42243]: 4.8 scrub ok 2024-03-24T12:00:01.556 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:00:01.556 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:00:01.556 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2279704278' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:00:01.557 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:01 smithi064 ceph-mon[42243]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T12:00:01.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:01 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T12:00:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:01 smithi193 ceph-mon[33593]: 4.8 scrub starts 2024-03-24T12:00:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:01 smithi193 ceph-mon[33593]: 4.8 scrub ok 2024-03-24T12:00:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:01 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:00:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:01 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:00:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:01 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2279704278' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:00:01.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:01 smithi193 ceph-mon[33593]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T12:00:01.626 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:01.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:01.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:01.766 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:01.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:01.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 61 16 94858 94842 97.1532 70.125 0.00297218 0.0102833 2024-03-24T12:00:01.869 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:01.900 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:01.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:02.002 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:02.034 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:02.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:02.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:02.173 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:02.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:02.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:02.318 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:02.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:02.419 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:02.421 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:02 smithi193 ceph-mon[33593]: 8.9 scrub ok 2024-03-24T12:00:02.421 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:02 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:00:02.421 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:02 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:00:02.421 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:02 smithi193 ceph-mon[33593]: pgmap v329: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 100 MiB/s wr, 1.58k op/s 2024-03-24T12:00:02.460 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:02.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:02.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:02 smithi064 ceph-mon[30223]: 8.9 scrub ok 2024-03-24T12:00:02.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:02 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:00:02.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:02 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:00:02.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:02 smithi064 ceph-mon[30223]: pgmap v329: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 100 MiB/s wr, 1.58k op/s 2024-03-24T12:00:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:02 smithi064 ceph-mon[42243]: 8.9 scrub ok 2024-03-24T12:00:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:02 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:00:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:02 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:00:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:02 smithi064 ceph-mon[42243]: pgmap v329: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 100 MiB/s wr, 1.58k op/s 2024-03-24T12:00:02.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:02.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:02.614 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:00:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:00:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T12:00:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T12:00:02.614 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:00:02.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:02.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:02.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:02.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 62 15 96058 96043 96.7969 75.0625 0.0108341 0.0103263 2024-03-24T12:00:02.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:02.888 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:02.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:02.989 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:03.027 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:03.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:03.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:03.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:03.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:03.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[42243]: 2.5 deep-scrub starts 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[42243]: 2.5 deep-scrub ok 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[42243]: 5.1e scrub starts 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[42243]: 5.1e scrub ok 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[30223]: 2.5 deep-scrub starts 2024-03-24T12:00:03.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[30223]: 2.5 deep-scrub ok 2024-03-24T12:00:03.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T12:00:03.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T12:00:03.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[30223]: 5.1e scrub starts 2024-03-24T12:00:03.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:03 smithi064 ceph-mon[30223]: 5.1e scrub ok 2024-03-24T12:00:03.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:03.308 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:03.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:03 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T12:00:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:03 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T12:00:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:03 smithi193 ceph-mon[33593]: 2.5 deep-scrub starts 2024-03-24T12:00:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:03 smithi193 ceph-mon[33593]: 2.5 deep-scrub ok 2024-03-24T12:00:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:03 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T12:00:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:03 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T12:00:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:03 smithi193 ceph-mon[33593]: 5.1e scrub starts 2024-03-24T12:00:03.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:03 smithi193 ceph-mon[33593]: 5.1e scrub ok 2024-03-24T12:00:03.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:03.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:00:03.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:03.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:03.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:03.589 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:03.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:03.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:03.730 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:03.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:03.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 63 16 97232 97216 96.4241 73.3125 0.0100638 0.0103663 2024-03-24T12:00:03.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:03.888 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:03.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:03.915 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:00:03.989 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:04.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:04.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:04.176 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:04.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:04.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[30223]: 8.5 scrub starts 2024-03-24T12:00:04.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[30223]: 2.1d deep-scrub starts 2024-03-24T12:00:04.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[30223]: 2.1d deep-scrub ok 2024-03-24T12:00:04.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[30223]: pgmap v330: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-24T12:00:04.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3000056019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:04.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[42243]: 8.5 scrub starts 2024-03-24T12:00:04.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[42243]: 2.1d deep-scrub starts 2024-03-24T12:00:04.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[42243]: 2.1d deep-scrub ok 2024-03-24T12:00:04.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[42243]: pgmap v330: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-24T12:00:04.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:04 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3000056019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:04.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:04.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:04 smithi193 ceph-mon[33593]: 8.5 scrub starts 2024-03-24T12:00:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:04 smithi193 ceph-mon[33593]: 2.1d deep-scrub starts 2024-03-24T12:00:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:04 smithi193 ceph-mon[33593]: 2.1d deep-scrub ok 2024-03-24T12:00:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:04 smithi193 ceph-mon[33593]: pgmap v330: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-24T12:00:04.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:04 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3000056019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:04.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:04.454 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:04.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:04.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:04.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:04.594 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:04.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:04.696 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:04.733 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:04.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:04.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 64 16 98390 98374 96.0483 72.375 0.0156244 0.0104071 2024-03-24T12:00:04.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:04.867 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:04.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:04.969 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:05.001 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:05.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:05.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:00:05.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:05.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:05.287 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:05.333 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:05.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:05.344 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:00:05.344 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:00:05.345 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:00:05.346 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:00:05.346 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:00:05.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:05.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:05 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T12:00:05.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:05 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T12:00:05.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:05.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:05.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:05 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T12:00:05.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:05 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T12:00:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:05 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T12:00:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:05 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T12:00:05.580 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:05.624 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:05.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:05.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:05.760 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:05.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:05.829 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 65 16 99592 99576 95.7263 75.125 0.00834693 0.0104414 2024-03-24T12:00:05.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:05.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:06.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:06.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:06.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:06.176 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:06.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:06.278 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:06.301 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:06 smithi064 ceph-mon[30223]: 8.5 scrub ok 2024-03-24T12:00:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:06 smithi064 ceph-mon[30223]: pgmap v331: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 710 op/s 2024-03-24T12:00:06.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:06 smithi064 ceph-mon[42243]: 8.5 scrub ok 2024-03-24T12:00:06.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:06 smithi064 ceph-mon[42243]: pgmap v331: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 710 op/s 2024-03-24T12:00:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:06.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:06.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:06.444 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:06.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:06.546 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:06.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:06.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:06 smithi193 ceph-mon[33593]: 8.5 scrub ok 2024-03-24T12:00:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:06 smithi193 ceph-mon[33593]: pgmap v331: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 710 op/s 2024-03-24T12:00:06.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:06.718 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:06.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:06.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:06.830 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 66 16 100591 100575 95.2207 62.4375 0.0837546 0.010495 2024-03-24T12:00:06.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:06.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:06.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:07.004 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:07.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:07.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:07.140 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:07.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:07.242 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:07.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:07.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T12:00:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T12:00:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3516152344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:00:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-mon[30223]: from='client.35111 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:00:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T12:00:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T12:00:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3516152344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:00:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-mon[42243]: from='client.35111 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:00:07.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:07.417 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:07.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:07.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:07.547 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:07.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:07.584 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:00:07.584 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:00:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:07 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T12:00:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:07 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T12:00:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:07 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3516152344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:00:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:07 smithi193 ceph-mon[33593]: from='client.35111 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:00:07.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:00:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:00:07.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:07.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:07.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:07.709 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:00:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:00:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:07.788 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:07.789 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 8 pg_num to 26 2024-03-24T12:00:07.827 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:07.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:07.830 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 67 16 101588 101572 94.7295 62.3125 0.0109647 0.0105475 2024-03-24T12:00:07.929 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:07.967 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:07.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:08.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:00:08.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:08.105 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:08.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:08.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:08.263 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 2.1e deep-scrub starts 2024-03-24T12:00:08.263 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 2.1e deep-scrub ok 2024-03-24T12:00:08.263 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 8.2 scrub starts 2024-03-24T12:00:08.263 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T12:00:08.263 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T12:00:08.263 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 6.1c deep-scrub starts 2024-03-24T12:00:08.263 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 6.1c deep-scrub ok 2024-03-24T12:00:08.263 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:08.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: pgmap v332: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-24T12:00:08.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 8.1 deep-scrub starts 2024-03-24T12:00:08.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: from='client.35111 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:00:08.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T12:00:08.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:00:08.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:08 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:00:08.269 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:08.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:08.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 2.1e deep-scrub starts 2024-03-24T12:00:08.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 2.1e deep-scrub ok 2024-03-24T12:00:08.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 8.2 scrub starts 2024-03-24T12:00:08.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T12:00:08.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T12:00:08.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 6.1c deep-scrub starts 2024-03-24T12:00:08.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 6.1c deep-scrub ok 2024-03-24T12:00:08.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:08.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: pgmap v332: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 8.1 deep-scrub starts 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: from='client.35111 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 2.1e deep-scrub starts 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 2.1e deep-scrub ok 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 8.2 scrub starts 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T12:00:08.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 6.1c deep-scrub starts 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 6.1c deep-scrub ok 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: pgmap v332: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 8.1 deep-scrub starts 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: from='client.35111 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:00:08.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:08 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:00:08.309 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:00:08.371 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:08.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:08.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:08.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:08.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:08.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:08.669 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:00:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:08.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:08.790 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:08.829 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:08.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:08.830 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 68 16 102515 102499 94.1884 57.9375 0.00802424 0.0106119 2024-03-24T12:00:08.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:08.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:09.065 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:09.101 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:09.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:09.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:09.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:09.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:09 smithi064 ceph-mon[30223]: 8.2 scrub ok 2024-03-24T12:00:09.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:09 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T12:00:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:09 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T12:00:09.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:09 smithi064 ceph-mon[42243]: 8.2 scrub ok 2024-03-24T12:00:09.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:09 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T12:00:09.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:09 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T12:00:09.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:09.387 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:09.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:09.488 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:09.508 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:09.520 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:09.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:09 smithi193 ceph-mon[33593]: 8.2 scrub ok 2024-03-24T12:00:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:09 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T12:00:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:09 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T12:00:09.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:09.661 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:09.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:09.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:09.798 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:09.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:09.831 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 69 16 103441 103425 93.6612 57.875 0.0218023 0.0106709 2024-03-24T12:00:09.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:09.933 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:09.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:10.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:10.081 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:10.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:10.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:10.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:10.231 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:00:10.290 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T12:00:10.290 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T12:00:10.290 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[30223]: pgmap v334: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-24T12:00:10.290 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:00:10.290 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/947602230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:00:10.290 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T12:00:10.290 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T12:00:10.291 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[42243]: pgmap v334: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-24T12:00:10.291 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:00:10.291 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/947602230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:00:10.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:10.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:10.469 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:10 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T12:00:10.469 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:10 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T12:00:10.469 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:10 smithi193 ceph-mon[33593]: pgmap v334: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-24T12:00:10.469 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:10 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:00:10.469 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/947602230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:00:10.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:10.508 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:10.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:10.604 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:00:10.604 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:00:10.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:10.634 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:10.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:10.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:10.761 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:10.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:10.831 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 70 16 103835 103819 92.6751 24.625 0.0328889 0.0106964 2024-03-24T12:00:10.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:10.887 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:10.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:10.988 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:11.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:11.012 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:11.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:11.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:11.136 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:11.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:11.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:11.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:11.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:11.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[42243]: 4.0 deep-scrub starts 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[42243]: 4.0 deep-scrub ok 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/947602230' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[42243]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[30223]: 4.0 deep-scrub starts 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[30223]: 4.0 deep-scrub ok 2024-03-24T12:00:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T12:00:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T12:00:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:00:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/947602230' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:00:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:11 smithi064 ceph-mon[30223]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T12:00:11.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:11.421 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:11.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:11.523 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:11.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:11 smithi193 ceph-mon[33593]: 4.0 deep-scrub starts 2024-03-24T12:00:11.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:11 smithi193 ceph-mon[33593]: 4.0 deep-scrub ok 2024-03-24T12:00:11.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:11 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T12:00:11.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:11 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T12:00:11.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:11 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:11.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:00:11.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/947602230' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:00:11.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:11 smithi193 ceph-mon[33593]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T12:00:11.555 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:11.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:11.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:00:11.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:11.694 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:11.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:11.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:11.822 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:11.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:11.831 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 71 16 104644 104628 92.0819 50.5625 0.00319749 0.0108475 2024-03-24T12:00:11.912 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:00:11.924 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:11.949 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:11.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:12.051 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:12.075 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:12.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:12.173 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:00:12.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:12.193 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:00:12.200 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:12.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:12.214 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:00:12.234 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:00:12.255 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:00:12.275 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:00:12.296 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:00:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: 8.f scrub starts 2024-03-24T12:00:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T12:00:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: pgmap v337: 168 pgs: 1 activating+degraded, 3 peering, 8 unknown, 156 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s; 1/303888 objects degraded (0.000%) 2024-03-24T12:00:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1465267429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: 8.f scrub starts 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: pgmap v337: 168 pgs: 1 activating+degraded, 3 peering, 8 unknown, 156 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s; 1/303888 objects degraded (0.000%) 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T12:00:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:12 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1465267429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:12.303 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:12.316 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:00:12.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:00:12.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:00:12.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:00:12.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:00:12.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:00:12.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:00:12.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:00:12.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:00:12.326 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:12.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: 8.f scrub starts 2024-03-24T12:00:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T12:00:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: pgmap v337: 168 pgs: 1 activating+degraded, 3 peering, 8 unknown, 156 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s; 1/303888 objects degraded (0.000%) 2024-03-24T12:00:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:00:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:00:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T12:00:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:12 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1465267429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:12.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:12.452 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:12.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:12.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:12.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:12.578 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:12.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:12.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:12.702 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:12.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:12.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:12.831 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 72 16 104644 104628 90.8032 0 - 0.0108475 2024-03-24T12:00:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:12.832 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:12.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:12.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:12.957 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:12.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:13.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:00:13.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:13.100 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:13.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:13.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:13.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:13.227 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:13.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:13.281 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:13.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:13.355 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:13.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:13.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:13.482 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:13.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:13.493 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:13 smithi193 ceph-mon[33593]: Health check failed: Degraded data redundancy: 1/303888 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:00:13.493 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:13 smithi193 ceph-mon[33593]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T12:00:13.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:13 smithi064 ceph-mon[30223]: Health check failed: Degraded data redundancy: 1/303888 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:00:13.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:13 smithi064 ceph-mon[30223]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T12:00:13.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:13 smithi064 ceph-mon[42243]: Health check failed: Degraded data redundancy: 1/303888 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:00:13.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:13 smithi064 ceph-mon[42243]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T12:00:13.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:13.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:13.708 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:13.733 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:13.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:13.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 73 16 104645 104629 89.5603 0.03125 2.0721 0.0108672 2024-03-24T12:00:13.835 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:13.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:13.874 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:13.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:13.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:14.019 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:14.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:14.080 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:00:14.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:00:14.080 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-24T12:00:14.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-24T12:00:14.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:14.179 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:14.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:14.281 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:14.334 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:14.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:14.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:14.468 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:14.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:14.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:14.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:14.640 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:14.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:14.650 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:14.742 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:14.781 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:14.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:14.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 74 16 105790 105774 89.3169 71.5625 0.013187 0.0111912 2024-03-24T12:00:14.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:14.927 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:14.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:14.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: pgmap v340: 168 pgs: 1 activating+degraded, 1 activating, 3 peering, 6 unknown, 1 active+recovery_wait+degraded, 156 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/264903 objects degraded (0.000%); 32 KiB/s, 0 objects/s recovering 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6665 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 197'6534 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6503 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6347 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 197'6497 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 197'6395 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:14.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6563 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6347 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6497 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 197'6503 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:14.953 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:14 smithi193 ceph-mon[33593]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:15.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: pgmap v340: 168 pgs: 1 activating+degraded, 1 activating, 3 peering, 6 unknown, 1 active+recovery_wait+degraded, 156 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/264903 objects degraded (0.000%); 32 KiB/s, 0 objects/s recovering 2024-03-24T12:00:15.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6665 2024-03-24T12:00:15.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 197'6534 2024-03-24T12:00:15.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6503 2024-03-24T12:00:15.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6347 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 197'6497 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 197'6395 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6563 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6347 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6497 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 197'6503 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:15.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[30223]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: pgmap v340: 168 pgs: 1 activating+degraded, 1 activating, 3 peering, 6 unknown, 1 active+recovery_wait+degraded, 156 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/264903 objects degraded (0.000%); 32 KiB/s, 0 objects/s recovering 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6665 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 197'6534 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6503 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6347 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 197'6497 2024-03-24T12:00:15.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 197'6395 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 197'6563 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6347 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6497 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 197'6503 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6646 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 197'6366 2024-03-24T12:00:15.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:14 smithi064 ceph-mon[42243]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 197'6831 2024-03-24T12:00:15.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:15.076 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:15.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:15.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:15.210 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:15.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:15.228 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:00:15.228 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:00:15.228 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T12:00:15.228 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T12:00:15.228 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:00:15.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:15.365 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:15.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:15.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:15.512 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:15.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:15.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:15.663 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:15.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:15.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:15.808 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:15.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:15.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 75 16 106830 106814 88.9926 65 0.0120234 0.011232 2024-03-24T12:00:15.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:15.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:00:15.954 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:15.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:16.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:16.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:16.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:16.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:16.256 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:16.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:16.334 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:00:16.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:16.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:16.435 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:16.502 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:16.538 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:16.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:16.639 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:16.671 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:16.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:16.772 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:16.804 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:00:16.805 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:16.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:16.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 76 15 107878 107863 88.6841 65.5625 0.0330814 0.0112704 2024-03-24T12:00:16.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:16.957 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:16.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:17.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:16 smithi064 ceph-mon[30223]: pgmap v342: 168 pgs: 1 activating+degraded, 1 activating, 3 peering, 6 unknown, 1 active+recovery_wait+degraded, 156 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/264903 objects degraded (0.000%); 32 KiB/s, 0 objects/s recovering 2024-03-24T12:00:17.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:16 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4155063872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:17.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:16 smithi064 ceph-mon[30223]: from='client.35129 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:17.026 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:16 smithi064 ceph-mon[42243]: pgmap v342: 168 pgs: 1 activating+degraded, 1 activating, 3 peering, 6 unknown, 1 active+recovery_wait+degraded, 156 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/264903 objects degraded (0.000%); 32 KiB/s, 0 objects/s recovering 2024-03-24T12:00:17.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:16 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4155063872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:17.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:16 smithi064 ceph-mon[42243]: from='client.35129 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:17.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:17.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:16 smithi193 ceph-mon[33593]: pgmap v342: 168 pgs: 1 activating+degraded, 1 activating, 3 peering, 6 unknown, 1 active+recovery_wait+degraded, 156 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/264903 objects degraded (0.000%); 32 KiB/s, 0 objects/s recovering 2024-03-24T12:00:17.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:16 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4155063872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:17.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:16 smithi193 ceph-mon[33593]: from='client.35129 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:00:17.089 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:17.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:17.191 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:17.228 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:17.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:17.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:17.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:17.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:17.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:17.503 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:17.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:17.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:00:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:00:17.605 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:17.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:17.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:17.646 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:00:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:00:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:17.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:17.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:17.780 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:17.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:17.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 77 16 108928 108912 88.3838 65.5625 0.00359927 0.0113087 2024-03-24T12:00:17.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T12:00:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T12:00:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T12:00:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T12:00:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T12:00:17.907 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T12:00:17.907 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T12:00:17.907 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:00:17.907 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T12:00:17.907 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T12:00:17.907 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T12:00:17.907 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T12:00:17.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:17 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:17.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:17 smithi064 ceph-mon[30223]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:00:17.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:17 smithi064 ceph-mon[30223]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T12:00:17.908 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:17 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:17.908 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:17 smithi064 ceph-mon[42243]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:00:17.908 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:17 smithi064 ceph-mon[42243]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T12:00:17.913 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:17.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:17.954 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:17 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:17.954 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:17 smithi193 ceph-mon[33593]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:00:17.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:17 smithi193 ceph-mon[33593]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T12:00:18.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:18.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:18.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:18.184 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:18.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:18.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:18.330 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:18.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:18.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:18.471 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:18.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:18.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:18.613 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:18.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:18.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:18.757 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:18.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:18.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 78 16 110062 110046 88.1593 70.875 0.0181197 0.0113384 2024-03-24T12:00:18.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:18.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:18.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:18 smithi193 ceph-mon[33593]: pgmap v344: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 927 B/s rd, 61 MiB/s wr, 970 op/s; 61958/323589 objects misplaced (19.147%); 10 MiB/s, 163 objects/s recovering 2024-03-24T12:00:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:18 smithi193 ceph-mon[33593]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:00:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:18 smithi193 ceph-mon[33593]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/264903 objects degraded (0.000%), 2 pgs degraded) 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221103 7.11 7:8c18531b:::benchmark_data_smithi064_87013_object100146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.064318+0000", 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.913729346, 2024-03-24T12:00:18.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.992514813, 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221103 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064318+0000", 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064318+0000", 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064320+0000", 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8020000000000001e-06 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064325+0000", 2024-03-24T12:00:18.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7200000000000003e-06 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064335+0000", 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6320000000000003e-06 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064342+0000", 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7100000000000001e-06 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064389+0000", 2024-03-24T12:00:18.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7324999999999998e-05 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064395+0000", 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6459999999999998e-06 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.052960+0000", 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.98856554500000005 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:18.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.053057+0000", 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6861999999999995e-05 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.053258+0000", 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020089399999999999 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.054391+0000", 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0011329230000000001 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056626+0000", 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022347259999999998 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056731+0000", 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010556 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056792+0000", 2024-03-24T12:00:18.987 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0520999999999999e-05 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056833+0000", 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0946999999999999e-05 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221132 7.11 7:8da10ffd:::benchmark_data_smithi064_87013_object100175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:18.988 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.088744+0000", 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.88930337599999, 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.97044713199999999, 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221132 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088744+0000", 2024-03-24T12:00:18.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088744+0000", 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088746+0000", 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7089999999999998e-06 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088752+0000", 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2870000000000004e-06 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088763+0000", 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1039000000000001e-05 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088771+0000", 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0250000000000002e-06 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088853+0000", 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1977000000000005e-05 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088863+0000", 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9310000000000003e-06 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.054709+0000", 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96584587899999996 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.054807+0000", 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8837000000000003e-05 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.054994+0000", 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000186906 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056868+0000", 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018733840000000001 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056880+0000", 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1989e-05 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059102+0000", 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022221060000000002 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059145+0000", 2024-03-24T12:00:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3281000000000001e-05 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059191+0000", 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5782000000000003e-05 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221143 7.11 7:8a258abe:::benchmark_data_smithi064_87013_object100186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.096184+0000", 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.88186315300001, 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96315127599999995, 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221143 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096184+0000", 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096184+0000", 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096186+0000", 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6729999999999999e-06 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096190+0000", 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3560000000000003e-06 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096201+0000", 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.078e-05 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096209+0000", 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3340000000000002e-06 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096251+0000", 2024-03-24T12:00:18.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1390999999999997e-05 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096258+0000", 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3130000000000001e-06 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.055724+0000", 2024-03-24T12:00:18.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95946589400000004 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.055837+0000", 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011359800000000001 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.055947+0000", 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000109696 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059053+0000", 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003105694 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059236+0000", 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018369099999999999 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059272+0000", 2024-03-24T12:00:19.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5809999999999998e-05 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059304+0000", 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1658000000000001e-05 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059335+0000", 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1387999999999998e-05 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.002 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221150 7.11 7:8e576a6c:::benchmark_data_smithi064_87013_object100193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.100516+0000", 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.877531382, 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95903160200000004, 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221150 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.003 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100516+0000", 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100516+0000", 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100518+0000", 2024-03-24T12:00:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5459999999999998e-06 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100523+0000", 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0699999999999997e-06 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100536+0000", 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3057999999999999e-05 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100544+0000", 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9580000000000007e-06 2024-03-24T12:00:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100608+0000", 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3167e-05 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100616+0000", 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0539999999999995e-06 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056914+0000", 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.956298431 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.057046+0000", 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000132352 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.057209+0000", 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016295099999999999 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059201+0000", 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00199175 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059379+0000", 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000178207 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059444+0000", 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4416000000000001e-05 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.009 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059501+0000", 2024-03-24T12:00:19.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:19.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7535999999999999e-05 2024-03-24T12:00:19.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.010 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059547+0000", 2024-03-24T12:00:19.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6106000000000002e-05 2024-03-24T12:00:19.010 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221183 7.11 7:8a724717:::benchmark_data_smithi064_87013_object100226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.130532+0000", 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.847515399, 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92921261700000002, 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.011 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221183 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130532+0000", 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130532+0000", 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130534+0000", 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9659999999999999e-06 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130539+0000", 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1340000000000003e-06 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130551+0000", 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2312e-05 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130559+0000", 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5480000000000004e-06 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130614+0000", 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5717999999999997e-05 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130622+0000", 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1709999999999996e-06 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.058069+0000", 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92744775800000001 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.058145+0000", 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5464999999999997e-05 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.058320+0000", 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00017536400000000001 2024-03-24T12:00:19.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059343+0000", 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001023045 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059610+0000", 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00026698899999999997 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059652+0000", 2024-03-24T12:00:19.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1705000000000001e-05 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059706+0000", 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4042000000000002e-05 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059744+0000", 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8399999999999998e-05 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221191 7.11 7:8949aaf4:::benchmark_data_smithi064_87013_object100234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.140691+0000", 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.83735655800001, 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92540493999999995, 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221191 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140691+0000", 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140691+0000", 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140692+0000", 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.395e-06 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140697+0000", 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6800000000000001e-06 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140714+0000", 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6946999999999999e-05 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140721+0000", 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8970000000000008e-06 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140779+0000", 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7873e-05 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140798+0000", 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8722000000000001e-05 2024-03-24T12:00:19.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059783+0000", 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91898496299999999 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059873+0000", 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9926999999999995e-05 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059994+0000", 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012076 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065889+0000", 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0058947899999999996 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065905+0000", 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6121e-05 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.023 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065951+0000", 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6213e-05 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066051+0000", 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000100466 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066096+0000", 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4186e-05 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.024 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221248 7.11 7:8cc49d56:::benchmark_data_smithi064_87013_object100291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.209849+0000", 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.76819793499999, 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85784487899999995, 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221248 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209849+0000", 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209849+0000", 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209851+0000", 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.671e-06 2024-03-24T12:00:19.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209855+0000", 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4320000000000001e-06 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209866+0000", 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0913999999999999e-05 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209873+0000", 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1300000000000003e-06 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209919+0000", 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5596999999999997e-05 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209927+0000", 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5789999999999998e-06 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.060800+0000", 2024-03-24T12:00:19.028 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85087338700000004 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.060874+0000", 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4499000000000006e-05 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.061587+0000", 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00071300599999999997 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065910+0000", 2024-03-24T12:00:19.029 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0043224789999999997 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066149+0000", 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000238605 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067582+0000", 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014336900000000001 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067644+0000", 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.143e-05 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067694+0000", 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0460000000000001e-05 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.031 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221301 7.11 7:8e59ab61:::benchmark_data_smithi064_87013_object100344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.269297+0000", 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.708750528, 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.79857248599999997, 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221301 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.032 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269297+0000", 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269297+0000", 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269298+0000", 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5600000000000001e-06 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.033 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269302+0000", 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0990000000000001e-06 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269311+0000", 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5210000000000002e-06 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269316+0000", 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1800000000000004e-06 2024-03-24T12:00:19.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269366+0000", 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0099999999999998e-05 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269376+0000", 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0369e-05 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.062706+0000", 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.79332951100000004 2024-03-24T12:00:19.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.062817+0000", 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000110881 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063002+0000", 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018522999999999999 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.036 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065915+0000", 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0029131719999999999 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067525+0000", 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001609929 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067753+0000", 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022786799999999999 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.037 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067825+0000", 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1600000000000006e-05 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067869+0000", 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4465999999999997e-05 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221321 7.11 7:8c46042a:::benchmark_data_smithi064_87013_object100364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.288567+0000", 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.68948057399999, 2024-03-24T12:00:19.038 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.77949712999999998, 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221321 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288567+0000", 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.039 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288567+0000", 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288568+0000", 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9149999999999999e-06 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288576+0000", 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7409999999999992e-06 2024-03-24T12:00:19.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288590+0000", 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3667e-05 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288598+0000", 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4530000000000002e-06 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.041 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288641+0000", 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2713000000000003e-05 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288649+0000", 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5680000000000002e-06 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.064069+0000", 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.77542028200000002 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.042 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.064204+0000", 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013507999999999999 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.064429+0000", 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022490699999999999 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066106+0000", 2024-03-24T12:00:19.043 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001677109 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067920+0000", 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018139650000000001 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067965+0000", 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4904000000000002e-05 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.044 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068019+0000", 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3826999999999998e-05 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068064+0000", 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4999000000000001e-05 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221398 7.11 7:8907d6c7:::benchmark_data_smithi064_87013_object100441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.367034+0000", 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.61101319900001, 2024-03-24T12:00:19.045 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.70618219199999999, 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221398 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367034+0000", 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.046 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367034+0000", 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367036+0000", 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9570000000000001e-06 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.047 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367040+0000", 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3839999999999999e-06 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367052+0000", 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1479e-05 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367059+0000", 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5680000000000002e-06 2024-03-24T12:00:19.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367122+0000", 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2909000000000004e-05 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367133+0000", 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1003e-05 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066169+0000", 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.69903535100000003 2024-03-24T12:00:19.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066248+0000", 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9142999999999998e-05 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066448+0000", 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020039400000000001 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068083+0000", 2024-03-24T12:00:19.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001635135 2024-03-24T12:00:19.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.051 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.051 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068118+0000", 2024-03-24T12:00:19.051 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4981999999999998e-05 2024-03-24T12:00:19.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.051 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:19.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:18 smithi064 ceph-mon[30223]: pgmap v344: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 927 B/s rd, 61 MiB/s wr, 970 op/s; 61958/323589 objects misplaced (19.147%); 10 MiB/s, 163 objects/s recovering 2024-03-24T12:00:19.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:18 smithi064 ceph-mon[30223]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:00:19.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:18 smithi064 ceph-mon[30223]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/264903 objects degraded (0.000%), 2 pgs degraded) 2024-03-24T12:00:19.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:19.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:18 smithi064 ceph-mon[42243]: pgmap v344: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 927 B/s rd, 61 MiB/s wr, 970 op/s; 61958/323589 objects misplaced (19.147%); 10 MiB/s, 163 objects/s recovering 2024-03-24T12:00:19.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:18 smithi064 ceph-mon[42243]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:00:19.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:18 smithi064 ceph-mon[42243]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/264903 objects degraded (0.000%), 2 pgs degraded) 2024-03-24T12:00:19.053 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073116+0000", 2024-03-24T12:00:19.053 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0049980170000000004 2024-03-24T12:00:19.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073170+0000", 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3362000000000002e-05 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073216+0000", 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6508000000000003e-05 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221445 7.11 7:8ba7bf8f:::benchmark_data_smithi064_87013_object100488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.438694+0000", 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.53935297999999, 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.63464338099999995, 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221445 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.055 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438694+0000", 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438694+0000", 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438695+0000", 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.269e-06 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438698+0000", 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0680000000000001e-06 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438706+0000", 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3390000000000004e-06 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438710+0000", 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.561e-06 2024-03-24T12:00:19.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438751+0000", 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0349999999999998e-05 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438757+0000", 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1630000000000001e-06 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068152+0000", 2024-03-24T12:00:19.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.62939536500000004 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068258+0000", 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010558200000000001 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068479+0000", 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022149299999999999 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073065+0000", 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045854399999999997 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073077+0000", 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2023e-05 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073257+0000", 2024-03-24T12:00:19.060 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018018 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073302+0000", 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4758000000000001e-05 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073338+0000", 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5790000000000001e-05 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221900 7.17 7:ed89da28:::benchmark_data_smithi064_87013_object100943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.089809+0000", 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.88823786399999, 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.97642383899999996, 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221900 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.062 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089809+0000", 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089809+0000", 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089810+0000", 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.077e-06 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.063 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089815+0000", 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5290000000000002e-06 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089825+0000", 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0208e-05 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089833+0000", 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4909999999999999e-06 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.064 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089890+0000", 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7324000000000002e-05 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089898+0000", 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1049999999999995e-06 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.059780+0000", 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.969881993 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.065 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.059897+0000", 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000117451 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.060151+0000", 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025358800000000001 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.064132+0000", 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0039813499999999998 2024-03-24T12:00:19.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066047+0000", 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0019148139999999999 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066117+0000", 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9938999999999996e-05 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066189+0000", 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.148e-05 2024-03-24T12:00:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066233+0000", 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4490000000000003e-05 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221927 7.17 7:ee472041:::benchmark_data_smithi064_87013_object100970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.113707+0000", 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.86433987199999, 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95626834000000005, 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.068 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221927 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113707+0000", 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113707+0000", 2024-03-24T12:00:19.069 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113709+0000", 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1909999999999999e-06 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113716+0000", 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.37e-06 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.070 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113728+0000", 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2583e-05 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113737+0000", 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4630000000000001e-06 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113808+0000", 2024-03-24T12:00:19.071 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0976000000000006e-05 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113817+0000", 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2070000000000001e-06 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.062331+0000", 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.94851381899999998 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.072 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.062532+0000", 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000201079 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.062753+0000", 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022100900000000001 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066273+0000", 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0035203779999999998 2024-03-24T12:00:19.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066360+0000", 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6365000000000004e-05 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.069880+0000", 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0035198579999999998 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.069929+0000", 2024-03-24T12:00:19.074 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8952000000000002e-05 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.069976+0000", 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7089999999999998e-05 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221945 7.17 7:e85c526a:::benchmark_data_smithi064_87013_object100988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.127910+0000", 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.85013670399999, 2024-03-24T12:00:19.075 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.94219929999999996, 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221945 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127910+0000", 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.076 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127910+0000", 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127912+0000", 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.635e-06 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127919+0000", 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3780000000000003e-06 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.077 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127930+0000", 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0708e-05 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127946+0000", 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5951e-05 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127999+0000", 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3359000000000002e-05 2024-03-24T12:00:19.078 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.128006+0000", 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4409999999999998e-06 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.064179+0000", 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.93617329199999999 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.064312+0000", 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013303800000000001 2024-03-24T12:00:19.079 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.064556+0000", 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024340599999999999 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066308+0000", 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001752569 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.069989+0000", 2024-03-24T12:00:19.080 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036805750000000002 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070033+0000", 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3878999999999998e-05 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070074+0000", 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1576999999999999e-05 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070110+0000", 2024-03-24T12:00:19.081 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5491999999999998e-05 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221981 7.17 7:ea5818cd:::benchmark_data_smithi064_87013_object101024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.154381+0000", 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.823666334, 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92266110999999995, 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.082 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221981 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154381+0000", 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154381+0000", 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.083 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154383+0000", 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.844e-06 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154388+0000", 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0930000000000002e-06 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154397+0000", 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2580000000000001e-06 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.084 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154403+0000", 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9689999999999999e-06 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154452+0000", 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9258999999999997e-05 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154463+0000", 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1090999999999999e-05 2024-03-24T12:00:19.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066406+0000", 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91194288999999995 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066526+0000", 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011948 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066731+0000", 2024-03-24T12:00:19.086 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020494600000000001 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070123+0000", 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0033922480000000001 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073339+0000", 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032162269999999999 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.076943+0000", 2024-03-24T12:00:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036038250000000002 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.076996+0000", 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3486000000000003e-05 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077042+0000", 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5494000000000002e-05 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.088 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222006 7.17 7:e9e2a120:::benchmark_data_smithi064_87013_object101049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.173925+0000", 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.80412238100001, 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.90330706000000005, 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222006 2024-03-24T12:00:19.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173925+0000", 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173925+0000", 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.090 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173926+0000", 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4169999999999999e-06 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173931+0000", 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.826e-06 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173942+0000", 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1046999999999999e-05 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.091 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173950+0000", 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6469999999999998e-06 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173997+0000", 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7599000000000003e-05 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.174003+0000", 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.412e-06 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.092 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.068132+0000", 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.89412962500000004 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.068239+0000", 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010655299999999999 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.068446+0000", 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020731699999999999 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.094 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.094 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073356+0000", 2024-03-24T12:00:19.094 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0049102829999999997 2024-03-24T12:00:19.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.095 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.095 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.076853+0000", 2024-03-24T12:00:19.095 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034965450000000002 2024-03-24T12:00:19.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077143+0000", 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028972399999999999 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077193+0000", 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0186999999999999e-05 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077232+0000", 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8878000000000003e-05 2024-03-24T12:00:19.096 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222025 7.17 7:edc5c377:::benchmark_data_smithi064_87013_object101068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.191039+0000", 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.78700807999999, 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88634509100000003, 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222025 2024-03-24T12:00:19.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191039+0000", 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191039+0000", 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191040+0000", 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.37e-06 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191045+0000", 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3930000000000001e-06 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191054+0000", 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8089999999999999e-06 2024-03-24T12:00:19.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191061+0000", 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7659999999999998e-06 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191114+0000", 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2343999999999999e-05 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191122+0000", 2024-03-24T12:00:19.100 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6230000000000003e-06 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070149+0000", 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.879026851 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070253+0000", 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000103705 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070456+0000", 2024-03-24T12:00:19.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000203437 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073361+0000", 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0029049169999999999 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.076899+0000", 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0035376990000000001 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077293+0000", 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039371700000000003 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077343+0000", 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0634999999999999e-05 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077384+0000", 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0825000000000003e-05 2024-03-24T12:00:19.103 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222028 7.17 7:ea98fedf:::benchmark_data_smithi064_87013_object101071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.193044+0000", 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.785003582, 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88705281400000002, 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222028 2024-03-24T12:00:19.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193044+0000", 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193044+0000", 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193045+0000", 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8400000000000002e-07 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193048+0000", 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2899999999999998e-06 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193056+0000", 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3080000000000008e-06 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193061+0000", 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0629999999999996e-06 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193107+0000", 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5776999999999999e-05 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193117+0000", 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7820000000000005e-06 2024-03-24T12:00:19.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.071768+0000", 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.87865084000000004 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.071875+0000", 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107205 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.072057+0000", 2024-03-24T12:00:19.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000181946 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.075219+0000", 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003162514 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077094+0000", 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001874887 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.079954+0000", 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028593780000000001 2024-03-24T12:00:19.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080043+0000", 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9956000000000004e-05 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080096+0000", 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2883999999999997e-05 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222050 7.17 7:eff9e39d:::benchmark_data_smithi064_87013_object101093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.214310+0000", 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.76373705500001, 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.86595284100000003, 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222050 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214310+0000", 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214310+0000", 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214312+0000", 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5e-06 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214316+0000", 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1139999999999999e-06 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214327+0000", 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1253999999999999e-05 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214335+0000", 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8520000000000004e-06 2024-03-24T12:00:19.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214426+0000", 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1668000000000004e-05 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214433+0000", 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9770000000000003e-06 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073387+0000", 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85895402700000001 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073495+0000", 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107889 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073680+0000", 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000184728 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077109+0000", 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034289519999999999 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.078140+0000", 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0010306200000000001 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080165+0000", 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020256010000000001 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080222+0000", 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7197000000000001e-05 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080263+0000", 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0462e-05 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.117 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222079 7.17 7:eeb9bced:::benchmark_data_smithi064_87013_object101122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.247113+0000", 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 145.73093378300001, 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.83330855000000004, 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222079 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:00:19.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247113+0000", 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247113+0000", 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247115+0000", 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.559e-06 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247119+0000", 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8260000000000003e-06 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247129+0000", 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8530000000000004e-06 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247134+0000", 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8019999999999997e-06 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247178+0000", 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3229000000000002e-05 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247185+0000", 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9489999999999999e-06 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.075262+0000", 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.828077536 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.075380+0000", 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000118319 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.075570+0000", 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018975400000000001 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077240+0000", 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016701400000000001 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.078175+0000", 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00093505400000000005 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080329+0000", 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021540510000000001 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080380+0000", 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0824e-05 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:00:19.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080422+0000", 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1653999999999998e-05 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:00:19.124 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:00:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:19.125 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:19.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:19.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:19.265 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:19.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:19.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:19.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:19.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:19.539 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:19.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:19.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:19.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:19.683 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:19.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:19.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:19.831 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:19.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:19.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:19.833 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 79 16 111317 111301 88.0361 78.4375 0.00562106 0.0113542 2024-03-24T12:00:19.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:00:19.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:20.014 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:20.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:20.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:20.190 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:20.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:20.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:20.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:20.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:20.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:20.454 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:20.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:20.555 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:20.589 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:20.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:20.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:20.735 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:20.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:20.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:00:20.834133+0000 min lat: 0.00128588 max lat: 2.09702 avg lat: 0.0113772 2024-03-24T12:00:20.833 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:00:20.833 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 80 16 112499 112483 87.8591 73.875 0.0142917 0.0113772 2024-03-24T12:00:20.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:20.878 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:20.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:00:20.964 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T12:00:20.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-24T12:00:20.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:21.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:20 smithi064 ceph-mon[30223]: pgmap v345: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 785 B/s rd, 187 MiB/s wr, 3k op/s; 61958/323589 objects misplaced (19.147%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T12:00:21.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:20 smithi064 ceph-mon[42243]: pgmap v345: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 785 B/s rd, 187 MiB/s wr, 3k op/s; 61958/323589 objects misplaced (19.147%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T12:00:21.057 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:21.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:21.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:20 smithi193 ceph-mon[33593]: pgmap v345: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 785 B/s rd, 187 MiB/s wr, 3k op/s; 61958/323589 objects misplaced (19.147%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T12:00:21.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:00:21.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:21.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:21.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:21.231 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:21.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:21.344 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:21.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:21.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:21.505 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:21.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:21.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:21.657 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:21.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:21.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:21.769 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:00:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:21.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:21.827 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:00:21.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 81 16 113448 113432 87.5066 59.3125 0.00356296 0.0114224 2024-03-24T12:00:21.880 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:00:21.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:21.922 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:00:21.968 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:00:21.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:21.971 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:21.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:22.015 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:00:22.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1732966956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:22.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:21 smithi064 ceph-mon[42243]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T12:00:22.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1732966956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:22.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:21 smithi064 ceph-mon[30223]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T12:00:22.060 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:00:22.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:22.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1732966956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:22.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:21 smithi193 ceph-mon[33593]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T12:00:22.105 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:00:22.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:00:22.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:00:22.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:00:22.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:00:22.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:00:22.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:00:22.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:00:22.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:00:22.117 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:22.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:22.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:22.259 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:22.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:22.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:22.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:22.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:22.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:22.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:22.546 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:22.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:22.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:22.686 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:22.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:22.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:22.823 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:22.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:22.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 82 16 114637 114621 87.3456 74.3125 0.0145957 0.0114444 2024-03-24T12:00:22.924 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:22.954 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:22.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:23.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:22 smithi064 ceph-mon[30223]: pgmap v346: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 160 MiB/s wr, 2.56k op/s; 62301/326280 objects misplaced (19.094%); 7.0 MiB/s, 112 objects/s recovering 2024-03-24T12:00:23.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:22 smithi064 ceph-mon[42243]: pgmap v346: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 160 MiB/s wr, 2.56k op/s; 62301/326280 objects misplaced (19.094%); 7.0 MiB/s, 112 objects/s recovering 2024-03-24T12:00:23.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:23.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:22 smithi193 ceph-mon[33593]: pgmap v346: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 160 MiB/s wr, 2.56k op/s; 62301/326280 objects misplaced (19.094%); 7.0 MiB/s, 112 objects/s recovering 2024-03-24T12:00:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:23.096 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:23.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:23.197 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:23.228 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:23.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:23.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:23.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:23.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:23.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:23.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:23.788 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:23.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:23.832 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 83 16 115678 115662 87.0771 65.0625 0.00593411 0.0114731 2024-03-24T12:00:23.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:23.928 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:23.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:24.030 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:24.062 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:24.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:24.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:24.188 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:24.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:24.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:24.323 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:24.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:24.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:24.465 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:24.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:24.566 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:24.599 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:24.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:24.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:24.740 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:24.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:24.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:24.833 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 84 16 116627 116611 86.7466 59.3125 0.00575192 0.0115164 2024-03-24T12:00:24.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:24.866 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:24.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:24.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:24.967 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:25.002 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:25.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:25.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:24 smithi064 ceph-mon[30223]: pgmap v348: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 197 MiB/s wr, 3.15k op/s; 62118/340392 objects misplaced (18.249%); 18 MiB/s, 295 objects/s recovering 2024-03-24T12:00:25.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:24 smithi064 ceph-mon[42243]: pgmap v348: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 197 MiB/s wr, 3.15k op/s; 62118/340392 objects misplaced (18.249%); 18 MiB/s, 295 objects/s recovering 2024-03-24T12:00:25.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:24 smithi193 ceph-mon[33593]: pgmap v348: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 197 MiB/s wr, 3.15k op/s; 62118/340392 objects misplaced (18.249%); 18 MiB/s, 295 objects/s recovering 2024-03-24T12:00:25.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:25.140 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:25.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:25.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:25.284 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:25.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:25.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:25.416 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:25.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:25.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:25.563 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:25.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:25.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:25.698 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:25.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:25.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:25.833 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 85 16 117600 117584 86.4411 60.8125 0.0146439 0.0115632 2024-03-24T12:00:25.865 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:25.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:25.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:25.990 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:25.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:26.092 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:26.127 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:26.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:26.229 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:26.264 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:26.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:26.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:26.390 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:26.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:26.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:26.525 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:26.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:26.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:26.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:26.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:26.783 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:26.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:26.833 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 86 16 118118 118102 85.8125 32.375 0.00681521 0.0116488 2024-03-24T12:00:26.886 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:26.920 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:26.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:27.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:27.067 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:27.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:27.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:26 smithi193 ceph-mon[33593]: pgmap v349: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 602 B/s rd, 101 MiB/s wr, 1.62k op/s; 62118/340392 objects misplaced (18.249%); 17 MiB/s, 277 objects/s recovering 2024-03-24T12:00:27.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:27.168 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:27.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:27.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:26 smithi064 ceph-mon[30223]: pgmap v349: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 602 B/s rd, 101 MiB/s wr, 1.62k op/s; 62118/340392 objects misplaced (18.249%); 17 MiB/s, 277 objects/s recovering 2024-03-24T12:00:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:26 smithi064 ceph-mon[42243]: pgmap v349: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 602 B/s rd, 101 MiB/s wr, 1.62k op/s; 62118/340392 objects misplaced (18.249%); 17 MiB/s, 277 objects/s recovering 2024-03-24T12:00:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:27.328 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:27.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:27.430 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:27.453 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:27.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:27.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:27.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:00:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:00:27.595 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:27.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:27.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:27.737 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:27.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:27.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:00:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:00:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:27.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:00:27.833 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 87 16 119079 119063 85.5165 60.0625 0.00318127 0.0116802 2024-03-24T12:00:27.839 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:27.864 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:27.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:27.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:28.010 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:28.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:28.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:28.145 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:28.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:28.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:28.271 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:28.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:28.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:28.373 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:28.409 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:28.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:28.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:28.552 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:28.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:28.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:28.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:28.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:28.833 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:28.833 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 88 16 120006 119990 85.2031 57.9375 0.0189227 0.0117318 2024-03-24T12:00:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:28.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:28.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:28.973 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:28.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:29.074 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:28 smithi193 ceph-mon[33593]: pgmap v350: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s; 61868/354840 objects misplaced (17.435%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:29.108 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:29.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:29.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:29.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:29.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:29.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:29.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:28 smithi064 ceph-mon[30223]: pgmap v350: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s; 61868/354840 objects misplaced (17.435%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:29.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:29.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:28 smithi064 ceph-mon[42243]: pgmap v350: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s; 61868/354840 objects misplaced (17.435%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:29.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:29.369 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:29.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:29.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:29.501 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:29.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:29.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:29.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:29.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:29.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:29.781 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:29.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:29.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 89 16 121041 121025 84.9725 64.6875 0.00417406 0.0117635 2024-03-24T12:00:29.883 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:29.921 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:29.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:30.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:30.063 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:30.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:30.165 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:30.196 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:30.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:30.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:30.321 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:30.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:30.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:30.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:30.466 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:30.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:30.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:30.608 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:30.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:30.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:30.742 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:30.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total time run: 90.0075 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total writes made: 122023 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Write size: 65536 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object size: 65536 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Bandwidth (MB/sec): 84.7312 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Bandwidth: 37.4877 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max bandwidth (MB/sec): 221.625 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min bandwidth (MB/sec): 0 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average IOPS: 1355 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev IOPS: 599.816 2024-03-24T12:00:30.834 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max IOPS: 3546 2024-03-24T12:00:30.835 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min IOPS: 0 2024-03-24T12:00:30.835 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average Latency(s): 0.0117978 2024-03-24T12:00:30.835 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Latency(s): 0.0308745 2024-03-24T12:00:30.835 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max latency(s): 2.09702 2024-03-24T12:00:30.835 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min latency(s): 0.00128588 2024-03-24T12:00:30.837 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:00:30.843 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:30.874 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:30.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:30.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:31.014 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:31.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:31.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:30 smithi193 ceph-mon[33593]: pgmap v351: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s; 61868/354840 objects misplaced (17.435%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:31.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:31.154 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:31.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:31.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:31.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:31.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:30 smithi064 ceph-mon[30223]: pgmap v351: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s; 61868/354840 objects misplaced (17.435%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:31.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:30 smithi064 ceph-mon[42243]: pgmap v351: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s; 61868/354840 objects misplaced (17.435%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:31.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:31.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:31.442 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:31.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:31.544 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:31.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:31.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:31.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:31.721 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:31.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:31.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:31.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:31.966 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:31.996 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:31.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:32.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:32.119 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:32.138 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:32.169 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:32.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:32.271 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:32.304 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:32.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:32.350 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:32.405 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:32.436 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:32.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:32.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:32.594 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:32.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:32.695 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:32.728 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:32.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:32.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:32.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:32.975 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:32 smithi064 ceph-mon[30223]: pgmap v352: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 62113/356595 objects misplaced (17.418%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:32.975 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:32 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:00:32.975 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:32 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:00:32.975 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:32 smithi064 ceph-mon[42243]: pgmap v352: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 62113/356595 objects misplaced (17.418%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:32.975 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:32 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:00:32.975 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:32 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:00:32.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:33.030 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:33.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:33.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:32 smithi193 ceph-mon[33593]: pgmap v352: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 62113/356595 objects misplaced (17.418%); 18 MiB/s, 284 objects/s recovering 2024-03-24T12:00:33.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:32 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:00:33.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:32 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:00:33.130 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:33.164 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:33.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:33.265 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:33.296 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:33.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:33.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:33.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:33.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:33.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:33.574 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:33.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:33.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:33.718 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:33.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:33.820 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:33.852 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:33.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:33.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:33.987 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:33.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:34.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:33 smithi064 ceph-mon[30223]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T12:00:34.043 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:33 smithi064 ceph-mon[42243]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T12:00:34.089 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:34.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:34.136 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:34.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:34.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:34.277 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:34.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:34.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:33 smithi193 ceph-mon[33593]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T12:00:34.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:34.416 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:34.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:34.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:34.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:34.567 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:34.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:34.668 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:34.704 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:34.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:34.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:34.839 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:34.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:34.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:34.982 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:34.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:35.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:35.127 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:35.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:35.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:35.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:35.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:35.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:35.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:34 smithi064 ceph-mon[30223]: pgmap v354: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 972 op/s; 60607/363783 objects misplaced (16.660%); 18 MiB/s, 283 objects/s recovering 2024-03-24T12:00:35.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:34 smithi064 ceph-mon[42243]: pgmap v354: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 972 op/s; 60607/363783 objects misplaced (16.660%); 18 MiB/s, 283 objects/s recovering 2024-03-24T12:00:35.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:34 smithi193 ceph-mon[33593]: pgmap v354: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 972 op/s; 60607/363783 objects misplaced (16.660%); 18 MiB/s, 283 objects/s recovering 2024-03-24T12:00:35.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:35.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:35.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:35.509 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:35.546 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:35.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:35.648 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:35.685 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:35.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:35.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:35.830 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:35.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:35.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:35.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:35.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:35.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:36.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:36.100 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:36.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:36.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:36.235 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:36.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:36.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:36.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:36.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:36.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:36.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:36.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:36.622 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:36.654 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:36.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:36.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:36.793 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:00:36.793 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:00:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:36.813 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:36.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:36.914 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:36.951 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:36.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:37.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:36 smithi064 ceph-mon[30223]: pgmap v355: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 972 op/s; 60607/363783 objects misplaced (16.660%); 18 MiB/s, 283 objects/s recovering 2024-03-24T12:00:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:36 smithi064 ceph-mon[30223]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T12:00:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:36 smithi064 ceph-mon[42243]: pgmap v355: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 972 op/s; 60607/363783 objects misplaced (16.660%); 18 MiB/s, 283 objects/s recovering 2024-03-24T12:00:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:36 smithi064 ceph-mon[42243]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T12:00:37.053 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:37.086 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:37.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:37.177 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:36 smithi193 ceph-mon[33593]: pgmap v355: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 972 op/s; 60607/363783 objects misplaced (16.660%); 18 MiB/s, 283 objects/s recovering 2024-03-24T12:00:37.177 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:36 smithi193 ceph-mon[33593]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T12:00:37.188 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:37.221 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:37.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:37.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:37.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:37.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:37.464 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:37.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:37.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:37.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:00:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:00:37.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:37.648 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:37.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:37.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:37.774 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:37.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:37.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:00:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:00:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:37.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:37.908 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:37.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:38.010 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:38.040 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:38.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:38.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:38.185 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:38.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:38.287 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:38.320 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:38.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:38.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:38.451 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:38.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:38.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:38.591 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:38.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:38.693 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:38.722 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:38.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:38.824 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:38.855 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:38.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:38.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:38.990 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:38.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[30223]: pgmap v357: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.37k op/s; 54417/345708 objects misplaced (15.741%); 23 MiB/s, 370 objects/s recovering 2024-03-24T12:00:39.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:00:39.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:00:39.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[30223]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T12:00:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[42243]: pgmap v357: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.37k op/s; 54417/345708 objects misplaced (15.741%); 23 MiB/s, 370 objects/s recovering 2024-03-24T12:00:39.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:00:39.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:00:39.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:38 smithi064 ceph-mon[42243]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T12:00:39.092 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:39.127 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:39.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:39.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:39.273 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:39.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:38 smithi193 ceph-mon[33593]: pgmap v357: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.37k op/s; 54417/345708 objects misplaced (15.741%); 23 MiB/s, 370 objects/s recovering 2024-03-24T12:00:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:38 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:00:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:38 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:00:39.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:38 smithi193 ceph-mon[33593]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T12:00:39.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:39.427 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:39.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:39.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:39.573 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:39.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:39.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:39.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:39.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:39.822 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:39.857 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:39.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:39.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:39.982 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:39.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:40.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:40.124 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:40.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:40.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:40.259 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:40.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:40.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:40.397 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:40.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:40.498 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:40.525 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:40.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:40.626 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:40.652 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:40.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:40.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:40.785 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:40.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T12:00:40.817 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T12:00:40.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:40.924 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:40.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:41.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:41.059 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:41.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:41.094 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:00:41.134 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:00:41.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:40 smithi064 ceph-mon[30223]: pgmap v359: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 795 B/s rd, 0 B/s wr, 937 op/s; 54417/345708 objects misplaced (15.741%); 15 MiB/s, 234 objects/s recovering 2024-03-24T12:00:41.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:40 smithi064 ceph-mon[30223]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T12:00:41.135 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:40 smithi064 ceph-mon[42243]: pgmap v359: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 795 B/s rd, 0 B/s wr, 937 op/s; 54417/345708 objects misplaced (15.741%); 15 MiB/s, 234 objects/s recovering 2024-03-24T12:00:41.135 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:40 smithi064 ceph-mon[42243]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T12:00:41.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:41.176 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:00:41.189 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:41.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:41.217 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:00:41.259 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:00:41.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:41.302 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:00:41.320 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:41.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:41.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:40 smithi193 ceph-mon[33593]: pgmap v359: 168 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 795 B/s rd, 0 B/s wr, 937 op/s; 54417/345708 objects misplaced (15.741%); 15 MiB/s, 234 objects/s recovering 2024-03-24T12:00:41.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:40 smithi193 ceph-mon[33593]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T12:00:41.342 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:00:41.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:00:41.389 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:00:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:00:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:00:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:00:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:00:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:00:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:00:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:00:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:00:41.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:41.456 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:41.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:41.558 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:41.594 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:41.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:41.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:41.743 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:41.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:41.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:41.890 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:41.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:00:41.936 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T12:00:41.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-24T12:00:41.992 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:42.032 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:42.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:42.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:42.199 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:42.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:42.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:00:42.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:00:42.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[30223]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4029264431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[30223]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[42243]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4029264431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:42.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:41 smithi064 ceph-mon[42243]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T12:00:42.301 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:42.334 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:42.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:41 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:00:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:41 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:00:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:41 smithi193 ceph-mon[33593]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T12:00:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:41 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4029264431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:00:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:41 smithi193 ceph-mon[33593]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T12:00:42.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:42.469 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:42.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:42.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:42.572 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:42.605 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:42.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:42.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:42.741 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:42.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:42.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:42.881 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:42.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:42.983 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:43.013 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:43.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:43.115 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:43.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:43.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:43.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:42 smithi064 ceph-mon[30223]: pgmap v362: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 942 op/s; 53693/342921 objects misplaced (15.658%); 26 MiB/s, 418 objects/s recovering 2024-03-24T12:00:43.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:43.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:43.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:43.243 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:42 smithi064 ceph-mon[42243]: pgmap v362: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 942 op/s; 53693/342921 objects misplaced (15.658%); 26 MiB/s, 418 objects/s recovering 2024-03-24T12:00:43.243 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:43.243 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:43.243 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:43.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:43.280 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:43.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:42 smithi193 ceph-mon[33593]: pgmap v362: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 942 op/s; 53693/342921 objects misplaced (15.658%); 26 MiB/s, 418 objects/s recovering 2024-03-24T12:00:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:00:43.383 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:43.418 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:43.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:43.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:43.564 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:43.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:43.618 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:43.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:43.695 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:43.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:43.796 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:43.834 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:43.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:43.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:44.007 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:44.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:44.108 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:44.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:44.148 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:44.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:44.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:44.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:44.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:44.431 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:44.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:44.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:44.576 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:44.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:44.677 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:44.713 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:44.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:44.815 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:44.841 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:44.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:44.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:44.944 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:00:44.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:44.975 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:44.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:45.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:45.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:45.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:45.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:45.227 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:45.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:45.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[30223]: pgmap v364: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 933 op/s; 48303/330600 objects misplaced (14.611%); 17 MiB/s, 274 objects/s recovering 2024-03-24T12:00:45.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:00:45.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:00:45.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[30223]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T12:00:45.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/928991719' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:00:45.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[42243]: pgmap v364: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 933 op/s; 48303/330600 objects misplaced (14.611%); 17 MiB/s, 274 objects/s recovering 2024-03-24T12:00:45.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:00:45.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:00:45.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[42243]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T12:00:45.262 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/928991719' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:00:45.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:00:45.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:44 smithi193 ceph-mon[33593]: pgmap v364: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 933 op/s; 48303/330600 objects misplaced (14.611%); 17 MiB/s, 274 objects/s recovering 2024-03-24T12:00:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:44 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:00:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:44 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:00:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:44 smithi193 ceph-mon[33593]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T12:00:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/928991719' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:00:45.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:45.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:45.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:45.511 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:45.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:45.613 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:45.643 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:45.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:45.745 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:45.783 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:45.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:45.884 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:45.917 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:45.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:46.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:46.056 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:46.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:46.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:46.201 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:46.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:45 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/928991719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:00:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:45 smithi064 ceph-mon[30223]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T12:00:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:45 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:45 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/928991719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:00:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:45 smithi064 ceph-mon[42243]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T12:00:46.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:45 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:00:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:45 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/928991719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:00:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:45 smithi193 ceph-mon[33593]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T12:00:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:46.339 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:46.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:46.440 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:46.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:46.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:46.568 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:46.604 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:46.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:46.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:46.728 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:46.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:46.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:46.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:46.974 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:47.009 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:47.115 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.154 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:47.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:47.254 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:47.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:46 smithi064 ceph-mon[30223]: pgmap v367: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s; 48303/330600 objects misplaced (14.611%); 21 MiB/s, 337 objects/s recovering 2024-03-24T12:00:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:46 smithi064 ceph-mon[42243]: pgmap v367: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s; 48303/330600 objects misplaced (14.611%); 21 MiB/s, 337 objects/s recovering 2024-03-24T12:00:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:46 smithi193 ceph-mon[33593]: pgmap v367: 168 pgs: 7 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 159 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s; 48303/330600 objects misplaced (14.611%); 21 MiB/s, 337 objects/s recovering 2024-03-24T12:00:47.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:00:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:00:47.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:47.430 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:47.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:47.579 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:47.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:47.705 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:00:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:00:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:47.715 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:47.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:47.858 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:47.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:48.003 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:48.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:48.025 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:48.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:48.166 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:48.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:48.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:48 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:00:48.262 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:48 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:00:48.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:48.301 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:48.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:48.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:48 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:00:48.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:48.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:48.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:48.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:48.566 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:48.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:48.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:48.702 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:48.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:48.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:48.836 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:48.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:48.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:48.975 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:48.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:49.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:49.114 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:49.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:49.215 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:49.262 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:49.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:49.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:49 smithi064 ceph-mon[30223]: pgmap v368: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 41456/315945 objects misplaced (13.121%); 38 MiB/s, 606 objects/s recovering 2024-03-24T12:00:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:49 smithi064 ceph-mon[42243]: pgmap v368: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 41456/315945 objects misplaced (13.121%); 38 MiB/s, 606 objects/s recovering 2024-03-24T12:00:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:49 smithi193 ceph-mon[33593]: pgmap v368: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 41456/315945 objects misplaced (13.121%); 38 MiB/s, 606 objects/s recovering 2024-03-24T12:00:49.363 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:49.396 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:49.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:49.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:49.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:49.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:49.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:00:49.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:49.683 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:49.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:49.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:49.866 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:49.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:49.969 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:50.015 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:50.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:50.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:50.158 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:50.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:50.260 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:50.295 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:50.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:50.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:50.473 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:50.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:50.530 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:00:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-03-24T12:00:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-03-24T12:00:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-03-24T12:00:50.531 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:00:50.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:50.608 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:50.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:50.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:50.754 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:50.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:50.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:50.891 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:50.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:50.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:51.029 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:51.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:51.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:51.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:51.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:51.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:51.297 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:51.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:51 smithi064 ceph-mon[30223]: pgmap v369: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 41456/315945 objects misplaced (13.121%); 30 MiB/s, 484 objects/s recovering 2024-03-24T12:00:51.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:51 smithi064 ceph-mon[42243]: pgmap v369: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 41456/315945 objects misplaced (13.121%); 30 MiB/s, 484 objects/s recovering 2024-03-24T12:00:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:51 smithi193 ceph-mon[33593]: pgmap v369: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 41456/315945 objects misplaced (13.121%); 30 MiB/s, 484 objects/s recovering 2024-03-24T12:00:51.399 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:51.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:51.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:51.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:51.577 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:51.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:51.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:51.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:51.710 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:51.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:51.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:51.848 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:51.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:51.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:51.979 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:51.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:52.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:52.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:52.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:52.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:52.247 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:52.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:52.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:52 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:00:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:52 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:00:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:52 smithi193 ceph-mon[33593]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T12:00:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:52 smithi193 ceph-mon[33593]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T12:00:52.347 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:52.381 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:52.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:52.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:52.523 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:52.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:52 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:00:52.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:52 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:00:52.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:52 smithi064 ceph-mon[30223]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T12:00:52.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:52 smithi064 ceph-mon[30223]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T12:00:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:52 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:00:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:52 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:00:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:52 smithi064 ceph-mon[42243]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T12:00:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:52 smithi064 ceph-mon[42243]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T12:00:52.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:52.664 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:52.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:52.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:52.793 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:52.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:52.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:52.941 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:52.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:53.042 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:53.068 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:53.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:53.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:53.215 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:53.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:53.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:53.317 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:53 smithi193 ceph-mon[33593]: pgmap v371: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 697 B/s rd, 0 B/s wr, 786 op/s; 39956/313284 objects misplaced (12.754%); 32 MiB/s, 519 objects/s recovering 2024-03-24T12:00:53.350 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:53.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:53.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:53.485 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:53.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:53.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:53 smithi064 ceph-mon[30223]: pgmap v371: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 697 B/s rd, 0 B/s wr, 786 op/s; 39956/313284 objects misplaced (12.754%); 32 MiB/s, 519 objects/s recovering 2024-03-24T12:00:53.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:53 smithi064 ceph-mon[42243]: pgmap v371: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 697 B/s rd, 0 B/s wr, 786 op/s; 39956/313284 objects misplaced (12.754%); 32 MiB/s, 519 objects/s recovering 2024-03-24T12:00:53.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:53.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:53.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:53.756 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:53.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:53.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:53.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:53.893 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:53.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:53.994 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:54.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:54.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:54.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:54.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:54.156 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:54.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:54.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:54.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:54.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:54 smithi193 ceph-mon[33593]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T12:00:54.389 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:54.419 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:54.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:54.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:54.546 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:54.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:54.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:54 smithi064 ceph-mon[30223]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T12:00:54.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:54 smithi064 ceph-mon[42243]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T12:00:54.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:54.685 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:54.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:54.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:54.810 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:54.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:54.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:54.952 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:54.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:55.053 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:55.085 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:55.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:55.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:55.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:55.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:55 smithi193 ceph-mon[33593]: pgmap v374: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 769 op/s; 32653/302100 objects misplaced (10.809%); 47 MiB/s, 757 objects/s recovering 2024-03-24T12:00:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:55 smithi193 ceph-mon[33593]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T12:00:55.365 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:55.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:55.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:55.490 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:55.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:55.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:55 smithi064 ceph-mon[30223]: pgmap v374: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 769 op/s; 32653/302100 objects misplaced (10.809%); 47 MiB/s, 757 objects/s recovering 2024-03-24T12:00:55.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:55 smithi064 ceph-mon[30223]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T12:00:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:55 smithi064 ceph-mon[42243]: pgmap v374: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 769 op/s; 32653/302100 objects misplaced (10.809%); 47 MiB/s, 757 objects/s recovering 2024-03-24T12:00:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:55 smithi064 ceph-mon[42243]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T12:00:55.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:55.618 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:55.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:55.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:55.772 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:55.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:55.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:00:55.874 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:55.904 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:55.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:56.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:56.066 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:56.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:56.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:56.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:56.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:56.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:56.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:56.364 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:56.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:56.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:56 smithi064 ceph-mon[30223]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T12:00:56.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:56 smithi064 ceph-mon[42243]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T12:00:56.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:56.499 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:56.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:56 smithi193 ceph-mon[33593]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T12:00:56.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:56.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:56.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:56.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:56.784 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:56.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:56.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:56.906 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:00:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:56.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T12:00:56.984 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T12:00:56.985 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T12:00:57.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:57.079 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:57.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:57.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:57.217 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:57.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:57.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:57 smithi064 ceph-mon[30223]: pgmap v377: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 32653/302100 objects misplaced (10.809%); 51 MiB/s, 817 objects/s recovering 2024-03-24T12:00:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:57 smithi064 ceph-mon[30223]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T12:00:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:57 smithi064 ceph-mon[42243]: pgmap v377: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 32653/302100 objects misplaced (10.809%); 51 MiB/s, 817 objects/s recovering 2024-03-24T12:00:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:57 smithi064 ceph-mon[42243]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T12:00:57.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:57.349 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:57.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:57.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:57.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:57.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:57.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:57.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:00:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:00:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:57 smithi193 ceph-mon[33593]: pgmap v377: 168 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 159 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 32653/302100 objects misplaced (10.809%); 51 MiB/s, 817 objects/s recovering 2024-03-24T12:00:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:57 smithi193 ceph-mon[33593]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T12:00:57.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:57.624 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:57.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:57.725 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:00:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:57.758 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:57.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:00:57.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:00:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:00:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:57.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:57.891 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:57.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:00:57.937 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T12:00:57.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-24T12:00:57.992 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:58.023 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:58.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:58.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:58.174 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:58.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:58.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:00:58.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:58 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:58.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:00:58 smithi064 ceph-mon[30223]: pgmap v379: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 894 op/s; 24664/290781 objects misplaced (8.482%); 60 MiB/s, 967 objects/s recovering 2024-03-24T12:00:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:58 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:00:58 smithi064 ceph-mon[42243]: pgmap v379: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 894 op/s; 24664/290781 objects misplaced (8.482%); 60 MiB/s, 967 objects/s recovering 2024-03-24T12:00:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:58.331 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:58.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:00:58.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:58.465 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:58.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:58.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:58.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:58 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:58.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:00:58 smithi193 ceph-mon[33593]: pgmap v379: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 894 op/s; 24664/290781 objects misplaced (8.482%); 60 MiB/s, 967 objects/s recovering 2024-03-24T12:00:58.589 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:58.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:58.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:58.733 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:58.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:58.835 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:58.868 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:58.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:58.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:00:59.008 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:59.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:00:59.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:59.151 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:59.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:59.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:59.286 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:59.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:00:59.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:00:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:59.413 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:59.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:00:59.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:00:59.548 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:59.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:00:59.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:00:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:59.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:00:59.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:00:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:59.825 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:00:59.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:00:59.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:00:59.963 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:00:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:59.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:00.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:00.110 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:00.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:00.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:00.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:00.265 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:00.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:00.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:00.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:00.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:00.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:00 smithi064 ceph-mon[30223]: pgmap v380: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 629 op/s; 24664/290781 objects misplaced (8.482%); 43 MiB/s, 680 objects/s recovering 2024-03-24T12:01:00.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:00 smithi064 ceph-mon[42243]: pgmap v380: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 629 op/s; 24664/290781 objects misplaced (8.482%); 43 MiB/s, 680 objects/s recovering 2024-03-24T12:01:00.573 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:00.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:00.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:00.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:00.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:00.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:00.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:00.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:00 smithi193 ceph-mon[33593]: pgmap v380: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 629 op/s; 24664/290781 objects misplaced (8.482%); 43 MiB/s, 680 objects/s recovering 2024-03-24T12:01:00.842 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:00.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:00.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:00.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:01.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:01.131 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:01.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:01.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:01.268 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:01.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:01.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:01.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:01.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:01.547 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:01.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:01.649 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:01.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:01.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:01.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:01.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:01.912 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:01:01.942 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:01.958 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:01:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:01.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:01.998 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:01:02.038 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:01:02.083 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:01:02.096 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:02.126 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:02.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:02.131 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:01:02.175 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:01:02.217 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:01:02.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:01:02.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:01:02.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:01:02.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:01:02.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:01:02.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:01:02.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:01:02.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:01:02.229 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:02.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:02.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:02.365 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:02.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:02.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:02.504 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:02.536 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:02.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:02 smithi193 ceph-mon[33593]: pgmap v381: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 663 op/s; 23251/287757 objects misplaced (8.080%); 47 MiB/s, 744 objects/s recovering 2024-03-24T12:01:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:02 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1685798978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:02 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:01:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:02 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:01:02.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:02.679 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:02.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:02.732 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:02 smithi064 ceph-mon[30223]: pgmap v381: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 663 op/s; 23251/287757 objects misplaced (8.080%); 47 MiB/s, 744 objects/s recovering 2024-03-24T12:01:02.732 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1685798978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:02.732 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:02 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:01:02.732 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:02 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:01:02.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:02 smithi064 ceph-mon[42243]: pgmap v381: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 663 op/s; 23251/287757 objects misplaced (8.080%); 47 MiB/s, 744 objects/s recovering 2024-03-24T12:01:02.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:02 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1685798978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:02.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:02 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:01:02.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:02 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:01:02.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:02.815 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:02.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:02.917 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:02.951 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:02.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:03.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:03.101 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:03.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:03.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:03.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:03.349 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:03.379 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:03.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:03.481 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:03.511 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:03.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:03.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:03 smithi064 ceph-mon[30223]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T12:01:03.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:03 smithi064 ceph-mon[42243]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T12:01:03.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:03.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:03.751 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:03.780 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:03.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:03.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:03 smithi193 ceph-mon[33593]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T12:01:03.882 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:03.884 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:03.909 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:03.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:04.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:04.056 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:04.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:04.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:04.197 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:04.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:04.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:04.338 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:04.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:04.391 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:01:04.440 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:04.472 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:04.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:04.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:04.605 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:04.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:04.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:04.745 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:04.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:04.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:04 smithi064 ceph-mon[30223]: pgmap v383: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 18800/275451 objects misplaced (6.825%); 53 MiB/s, 842 objects/s recovering 2024-03-24T12:01:04.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4244358138' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:04.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:04 smithi064 ceph-mon[42243]: pgmap v383: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 18800/275451 objects misplaced (6.825%); 53 MiB/s, 842 objects/s recovering 2024-03-24T12:01:04.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:04 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4244358138' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:04.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:04 smithi193 ceph-mon[33593]: pgmap v383: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 18800/275451 objects misplaced (6.825%); 53 MiB/s, 842 objects/s recovering 2024-03-24T12:01:04.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:04 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4244358138' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:04.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:04.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:04.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:04.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:05.007 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:05.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:05.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:05.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:05.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:05.279 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:05.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:05.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:05.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:05.474 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:05 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:01:05.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:05 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:01:05.509 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:05.547 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:05.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:05.649 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:05.672 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:05.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:05.773 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:05.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:05 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4244358138' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:01:05.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:05 smithi064 ceph-mon[30223]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T12:01:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:05 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4244358138' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:01:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:05 smithi064 ceph-mon[42243]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T12:01:05.805 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:05.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:05.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:05 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:01:05.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:05 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4244358138' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:01:05.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:05 smithi193 ceph-mon[33593]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T12:01:05.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:05.929 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:01:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:05.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:06.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:06.084 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:06.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:06.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:06.221 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:06.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:06.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:06.346 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:06.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:06.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:06.486 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:06.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:06.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:06.612 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:06.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:06.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:06 smithi064 ceph-mon[30223]: pgmap v385: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 639 op/s; 18800/275451 objects misplaced (6.825%); 21 MiB/s, 331 objects/s recovering 2024-03-24T12:01:06.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:06 smithi064 ceph-mon[30223]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T12:01:06.688 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:06 smithi064 ceph-mon[42243]: pgmap v385: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 639 op/s; 18800/275451 objects misplaced (6.825%); 21 MiB/s, 331 objects/s recovering 2024-03-24T12:01:06.688 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:06 smithi064 ceph-mon[42243]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T12:01:06.714 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:06.749 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:06.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:06.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:06 smithi193 ceph-mon[33593]: pgmap v385: 168 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 161 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 639 op/s; 18800/275451 objects misplaced (6.825%); 21 MiB/s, 331 objects/s recovering 2024-03-24T12:01:06.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:06 smithi193 ceph-mon[33593]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T12:01:06.850 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:06.877 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:06.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:06.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:07.024 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:07.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:07.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:07.153 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:07.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:07.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:07.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:07.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:07.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:07.440 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:07.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:07.542 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:07.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:07.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:07.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:01:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:01:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:07 smithi193 ceph-mon[33593]: 8.4 scrub starts 2024-03-24T12:01:07.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:07 smithi193 ceph-mon[33593]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T12:01:07.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:07.719 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:07.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:07.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:07 smithi064 ceph-mon[30223]: 8.4 scrub starts 2024-03-24T12:01:07.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:07 smithi064 ceph-mon[30223]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T12:01:07.757 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:01:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:01:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:07.757 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:07 smithi064 ceph-mon[42243]: 8.4 scrub starts 2024-03-24T12:01:07.757 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:07 smithi064 ceph-mon[42243]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T12:01:07.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:07.862 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:07.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:07.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:08.006 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:08.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:08.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:08.144 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:08.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:08.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:08.296 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:08.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:08.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:08.436 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:08.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:08.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:08.573 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:08.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:08.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:08.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:08.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:08.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:08.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:08.872 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:08.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:08.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:09.011 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:09.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:09.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:08 smithi064 ceph-mon[30223]: 8.4 scrub ok 2024-03-24T12:01:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:08 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:08 smithi064 ceph-mon[30223]: 8.3 scrub starts 2024-03-24T12:01:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:08 smithi064 ceph-mon[30223]: pgmap v388: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 12548/263400 objects misplaced (4.764%); 37 MiB/s, 593 objects/s recovering 2024-03-24T12:01:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:08 smithi064 ceph-mon[42243]: 8.4 scrub ok 2024-03-24T12:01:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:08 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:08 smithi064 ceph-mon[42243]: 8.3 scrub starts 2024-03-24T12:01:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:08 smithi064 ceph-mon[42243]: pgmap v388: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 12548/263400 objects misplaced (4.764%); 37 MiB/s, 593 objects/s recovering 2024-03-24T12:01:09.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:08 smithi193 ceph-mon[33593]: 8.4 scrub ok 2024-03-24T12:01:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:08 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:08 smithi193 ceph-mon[33593]: 8.3 scrub starts 2024-03-24T12:01:09.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:08 smithi193 ceph-mon[33593]: pgmap v388: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 12548/263400 objects misplaced (4.764%); 37 MiB/s, 593 objects/s recovering 2024-03-24T12:01:09.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:09.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:09.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:09.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:09.293 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:09.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:09.394 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:09.429 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:09.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:09.530 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:09.562 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:09.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:09.663 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:09.696 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:09.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:09.723 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:01:09.723 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:01:09.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:09.852 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:09.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:09.956 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:09.991 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:09.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:10.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:09 smithi064 ceph-mon[30223]: 8.3 scrub ok 2024-03-24T12:01:10.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:09 smithi064 ceph-mon[30223]: 8.8 scrub starts 2024-03-24T12:01:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:09 smithi064 ceph-mon[42243]: 8.3 scrub ok 2024-03-24T12:01:10.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:09 smithi064 ceph-mon[42243]: 8.8 scrub starts 2024-03-24T12:01:10.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:09 smithi193 ceph-mon[33593]: 8.3 scrub ok 2024-03-24T12:01:10.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:09 smithi193 ceph-mon[33593]: 8.8 scrub starts 2024-03-24T12:01:10.093 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:10.128 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:10.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:10.231 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:10.260 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:10.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:10.362 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:10.390 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:10.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:10.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:10.492 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:10.521 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:10.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:10.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:10.654 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:10.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:10.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:01:10.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:10.790 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:10.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:10.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:01:10.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:10.949 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:10.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:11.050 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:11.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:10 smithi064 ceph-mon[30223]: 8.8 scrub ok 2024-03-24T12:01:11.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:10 smithi064 ceph-mon[30223]: 8.19 scrub starts 2024-03-24T12:01:11.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:10 smithi064 ceph-mon[30223]: pgmap v389: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 670 op/s; 12548/263400 objects misplaced (4.764%); 30 MiB/s, 487 objects/s recovering 2024-03-24T12:01:11.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:10 smithi064 ceph-mon[30223]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T12:01:11.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:10 smithi064 ceph-mon[42243]: 8.8 scrub ok 2024-03-24T12:01:11.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:10 smithi064 ceph-mon[42243]: 8.19 scrub starts 2024-03-24T12:01:11.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:10 smithi064 ceph-mon[42243]: pgmap v389: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 670 op/s; 12548/263400 objects misplaced (4.764%); 30 MiB/s, 487 objects/s recovering 2024-03-24T12:01:11.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:10 smithi064 ceph-mon[42243]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T12:01:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:11.082 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:11.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:11.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:10 smithi193 ceph-mon[33593]: 8.8 scrub ok 2024-03-24T12:01:11.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:10 smithi193 ceph-mon[33593]: 8.19 scrub starts 2024-03-24T12:01:11.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:10 smithi193 ceph-mon[33593]: pgmap v389: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 670 op/s; 12548/263400 objects misplaced (4.764%); 30 MiB/s, 487 objects/s recovering 2024-03-24T12:01:11.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:10 smithi193 ceph-mon[33593]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T12:01:11.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:11.235 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:11.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:11.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:11.376 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:11.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:11.478 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:11.510 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:11.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:11.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:11.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:11.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:11.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:11.792 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:11.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:11.894 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:11.906 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T12:01:11.906 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T12:01:11.906 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T12:01:11.907 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T12:01:11.928 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:11.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:12.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:12.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[30223]: 8.19 scrub ok 2024-03-24T12:01:12.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[30223]: 8.11 scrub starts 2024-03-24T12:01:12.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[30223]: 8.11 scrub ok 2024-03-24T12:01:12.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[30223]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T12:01:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[42243]: 8.19 scrub ok 2024-03-24T12:01:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[42243]: 8.11 scrub starts 2024-03-24T12:01:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[42243]: 8.11 scrub ok 2024-03-24T12:01:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[42243]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T12:01:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:12.077 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:12.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:12.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:11 smithi193 ceph-mon[33593]: 8.19 scrub ok 2024-03-24T12:01:12.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:11 smithi193 ceph-mon[33593]: 8.11 scrub starts 2024-03-24T12:01:12.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:11 smithi193 ceph-mon[33593]: 8.11 scrub ok 2024-03-24T12:01:12.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:11 smithi193 ceph-mon[33593]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T12:01:12.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:12.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:12.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:12.221 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:12.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:12.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:12.363 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:12.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:12.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:12.499 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:12.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:12.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:12.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:12.626 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:12.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:12.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:12.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:12.753 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:12.775 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:12.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:12.877 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:12.911 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:12.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:13.012 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:13.035 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:13.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:13.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:12 smithi193 ceph-mon[33593]: pgmap v392: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 163 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 869 B/s rd, 0 B/s wr, 819 op/s; 12233/260997 objects misplaced (4.687%); 35 MiB/s, 554 objects/s recovering 2024-03-24T12:01:13.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:12 smithi193 ceph-mon[33593]: 8.14 scrub starts 2024-03-24T12:01:13.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:12 smithi193 ceph-mon[33593]: 8.14 scrub ok 2024-03-24T12:01:13.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:13.168 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:13.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:13.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:13.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:12 smithi064 ceph-mon[42243]: pgmap v392: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 163 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 869 B/s rd, 0 B/s wr, 819 op/s; 12233/260997 objects misplaced (4.687%); 35 MiB/s, 554 objects/s recovering 2024-03-24T12:01:13.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:12 smithi064 ceph-mon[42243]: 8.14 scrub starts 2024-03-24T12:01:13.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:12 smithi064 ceph-mon[42243]: 8.14 scrub ok 2024-03-24T12:01:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:12 smithi064 ceph-mon[30223]: pgmap v392: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 163 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 869 B/s rd, 0 B/s wr, 819 op/s; 12233/260997 objects misplaced (4.687%); 35 MiB/s, 554 objects/s recovering 2024-03-24T12:01:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:12 smithi064 ceph-mon[30223]: 8.14 scrub starts 2024-03-24T12:01:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:12 smithi064 ceph-mon[30223]: 8.14 scrub ok 2024-03-24T12:01:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:13.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:13.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:13.443 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:13.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:13.545 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:13.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:13.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:13.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:13.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:13.719 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:13.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:13.820 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:13.854 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:13.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:13.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:13.995 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:13.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:14.096 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:14.121 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:14.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:14.223 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:14.255 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:14.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:14.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:14.390 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:14.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:14.492 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:14.519 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:14.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:14.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:14.649 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:14.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:14.751 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:14.775 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:14.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:14.877 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:14.912 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:14.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:14.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:15.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:15.050 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:15.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:15.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:15.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:15.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:15.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:14 smithi064 ceph-mon[30223]: pgmap v393: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 7483/249459 objects misplaced (3.000%) 2024-03-24T12:01:15.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:14 smithi064 ceph-mon[30223]: 8.2 scrub starts 2024-03-24T12:01:15.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:14 smithi064 ceph-mon[30223]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T12:01:15.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:14 smithi064 ceph-mon[30223]: 8.2 scrub ok 2024-03-24T12:01:15.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:14 smithi064 ceph-mon[42243]: pgmap v393: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 7483/249459 objects misplaced (3.000%) 2024-03-24T12:01:15.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:14 smithi064 ceph-mon[42243]: 8.2 scrub starts 2024-03-24T12:01:15.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:14 smithi064 ceph-mon[42243]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T12:01:15.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:14 smithi064 ceph-mon[42243]: 8.2 scrub ok 2024-03-24T12:01:15.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:15.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:15.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:14 smithi193 ceph-mon[33593]: pgmap v393: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 7483/249459 objects misplaced (3.000%) 2024-03-24T12:01:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:14 smithi193 ceph-mon[33593]: 8.2 scrub starts 2024-03-24T12:01:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:14 smithi193 ceph-mon[33593]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T12:01:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:14 smithi193 ceph-mon[33593]: 8.2 scrub ok 2024-03-24T12:01:15.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:01:15.426 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T12:01:15.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-24T12:01:15.438 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:15.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:15.450 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:15.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:15.565 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:15.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:15.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:15.696 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:15.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:15.798 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:15.823 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:15.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:15.925 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:15.965 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:15.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:15.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[30223]: 5.1b scrub starts 2024-03-24T12:01:15.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[30223]: 5.1b scrub ok 2024-03-24T12:01:15.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[30223]: 8.18 scrub starts 2024-03-24T12:01:15.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[30223]: 8.18 scrub ok 2024-03-24T12:01:15.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[30223]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T12:01:15.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[30223]: 8.7 scrub starts 2024-03-24T12:01:15.974 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[42243]: 5.1b scrub starts 2024-03-24T12:01:15.974 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[42243]: 5.1b scrub ok 2024-03-24T12:01:15.974 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[42243]: 8.18 scrub starts 2024-03-24T12:01:15.974 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[42243]: 8.18 scrub ok 2024-03-24T12:01:15.974 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[42243]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T12:01:15.974 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:15 smithi064 ceph-mon[42243]: 8.7 scrub starts 2024-03-24T12:01:16.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:16.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:16.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:16.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:16.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:16.325 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:15 smithi193 ceph-mon[33593]: 5.1b scrub starts 2024-03-24T12:01:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:15 smithi193 ceph-mon[33593]: 5.1b scrub ok 2024-03-24T12:01:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:15 smithi193 ceph-mon[33593]: 8.18 scrub starts 2024-03-24T12:01:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:15 smithi193 ceph-mon[33593]: 8.18 scrub ok 2024-03-24T12:01:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:15 smithi193 ceph-mon[33593]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T12:01:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:15 smithi193 ceph-mon[33593]: 8.7 scrub starts 2024-03-24T12:01:16.348 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:16.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:16.450 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:16.474 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:16.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:16.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:16.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:16.604 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:16.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:16.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:16.730 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:16.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:16.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:16.860 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:16.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:16.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:17.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:17.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:17.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:17.144 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:17.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:17.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:16 smithi193 ceph-mon[33593]: pgmap v396: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 7483/249459 objects misplaced (3.000%) 2024-03-24T12:01:17.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:16 smithi193 ceph-mon[33593]: 8.7 scrub ok 2024-03-24T12:01:17.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:16 smithi193 ceph-mon[33593]: 8.12 deep-scrub starts 2024-03-24T12:01:17.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:16 smithi193 ceph-mon[33593]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T12:01:17.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:16 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T12:01:17.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:16 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T12:01:17.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:16 smithi193 ceph-mon[33593]: 8.0 scrub starts 2024-03-24T12:01:17.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:16 smithi193 ceph-mon[33593]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T12:01:17.246 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:17.285 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:17.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[30223]: pgmap v396: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 7483/249459 objects misplaced (3.000%) 2024-03-24T12:01:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[30223]: 8.7 scrub ok 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[30223]: 8.12 deep-scrub starts 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[30223]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[30223]: 8.0 scrub starts 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[30223]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[42243]: pgmap v396: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 7483/249459 objects misplaced (3.000%) 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[42243]: 8.7 scrub ok 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[42243]: 8.12 deep-scrub starts 2024-03-24T12:01:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[42243]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T12:01:17.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T12:01:17.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T12:01:17.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[42243]: 8.0 scrub starts 2024-03-24T12:01:17.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:16 smithi064 ceph-mon[42243]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T12:01:17.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:17.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:17.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:17.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:17.570 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:17.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:17.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:01:17.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:17.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:17.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:17.721 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:01:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:17.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:17.844 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:17.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:17.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:17.975 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:17.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:18.037 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[30223]: 8.0 scrub ok 2024-03-24T12:01:18.037 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[30223]: 8.6 scrub starts 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[30223]: 8.15 scrub starts 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[30223]: 8.12 scrub starts 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[30223]: 8.16 deep-scrub starts 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[30223]: 8.6 scrub ok 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[42243]: 8.0 scrub ok 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[42243]: 8.6 scrub starts 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[42243]: 8.15 scrub starts 2024-03-24T12:01:18.038 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[42243]: 8.12 scrub starts 2024-03-24T12:01:18.039 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[42243]: 8.16 deep-scrub starts 2024-03-24T12:01:18.039 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:17 smithi064 ceph-mon[42243]: 8.6 scrub ok 2024-03-24T12:01:18.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:18.082 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:18.123 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:18.141 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:18.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:18.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:18.279 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:18.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-mon[33593]: 8.0 scrub ok 2024-03-24T12:01:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-mon[33593]: 8.6 scrub starts 2024-03-24T12:01:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:01:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:01:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-mon[33593]: 8.15 scrub starts 2024-03-24T12:01:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-mon[33593]: 8.12 scrub starts 2024-03-24T12:01:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-mon[33593]: 8.16 deep-scrub starts 2024-03-24T12:01:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:17 smithi193 ceph-mon[33593]: 8.6 scrub ok 2024-03-24T12:01:18.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:18.415 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:18.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:18.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:18.545 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:18.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:18.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:18.691 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:18.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:18.792 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:18.820 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:18.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:18.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:18.955 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:18.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:19.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:19.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: pgmap v399: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 164 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4950/239025 objects misplaced (2.071%) 2024-03-24T12:01:19.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: 8.15 scrub ok 2024-03-24T12:01:19.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: 8.12 scrub ok 2024-03-24T12:01:19.002 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: 8.1 scrub starts 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: 8.5 scrub starts 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: 8.13 scrub starts 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: 8.1 scrub ok 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: pgmap v399: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 164 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4950/239025 objects misplaced (2.071%) 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: 8.15 scrub ok 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: 8.12 scrub ok 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: 8.1 scrub starts 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: 8.5 scrub starts 2024-03-24T12:01:19.003 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: 8.13 scrub starts 2024-03-24T12:01:19.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: 8.1 scrub ok 2024-03-24T12:01:19.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:01:19.004 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:18 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:01:19.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:19.106 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:19.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:19.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:19.251 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:19.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: pgmap v399: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 164 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4950/239025 objects misplaced (2.071%) 2024-03-24T12:01:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: 8.15 scrub ok 2024-03-24T12:01:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: 8.12 scrub ok 2024-03-24T12:01:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: 8.1 scrub starts 2024-03-24T12:01:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: 8.5 scrub starts 2024-03-24T12:01:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: 8.13 scrub starts 2024-03-24T12:01:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: 8.1 scrub ok 2024-03-24T12:01:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:01:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:18 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:01:19.353 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:19.386 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:19.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:19.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:19.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:19.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:19.672 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:19.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:19.774 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:19.804 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:19.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:19.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:19.950 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:19.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:19.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:20.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:20.092 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:20.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: 8.5 scrub ok 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: 8.9 scrub starts 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: 8.13 scrub ok 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T12:01:20.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: 8.5 scrub ok 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: 8.9 scrub starts 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: 8.13 scrub ok 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T12:01:20.113 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:19 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T12:01:20.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:20.198 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:01:20.198 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:01:20.198 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:01:20.198 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:01:20.198 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:01:20.198 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:01:20.238 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:20.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:20.240 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:01:20.281 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:01:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:01:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:01:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:01:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:01:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:01:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:01:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:01:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:01:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: 8.5 scrub ok 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: 8.9 scrub starts 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: 8.13 scrub ok 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T12:01:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:19 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T12:01:20.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:20.385 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:20.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:20.487 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:20.520 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:20.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:20.622 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:20.653 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:20.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:20.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:20.793 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:20.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:20.894 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:20.926 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:20.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:20.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: pgmap v401: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 164 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4950/239025 objects misplaced (2.071%) 2024-03-24T12:01:20.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: 4.1b scrub starts 2024-03-24T12:01:20.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: 4.1b scrub ok 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: 8.9 scrub ok 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1736230745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: 8.17 scrub starts 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: 2.16 scrub starts 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[30223]: 2.16 scrub ok 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: pgmap v401: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 164 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4950/239025 objects misplaced (2.071%) 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: 4.1b scrub starts 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: 4.1b scrub ok 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: 8.9 scrub ok 2024-03-24T12:01:20.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T12:01:20.985 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1736230745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:20.985 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:01:20.985 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:01:20.985 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: 8.17 scrub starts 2024-03-24T12:01:20.985 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: 2.16 scrub starts 2024-03-24T12:01:20.985 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:20 smithi064 ceph-mon[42243]: 2.16 scrub ok 2024-03-24T12:01:21.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:21.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:21.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:21.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:21.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: pgmap v401: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 164 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4950/239025 objects misplaced (2.071%) 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: 4.1b scrub starts 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: 4.1b scrub ok 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: 8.9 scrub ok 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1736230745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: 8.17 scrub starts 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: 2.16 scrub starts 2024-03-24T12:01:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:20 smithi193 ceph-mon[33593]: 2.16 scrub ok 2024-03-24T12:01:21.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:21.344 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:21.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:21.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:21.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:21.578 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:21.610 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:21.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:21.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:21.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:21.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:21.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:21.895 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:21.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:01:21.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:22.040 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:22.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:22.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:22.183 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:22.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:22.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: 8.17 scrub ok 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: 6.19 scrub starts 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: 6.19 scrub ok 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2959058643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[30223]: from='client.35183 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: 8.17 scrub ok 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T12:01:22.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T12:01:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T12:01:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: 6.19 scrub starts 2024-03-24T12:01:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: 6.19 scrub ok 2024-03-24T12:01:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2959058643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:01:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:21 smithi064 ceph-mon[42243]: from='client.35183 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:01:22.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:22.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:22.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:22.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: 8.17 scrub ok 2024-03-24T12:01:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T12:01:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T12:01:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T12:01:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T12:01:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: 6.19 scrub starts 2024-03-24T12:01:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: 6.19 scrub ok 2024-03-24T12:01:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2959058643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:01:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:21 smithi193 ceph-mon[33593]: from='client.35183 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:01:22.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:22.463 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:22.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:22.565 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:22.600 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:22.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:22.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:22.732 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:22.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:22.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:22.866 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:22.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:22.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:23.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:23.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:23.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:23.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:23.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:23.285 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:23.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[30223]: pgmap v403: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 165 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 938 B/s rd, 0 op/s; 4950/235920 objects misplaced (2.098%) 2024-03-24T12:01:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[30223]: 4.d deep-scrub starts 2024-03-24T12:01:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[30223]: 4.d deep-scrub ok 2024-03-24T12:01:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[30223]: from='client.35183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:01:23.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[30223]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T12:01:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[42243]: pgmap v403: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 165 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 938 B/s rd, 0 op/s; 4950/235920 objects misplaced (2.098%) 2024-03-24T12:01:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[42243]: 4.d deep-scrub starts 2024-03-24T12:01:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[42243]: 4.d deep-scrub ok 2024-03-24T12:01:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[42243]: from='client.35183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:01:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:22 smithi064 ceph-mon[42243]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T12:01:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:22 smithi193 ceph-mon[33593]: pgmap v403: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 165 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 938 B/s rd, 0 op/s; 4950/235920 objects misplaced (2.098%) 2024-03-24T12:01:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:22 smithi193 ceph-mon[33593]: 4.d deep-scrub starts 2024-03-24T12:01:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:22 smithi193 ceph-mon[33593]: 4.d deep-scrub ok 2024-03-24T12:01:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:22 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:22 smithi193 ceph-mon[33593]: from='client.35183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:01:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:22 smithi193 ceph-mon[33593]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T12:01:23.387 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:23.415 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:23.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:23.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:23.517 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:23.551 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:23.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:23.652 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:23.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:01:23.675 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:23.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:23.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:23.831 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:23.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:23.933 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:23.964 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:23.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:23.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:24.066 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:24.099 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:24.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:24.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:24.240 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:24.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:24.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:24.375 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:24.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:24.477 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:24.512 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:24.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:24.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:24.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:24.752 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:24.775 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:24.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:24.877 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:24.919 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:24.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:25.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:25.055 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:25.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:25.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:25.181 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:25.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:25.282 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:25.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:24 smithi064 ceph-mon[30223]: pgmap v405: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4668/224517 objects misplaced (2.079%) 2024-03-24T12:01:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:24 smithi064 ceph-mon[42243]: pgmap v405: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4668/224517 objects misplaced (2.079%) 2024-03-24T12:01:25.342 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:24 smithi193 ceph-mon[33593]: pgmap v405: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4668/224517 objects misplaced (2.079%) 2024-03-24T12:01:25.370 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:25.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:25.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:25.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:25.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:25.644 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:25.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:25.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:25.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:25.885 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:25.914 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:25.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:26.016 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:26.055 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:26.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:26.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:26.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:26.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:26.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:26.356 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:26.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:26.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:26.513 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:26.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:26.615 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:26.648 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:26.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:26.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:26.783 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:26.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:26.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:26.910 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:01:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:175023 8.10 8:0de9df47:::benchmark_data_smithi064_120708_object52997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:01:26.883947+0000", 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.025819683999999999, 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025855473, 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 175023 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.883947+0000", 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.883947+0000", 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.883950+0000", 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-06 2024-03-24T12:01:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.883961+0000", 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0528e-05 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.883982+0000", 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1883999999999999e-05 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.883993+0000", 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.884252+0000", 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025812900000000001 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.884621+0000", 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036921700000000003 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,4,5,7", 2024-03-24T12:01:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.885385+0000", 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076415599999999999 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888402+0000", 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030167810000000001 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888455+0000", 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3427999999999997e-05 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888600+0000", 2024-03-24T12:01:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014509599999999999 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.894516+0000", 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059162210000000002 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:175028 8.10 8:0f8fd1b4:::benchmark_data_smithi064_120708_object53002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:01:26.888909+0000", 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.020857803000000001, 2024-03-24T12:01:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021021914999999999, 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 175028 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888909+0000", 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888909+0000", 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888911+0000", 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4669999999999998e-06 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888917+0000", 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1099999999999999e-06 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888932+0000", 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4589999999999999e-05 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.888940+0000", 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.889100+0000", 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000160032 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.889419+0000", 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000318614 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,4,5,7", 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.891642+0000", 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002223559 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.894742+0000", 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030999280000000001 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.896396+0000", 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001653601 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.896577+0000", 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018142700000000001 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.900896+0000", 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043190440000000002 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:175024 8.0 e230/196 8:00c2085e:::benchmark_data_smithi064_120708_object52998:head v 230'8429, mlcod=230'8426)", 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:01:26.903642+0000", 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0061241100000000003, 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063992199999999997, 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903642+0000", 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903642+0000", 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903644+0000", 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.984e-06 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903655+0000", 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0672999999999999e-05 2024-03-24T12:01:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903673+0000", 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7577000000000001e-05 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903682+0000", 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7690000000000008e-06 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907187+0000", 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035041790000000001 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907219+0000", 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2481000000000003e-05 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:175036 8.a e230/178 8:5c0db7e2:::benchmark_data_smithi064_120708_object53010:head v 230'11039, mlcod=230'11036)", 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:01:26.903710+0000", 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0060568929999999998, 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063987660000000002, 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:01:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903710+0000", 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903710+0000", 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903710+0000", 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.16e-07 2024-03-24T12:01:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903716+0000", 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.682e-06 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903722+0000", 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3029999999999996e-06 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903728+0000", 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1510000000000003e-06 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903806+0000", 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8091999999999997e-05 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.903828+0000", 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1835e-05 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:175040 8.15 e230/218 8:ae51c1e2:::benchmark_data_smithi064_120708_object53014:head v 230'8752, mlcod=230'8750)", 2024-03-24T12:01:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:01:26.907509+0000", 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0022580220000000002, 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026701989999999998, 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907509+0000", 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907509+0000", 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907512+0000", 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5829999999999998e-06 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907520+0000", 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.123e-06 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907535+0000", 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4965e-05 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907543+0000", 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8450000000000003e-06 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907595+0000", 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1641999999999998e-05 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:26.907614+0000", 2024-03-24T12:01:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8859e-05 2024-03-24T12:01:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:01:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 5 2024-03-24T12:01:26.932 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:01:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:26.946 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:26.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:27.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:26 smithi064 ceph-mon[42243]: pgmap v406: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 4668/224517 objects misplaced (2.079%) 2024-03-24T12:01:27.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:27.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:27.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:26 smithi064 ceph-mon[42243]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T12:01:27.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:26 smithi064 ceph-mon[30223]: pgmap v406: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 4668/224517 objects misplaced (2.079%) 2024-03-24T12:01:27.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:27.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:27.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:26 smithi064 ceph-mon[30223]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T12:01:27.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:27.085 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:27.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:27.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:27.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:26 smithi193 ceph-mon[33593]: pgmap v406: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 4668/224517 objects misplaced (2.079%) 2024-03-24T12:01:27.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:27.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:27.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:26 smithi193 ceph-mon[33593]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T12:01:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:27.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:27.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:27.371 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:27.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:27.472 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:27.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:27.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:27.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:01:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:01:27.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:27.633 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:27.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:27.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:27.765 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:27.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:27.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:27.777 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:01:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:01:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:27.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:27.905 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:27.906 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:27.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:28.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:28.046 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T12:01:28.046 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T12:01:28.046 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T12:01:28.046 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T12:01:28.046 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T12:01:28.046 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T12:01:28.046 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T12:01:28.047 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:01:28.047 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T12:01:28.047 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T12:01:28.047 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T12:01:28.047 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T12:01:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:28.058 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:28.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:28.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:28.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:28.301 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:28.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:01:28.336 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:28.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:28.438 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:28.466 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:28.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:28.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:01:28.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:28.638 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:28.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:28.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:28.768 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:28.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:28.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:28.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:28.919 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:28.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:29.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:29.051 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:29.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:29.153 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:29.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:29.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:28 smithi064 ceph-mon[42243]: pgmap v408: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 4318/208071 objects misplaced (2.075%) 2024-03-24T12:01:29.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:29.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:28 smithi064 ceph-mon[30223]: pgmap v408: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 4318/208071 objects misplaced (2.075%) 2024-03-24T12:01:29.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:29.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:01:29.202 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T12:01:29.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-24T12:01:29.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:29.318 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:29.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:29.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:29.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:28 smithi193 ceph-mon[33593]: pgmap v408: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 4318/208071 objects misplaced (2.075%) 2024-03-24T12:01:29.420 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:29.448 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:29.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:29.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:29.610 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:29.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:29.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:29.749 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:29.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:29.851 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:29.889 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:29.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:29.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:30.053 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:30.056 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:30.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:30.158 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:30.186 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:30.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:30.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:30.323 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:30.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:30.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:30.464 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:30.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:30.566 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:30.601 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:30.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:30.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:30.744 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:30.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:30.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:30.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:30.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:30.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:31.011 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:31.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:31.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:31.146 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:31.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:31.194 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:30 smithi064 ceph-mon[30223]: pgmap v409: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4318/208071 objects misplaced (2.075%) 2024-03-24T12:01:31.194 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:30 smithi064 ceph-mon[30223]: 8.16 deep-scrub ok 2024-03-24T12:01:31.194 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:30 smithi064 ceph-mon[42243]: pgmap v409: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4318/208071 objects misplaced (2.075%) 2024-03-24T12:01:31.194 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:30 smithi064 ceph-mon[42243]: 8.16 deep-scrub ok 2024-03-24T12:01:31.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:31.286 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:31.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:30 smithi193 ceph-mon[33593]: pgmap v409: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4318/208071 objects misplaced (2.075%) 2024-03-24T12:01:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:30 smithi193 ceph-mon[33593]: 8.16 deep-scrub ok 2024-03-24T12:01:31.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:31.427 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:31.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:31.528 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:31.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:31.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:31.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:31.701 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:31.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:31.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:31.843 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:31.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:31.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:31.985 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:31.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:32.086 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:32.119 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:32.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:32.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:32.263 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:32.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:32.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:32.396 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:32.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:32.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:32.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:32.541 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:32.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:32.643 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:32.677 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:32.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:32.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:32.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:32.811 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:32.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:32.912 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:32.944 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:32.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:33.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:33.088 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:33.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:33.189 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:33.224 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:33.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:33.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:32 smithi064 ceph-mon[30223]: pgmap v410: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 545 B/s rd, 0 op/s; 4318/203982 objects misplaced (2.117%) 2024-03-24T12:01:33.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:32 smithi064 ceph-mon[42243]: pgmap v410: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 545 B/s rd, 0 op/s; 4318/203982 objects misplaced (2.117%) 2024-03-24T12:01:33.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:33.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:32 smithi193 ceph-mon[33593]: pgmap v410: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 166 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 545 B/s rd, 0 op/s; 4318/203982 objects misplaced (2.117%) 2024-03-24T12:01:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:33.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:33.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:33.507 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:33.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:33.609 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:33.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:33.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:33.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:33.778 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:33.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:33.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:33.920 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:33.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:33.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:34.022 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:34.055 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:34.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:34.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:34.192 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:34.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:34.267 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:34.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:34.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:34.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:34.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:34.469 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:34.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:34.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:34.598 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:34.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:34.699 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:34.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:34.730 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:34.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:34.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:34.865 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:34.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:34.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:34.990 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:34.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:35.092 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:35.130 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:35.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:35.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:35.274 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:35.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:35.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:34 smithi064 ceph-mon[30223]: pgmap v411: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s; 3962/190833 objects misplaced (2.076%) 2024-03-24T12:01:35.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:34 smithi064 ceph-mon[42243]: pgmap v411: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s; 3962/190833 objects misplaced (2.076%) 2024-03-24T12:01:35.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:34 smithi193 ceph-mon[33593]: pgmap v411: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s; 3962/190833 objects misplaced (2.076%) 2024-03-24T12:01:35.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:35.403 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:35.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:35.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:35.535 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:35.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:35.637 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:35.672 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:35.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:35.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:35.818 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:35.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:35.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:35.966 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:35.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:36.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:36.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:36.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:36.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:36.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:36.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:36.348 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:36.381 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:36.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:36.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:36.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:36.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:36.622 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:36.657 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:36.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:36.759 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:36.794 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:36.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:36.895 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:36.929 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:36.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:36.973 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:01:37.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:37.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:36 smithi064 ceph-mon[30223]: pgmap v412: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s; 3962/190833 objects misplaced (2.076%) 2024-03-24T12:01:37.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:36 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2918621133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:37.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:36 smithi064 ceph-mon[30223]: from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:36 smithi064 ceph-mon[42243]: pgmap v412: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s; 3962/190833 objects misplaced (2.076%) 2024-03-24T12:01:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:36 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2918621133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:37.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:36 smithi064 ceph-mon[42243]: from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:37.068 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:37.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:37.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:37.211 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:37.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:37.241 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:36 smithi193 ceph-mon[33593]: pgmap v412: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s; 3962/190833 objects misplaced (2.076%) 2024-03-24T12:01:37.241 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:36 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2918621133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:37.241 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:36 smithi193 ceph-mon[33593]: from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:01:37.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:37.347 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:37.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:37.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:37.482 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:37.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:37.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:37.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:37.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:01:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:01:37.614 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:01:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:01:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:37.627 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:37.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:37.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:37.763 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:37.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:37.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:37.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:37.903 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:37.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:38.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:38.040 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:38.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:38.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:38.142 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:38.174 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:38.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:38.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:37 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:38.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:37 smithi064 ceph-mon[30223]: from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:01:38.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:37 smithi064 ceph-mon[30223]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T12:01:38.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:37 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:38.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:37 smithi064 ceph-mon[42243]: from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:01:38.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:37 smithi064 ceph-mon[42243]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T12:01:38.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:38.319 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:38.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:38.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:37 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:38.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:37 smithi193 ceph-mon[33593]: from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:01:38.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:37 smithi193 ceph-mon[33593]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T12:01:38.405 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:01:38.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:38.445 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:01:38.450 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:38.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:38.482 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:01:38.537 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:01:38.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:38.584 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:01:38.595 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:38.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:38.622 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:01:38.662 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:01:38.697 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:38.702 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:01:38.711 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:01:38.711 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:01:38.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:01:38.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:01:38.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:01:38.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:01:38.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:01:38.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:01:38.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:38.732 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:38.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:38.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:38.871 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:38.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:38.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:39.002 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:39.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:39.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:39.135 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:39.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:39.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:39.266 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:39.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:39.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:38 smithi064 ceph-mon[30223]: pgmap v414: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 702 op/s; 3546/171327 objects misplaced (2.070%) 2024-03-24T12:01:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1163773445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:38 smithi064 ceph-mon[42243]: pgmap v414: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 702 op/s; 3546/171327 objects misplaced (2.070%) 2024-03-24T12:01:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1163773445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:38 smithi193 ceph-mon[33593]: pgmap v414: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 702 op/s; 3546/171327 objects misplaced (2.070%) 2024-03-24T12:01:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1163773445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:39.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:39.406 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:39.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:39.509 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:39.543 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:39.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:39.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:39.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:39.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:39.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:39.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:39.812 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:39.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:39.914 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:39.947 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:39.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:40.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:40.095 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:40.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:40.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:40.235 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:40.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:40.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:40.392 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:40.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:40.493 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:40.522 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:40.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:40.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:40.656 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:40.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:40.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:40.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:40.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T12:01:40.898 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T12:01:40.899 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T12:01:40.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:40.948 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:40.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:40.959 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:01:40.959 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:01:41.050 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:41.078 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:41.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:41.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:41 smithi064 ceph-mon[30223]: pgmap v415: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 702 op/s; 3546/171327 objects misplaced (2.070%) 2024-03-24T12:01:41.158 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:41 smithi064 ceph-mon[42243]: pgmap v415: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 702 op/s; 3546/171327 objects misplaced (2.070%) 2024-03-24T12:01:41.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:41.231 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:41.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:41.333 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:41.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:41 smithi193 ceph-mon[33593]: pgmap v415: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 702 op/s; 3546/171327 objects misplaced (2.070%) 2024-03-24T12:01:41.367 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:41.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:41.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:41.493 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:41.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:41.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:41.625 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:41.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:41.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:41.757 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:41.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:41.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:01:41.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:41.908 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:41.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:42.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:42.055 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:42.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:42.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:42.203 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:42.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:42.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:42.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:42 smithi064 ceph-mon[30223]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T12:01:42.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:42.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:42 smithi064 ceph-mon[42243]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T12:01:42.305 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:42.334 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:42.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:42 smithi193 ceph-mon[33593]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T12:01:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:01:42.354 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T12:01:42.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-24T12:01:42.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:01:42.436 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:42.469 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:42.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:42.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:42.667 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:42.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:42.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:42.809 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:42.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:42.910 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:42.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:01:42.946 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:42.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:42.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:43.023 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:43.033 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:43.047 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:43.082 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:43.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:43.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:43 smithi064 ceph-mon[30223]: pgmap v416: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 738 op/s; 3546/166158 objects misplaced (2.134%) 2024-03-24T12:01:43.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:43 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:01:43.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:43 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:01:43.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:43.155 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:43 smithi064 ceph-mon[42243]: pgmap v416: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 738 op/s; 3546/166158 objects misplaced (2.134%) 2024-03-24T12:01:43.155 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:43 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:01:43.155 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:43 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:01:43.155 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:43.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:43.245 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:43.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:43 smithi193 ceph-mon[33593]: pgmap v416: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 738 op/s; 3546/166158 objects misplaced (2.134%) 2024-03-24T12:01:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:43 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:01:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:43 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:01:43.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:43.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:43.436 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:43.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:43.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:43.584 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:43.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:43.685 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:43.718 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:43.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:43.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:43.856 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:43.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:43.958 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:43.998 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:43.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:44.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:44.140 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:44.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:44.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:44.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:44 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:01:44.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:44 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:01:44.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:44 smithi064 ceph-mon[30223]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T12:01:44.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:44 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:01:44.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:44 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:01:44.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:44 smithi064 ceph-mon[42243]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T12:01:44.281 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:44.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:44.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:44 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:01:44.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:44 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:01:44.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:44 smithi193 ceph-mon[33593]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T12:01:44.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:44.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:44.425 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:44.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:44.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:44.571 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:44.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:44.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:44.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:44.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:44.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:44.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:44.962 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:44.994 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:44.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:45.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:45.137 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:45.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:45.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:45.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:45.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:45.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:45 smithi064 ceph-mon[30223]: pgmap v419: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 1.48k op/s; 3136/151518 objects misplaced (2.070%) 2024-03-24T12:01:45.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:45 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:01:45.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:45 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:01:45.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:45 smithi064 ceph-mon[30223]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T12:01:45.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:45 smithi064 ceph-mon[42243]: pgmap v419: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 1.48k op/s; 3136/151518 objects misplaced (2.070%) 2024-03-24T12:01:45.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:45 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:01:45.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:45 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:01:45.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:45 smithi064 ceph-mon[42243]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T12:01:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:45 smithi193 ceph-mon[33593]: pgmap v419: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 1.48k op/s; 3136/151518 objects misplaced (2.070%) 2024-03-24T12:01:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:45 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:01:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:45 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:01:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:45 smithi193 ceph-mon[33593]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T12:01:45.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:45.413 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:45.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:45.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:45.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:45.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:45.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:45.699 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:45.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:45.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:45.801 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:45.881 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:45.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:45.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:46.021 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:46.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:46.123 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:46.154 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:46.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:46.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:46.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:46 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:46 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:46 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:46.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:46.415 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:46.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:46.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:46 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:46.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:46 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:46.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:46 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:46.431 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:46 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:46.431 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:46 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:46.431 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:46 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:01:46.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:46.545 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:46.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:46.647 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:46.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:46.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:46.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:46.815 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:46.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:46.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:46.954 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:46.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:47.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:47.085 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:47.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:47.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:47.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:47.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:47.329 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:47.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:01:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:01:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:47 smithi193 ceph-mon[33593]: pgmap v421: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 3136/151518 objects misplaced (2.070%) 2024-03-24T12:01:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:47 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:01:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:47 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:01:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:47 smithi193 ceph-mon[33593]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T12:01:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:47 smithi193 ceph-mon[33593]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T12:01:47.352 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:47.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[30223]: pgmap v421: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 3136/151518 objects misplaced (2.070%) 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[30223]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[30223]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[42243]: pgmap v421: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 3136/151518 objects misplaced (2.070%) 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:01:47.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[42243]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T12:01:47.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-mon[42243]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T12:01:47.454 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:47.486 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:47.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:47.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:47.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:47.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:47.761 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:47.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:47.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:01:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:01:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:47.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:47.899 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:47.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:48.001 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:48.038 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:48.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:48.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:48.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:48.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:48.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:48.279 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:48.312 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:48.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:48.414 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:48.446 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:48.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:48.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:48.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:48.586 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:48.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:48.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:48.724 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:48.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:48.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:48.872 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:48.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:48.974 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:49.007 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:49.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:49.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:49.139 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:49.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:49.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:49.285 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:49.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:49 smithi193 ceph-mon[33593]: pgmap v424: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 2780/133347 objects misplaced (2.085%) 2024-03-24T12:01:49.387 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:49.410 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:49.410 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:49 smithi064 ceph-mon[30223]: pgmap v424: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 2780/133347 objects misplaced (2.085%) 2024-03-24T12:01:49.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:49.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:49 smithi064 ceph-mon[42243]: pgmap v424: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 2780/133347 objects misplaced (2.085%) 2024-03-24T12:01:49.417 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:49.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:49.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:49.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:49.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:49.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:49.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:49.698 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:49.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:49.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:49.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:49.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:49.987 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:49.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:50.088 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:50.121 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:50.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:50.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:50.267 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:50.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:50.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:50.410 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:50.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:50.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:50.545 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:50.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:50.609 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T12:01:50.609 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T12:01:50.609 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T12:01:50.609 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T12:01:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T12:01:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T12:01:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T12:01:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:01:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T12:01:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T12:01:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T12:01:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T12:01:50.647 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:50.678 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:50.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:50.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:50.696 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:01:50.742 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:01:50.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:50.781 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:01:50.811 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:50.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:50.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:01:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:01:50.904 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:01:50.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:50.943 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:01:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:50.953 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:50.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:50.984 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:01:50.989 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:01:50.989 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:01:50.990 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:01:50.990 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:01:50.990 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:01:50.990 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:01:50.990 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:01:50.990 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:01:51.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:51.103 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:51.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:51.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:51.239 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:51.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:51 smithi064 ceph-mon[30223]: pgmap v425: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 2780/133347 objects misplaced (2.085%) 2024-03-24T12:01:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:51 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2244043434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:51 smithi064 ceph-mon[42243]: pgmap v425: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 2780/133347 objects misplaced (2.085%) 2024-03-24T12:01:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:51 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2244043434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:51 smithi193 ceph-mon[33593]: pgmap v425: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 2780/133347 objects misplaced (2.085%) 2024-03-24T12:01:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:51 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2244043434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:01:51.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:51.380 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:51.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:51.482 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:51.515 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:51.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:51.616 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:51.656 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:51.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:51.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:51.799 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:51.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:01:51.854 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T12:01:51.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-24T12:01:51.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:51.992 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:51.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:52.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:52.126 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:52.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:52.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:52.264 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:52.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:52.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:52.404 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:52.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:52.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:52.550 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:52.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:52.652 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:52.690 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:52.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:52.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:52.826 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:52.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:52.928 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:52.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:52.963 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:52.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:53.064 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:53.100 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:53.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:53.201 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:53.231 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:53.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:53.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:53 smithi193 ceph-mon[33593]: pgmap v426: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 705 B/s rd, 0 B/s wr, 1.05k op/s; 2780/128733 objects misplaced (2.160%) 2024-03-24T12:01:53.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:53 smithi064 ceph-mon[30223]: pgmap v426: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 705 B/s rd, 0 B/s wr, 1.05k op/s; 2780/128733 objects misplaced (2.160%) 2024-03-24T12:01:53.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:53 smithi064 ceph-mon[42243]: pgmap v426: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 705 B/s rd, 0 B/s wr, 1.05k op/s; 2780/128733 objects misplaced (2.160%) 2024-03-24T12:01:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:53.375 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:53.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:53.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:53.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:53.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:53.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:53.654 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:53.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:53.756 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:53.791 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:53.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:53.893 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:53.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:53.951 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:53.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:54.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:54.085 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:54.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:54.187 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:54.215 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:54.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:54.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:54.351 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:54.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:54.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:54.487 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:54.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:54.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:54.624 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:54.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:54.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:54.765 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:54.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:54.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:54.903 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:54.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:55.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:55.049 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:55.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:55.102 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:01:55.151 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:55.182 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:55.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:55.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:55.311 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:55.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:55 smithi193 ceph-mon[33593]: pgmap v427: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 2406/114831 objects misplaced (2.095%) 2024-03-24T12:01:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:55 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/262324067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:01:55.413 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:55.446 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:55.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:55.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:55.548 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:55 smithi064 ceph-mon[30223]: pgmap v427: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 2406/114831 objects misplaced (2.095%) 2024-03-24T12:01:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:55 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/262324067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:01:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:55 smithi064 ceph-mon[42243]: pgmap v427: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 2406/114831 objects misplaced (2.095%) 2024-03-24T12:01:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:55 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/262324067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:01:55.584 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:55.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:55.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:55.724 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:55.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:55.826 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:55.861 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:55.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:55.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:56.006 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:56.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:56.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:56.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:56.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:56.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:56.274 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:56.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:56 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:56 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/262324067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:01:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:56 smithi193 ceph-mon[33593]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T12:01:56.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:56.416 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:56.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:56.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:56.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:56 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:56.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:56 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/262324067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:01:56.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:56 smithi064 ceph-mon[30223]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T12:01:56.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:56 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:01:56.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:56 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/262324067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:01:56.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:56 smithi064 ceph-mon[42243]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T12:01:56.553 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:56.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:56.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:56.697 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:56.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:56.798 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:56.833 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:56.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:56.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:56.971 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:56.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:57.073 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:57.101 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:57.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:57.189 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-mon[30223]: pgmap v429: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 2406/114831 objects misplaced (2.095%) 2024-03-24T12:01:57.189 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:57.190 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-mon[30223]: 5.16 scrub starts 2024-03-24T12:01:57.190 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-mon[30223]: 5.16 scrub ok 2024-03-24T12:01:57.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-mon[42243]: pgmap v429: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 2406/114831 objects misplaced (2.095%) 2024-03-24T12:01:57.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:57.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-mon[42243]: 5.16 scrub starts 2024-03-24T12:01:57.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-mon[42243]: 5.16 scrub ok 2024-03-24T12:01:57.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:57.232 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:57.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:57.309 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215974 7.12 7:49827d29:::benchmark_data_smithi064_87013_object95017:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.912925+0000", 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 252.38638844600001, 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1524356259999999, 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215974 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912925+0000", 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912925+0000", 2024-03-24T12:01:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912927+0000", 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9910000000000001e-06 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912932+0000", 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5979999999999999e-06 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.004832+0000", 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091900446999999996 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.004846+0000", 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4114000000000001e-05 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009898+0000", 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050523190000000004 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009908+0000", 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.065275+0000", 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0553668060000001 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.065361+0000", 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5686999999999995e-05 2024-03-24T12:01:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30713 8.7 8:e176b16f:::benchmark_data_smithi064_120708_object30712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.805040+0000", 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.49427376200001, 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.131868809, 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30713 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805040+0000", 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805040+0000", 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805041+0000", 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805077+0000", 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5411999999999999e-05 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805092+0000", 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.575e-05 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805101+0000", 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8470000000000002e-06 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.806250+0000", 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011484959999999999 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.806361+0000", 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011084 2024-03-24T12:01:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.806848+0000", 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000487164 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.810513+0000", 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036655379999999999 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814134+0000", 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003620393 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936775+0000", 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12264096400000001 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936843+0000", 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8485999999999994e-05 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936908+0000", 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5298999999999999e-05 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30732 8.7 8:e01cf41c:::benchmark_data_smithi064_120708_object30731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.811503+0000", 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.487810354, 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763239600000001, 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30732 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811503+0000", 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811503+0000", 2024-03-24T12:01:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811504+0000", 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.02e-07 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811524+0000", 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.986e-05 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811531+0000", 2024-03-24T12:01:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0079999999999996e-06 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811536+0000", 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5430000000000001e-06 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814205+0000", 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026690009999999998 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814384+0000", 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017845 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814516+0000", 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013214100000000001 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936989+0000", 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.122473472 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937025+0000", 2024-03-24T12:01:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5799000000000001e-05 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939021+0000", 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019962000000000001 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939102+0000", 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939135+0000", 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3544999999999998e-05 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30738 8.a 8:522e4dbb:::benchmark_data_smithi064_120708_object30737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.814514+0000", 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.48479959900001, 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13393280499999999, 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30738 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814514+0000", 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814514+0000", 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814516+0000", 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1040000000000001e-06 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814539+0000", 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3587999999999998e-05 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814552+0000", 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.222e-05 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814560+0000", 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7499999999999992e-06 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814623+0000", 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2510999999999998e-05 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936947+0000", 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.122324012 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937574+0000", 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000626741 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.940677+0000", 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031032949999999998 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.943485+0000", 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028083100000000001 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.947014+0000", 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035284399999999999 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.948401+0000", 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013869589999999999 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.948447+0000", 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5874999999999997e-05 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30739 8.3 8:cfd8eaff:::benchmark_data_smithi064_120708_object30738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.814697+0000", 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.48461595200001, 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130831052, 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30739 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814697+0000", 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814697+0000", 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814699+0000", 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814720+0000", 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1676e-05 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814729+0000", 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6400000000000003e-06 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814737+0000", 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7049999999999999e-06 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937094+0000", 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.122357138 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937377+0000", 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028292399999999999 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937530+0000", 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153752 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939685+0000", 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021549360000000001 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.942092+0000", 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024068639999999999 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945380+0000", 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003288146 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945480+0000", 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9499000000000004e-05 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945528+0000", 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8529000000000002e-05 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30741 8.3 8:c156d5da:::benchmark_data_smithi064_120708_object30740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.815142+0000", 2024-03-24T12:01:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.48417118099999, 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13056442700000001, 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30741 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.815142+0000", 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.815142+0000", 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.815144+0000", 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.815165+0000", 2024-03-24T12:01:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1237999999999999e-05 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936625+0000", 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121460399 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936650+0000", 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4215000000000001e-05 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939720+0000", 2024-03-24T12:01:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030704870000000002 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939835+0000", 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115309 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939996+0000", 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016097600000000001 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.942080+0000", 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002083807 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.943585+0000", 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001505008 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945578+0000", 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019926800000000001 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945667+0000", 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9141000000000006e-05 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945707+0000", 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9694000000000003e-05 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:31212 8.3 8:cc964d41:::benchmark_data_smithi064_120708_object31211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:13.148461+0000", 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.15085275800001, 2024-03-24T12:01:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.129072409, 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31212 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.148461+0000", 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.148461+0000", 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.148462+0000", 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6279999999999999e-06 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.148485+0000", 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2901999999999999e-05 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.258790+0000", 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110304771 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.258810+0000", 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0080000000000001e-05 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.260729+0000", 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019188930000000001 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.260872+0000", 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014303800000000001 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.261004+0000", 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013254 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.263354+0000", 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023496070000000001 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:01:57.349 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.263481+0000", 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012694099999999999 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.265111+0000", 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016302350000000001 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.265281+0000", 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016961799999999999 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.277533+0000", 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012252156 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:34129 8.3 8:ccbd3bd8:::benchmark_data_smithi064_120708_object34128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:01:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.762014+0000", 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.53729970699999, 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16982466299999999, 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34129 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762014+0000", 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762014+0000", 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762015+0000", 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.734e-06 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762050+0000", 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4314999999999999e-05 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762061+0000", 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0957e-05 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762069+0000", 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5369999999999997e-06 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.764178+0000", 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002108651 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.764606+0000", 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042792699999999998 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.764779+0000", 2024-03-24T12:01:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017316099999999999 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928527+0000", 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.163748002 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928538+0000", 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1559000000000001e-05 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931737+0000", 2024-03-24T12:01:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031986919999999999 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931793+0000", 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6141999999999999e-05 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931838+0000", 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4985999999999999e-05 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:34130 8.b e176/170 8:dce7680e:::benchmark_data_smithi064_120708_object34129:head v 176'2067, mlcod=176'2065)", 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.762745+0000", 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.53656867699999, 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16556532800000001, 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762745+0000", 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762745+0000", 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762746+0000", 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.001e-06 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762774+0000", 2024-03-24T12:01:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8359000000000001e-05 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762783+0000", 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6500000000000002e-06 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762789+0000", 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2840000000000003e-06 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.763014+0000", 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022540099999999999 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.763033+0000", 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8423999999999998e-05 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766313+0000", 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00327999 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928310+0000", 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.161997219 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:34127 8.f e176/170 8:f45926e9:::benchmark_data_smithi064_120708_object34126:head v 176'2120, mlcod=176'2115)", 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.765432+0000", 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.53388133000001, 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16544773300000001, 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765432+0000", 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765432+0000", 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765433+0000", 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7400000000000003e-07 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765457+0000", 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.455e-05 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765465+0000", 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6709999999999999e-06 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765471+0000", 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8109999999999999e-06 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765567+0000", 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5957000000000003e-05 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765586+0000", 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9292000000000001e-05 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.930801+0000", 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16521519200000001 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.930880+0000", 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8685999999999998e-05 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:34132 8.f e176/170 8:f6dfb55d:::benchmark_data_smithi064_120708_object34131:head v 176'2121, mlcod=176'2115)", 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.766181+0000", 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.53313214100001, 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16485608900000001, 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766181+0000", 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766181+0000", 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766182+0000", 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0950000000000001e-06 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766206+0000", 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3589000000000001e-05 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766215+0000", 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9779999999999994e-06 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766222+0000", 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6819999999999997e-06 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928430+0000", 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.162208239 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928454+0000", 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4677000000000001e-05 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931008+0000", 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025538549999999998 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931037+0000", 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8974000000000001e-05 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:34140 8.a 8:540add6c:::benchmark_data_smithi064_120708_object34139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.767290+0000", 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.53202301600001, 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18283492000000001, 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34140 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.767290+0000", 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.767290+0000", 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.767293+0000", 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7209999999999999e-06 2024-03-24T12:01:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.767316+0000", 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2836999999999998e-05 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928285+0000", 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16096931 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928302+0000", 2024-03-24T12:01:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7053000000000001e-05 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928700+0000", 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039811700000000002 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928828+0000", 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127329 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.929223+0000", 2024-03-24T12:01:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039511 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.930744+0000", 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015214829999999999 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.932965+0000", 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022207749999999999 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.949980+0000", 2024-03-24T12:01:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017014486999999998 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.950081+0000", 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010141699999999999 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.950125+0000", 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4280999999999998e-05 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35033.0:34129 8.3 e176/170 ondisk, result = 0)", 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.768222+0000", 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.531091581, 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16362191400000001, 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.768222+0000", 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.768222+0000", 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.768223+0000", 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.139e-06 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.768227+0000", 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9809999999999997e-06 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928596+0000", 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16036869000000001 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928604+0000", 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1880000000000002e-06 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931725+0000", 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003121268 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931732+0000", 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8900000000000001e-06 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931844+0000", 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111758 2024-03-24T12:01:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35033.0:34129 8.3 e176/170 ondisk, result = 0)", 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.769164+0000", 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.530149223, 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15937512700000001, 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.769164+0000", 2024-03-24T12:01:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.769164+0000", 2024-03-24T12:01:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.769165+0000", 2024-03-24T12:01:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.105e-06 2024-03-24T12:01:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.769169+0000", 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9530000000000001e-06 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928286+0000", 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15911722 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928301+0000", 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4228e-05 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928515+0000", 2024-03-24T12:01:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021393700000000001 2024-03-24T12:01:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928521+0000", 2024-03-24T12:01:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6579999999999996e-06 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928539+0000", 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8026e-05 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:35795 8.7 8:eacf5fe1:::benchmark_data_smithi064_120708_object35794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:15.841335+0000", 2024-03-24T12:01:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 161.45797853600001, 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14967005899999999, 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35795 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841335+0000", 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841335+0000", 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841336+0000", 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841359+0000", 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3053000000000001e-05 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841366+0000", 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4250000000000001e-06 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841373+0000", 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1049999999999997e-06 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841459+0000", 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6024000000000002e-05 2024-03-24T12:01:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841630+0000", 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017069700000000001 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841788+0000", 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158407 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.843971+0000", 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00218286 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.844208+0000", 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023623000000000001 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.989481+0000", 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14527379400000001 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.990963+0000", 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00148116 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.991005+0000", 2024-03-24T12:01:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2361999999999999e-05 2024-03-24T12:01:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104598 8.17 8:eb9d3bdd:::benchmark_data_smithi064_120708_object104597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.755205+0000", 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.54410826500001, 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.088392112, 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104598 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755205+0000", 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755205+0000", 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755208+0000", 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6960000000000001e-06 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755325+0000", 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117261 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755338+0000", 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3380000000000001e-05 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755352+0000", 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3876999999999999e-05 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755418+0000", 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5297999999999997e-05 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755425+0000", 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2949999999999996e-06 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.824824+0000", 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.069399319 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.825008+0000", 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018328800000000001 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.825376+0000", 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036875500000000002 2024-03-24T12:01:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.830186+0000", 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048092509999999996 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.830236+0000", 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0263000000000003e-05 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832844+0000", 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026079599999999999 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.833622+0000", 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000778238 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.843390+0000", 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097680479999999997 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.843462+0000", 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1722000000000002e-05 2024-03-24T12:01:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.843597+0000", 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135461 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104612 8.17 8:e8feba9a:::benchmark_data_smithi064_120708_object104611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.755373+0000", 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.54393994900001, 2024-03-24T12:01:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0905727569999999, 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104612 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755373+0000", 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755373+0000", 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755374+0000", 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0400000000000005e-07 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755396+0000", 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1469000000000001e-05 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755405+0000", 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7530000000000007e-06 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755412+0000", 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2080000000000003e-06 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755458+0000", 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6656999999999997e-05 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755467+0000", 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9339999999999995e-06 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.827928+0000", 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.07246029 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.828092+0000", 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164038 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.828454+0000", 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036203500000000003 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832895+0000", 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044415230000000002 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832911+0000", 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5593000000000001e-05 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832949+0000", 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8111999999999997e-05 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.833669+0000", 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071961200000000001 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.845732+0000", 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012063212 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.845802+0000", 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0227000000000004e-05 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.845946+0000", 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014418999999999999 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104620 8.17 8:ed89d4bc:::benchmark_data_smithi064_120708_object104619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.755421+0000", 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.54389218599999, 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.094105661, 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104620 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755421+0000", 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755421+0000", 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755422+0000", 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3099999999999997e-07 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755444+0000", 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1926e-05 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755451+0000", 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1119999999999998e-06 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755457+0000", 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1990000000000003e-06 2024-03-24T12:01:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755499+0000", 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1832999999999997e-05 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755508+0000", 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.037e-06 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.830590+0000", 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0750823780000001 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.830765+0000", 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174134 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831252+0000", 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048734400000000002 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.833578+0000", 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002326355 2024-03-24T12:01:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.833636+0000", 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7837000000000002e-05 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.838464+0000", 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048280160000000001 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.842219+0000", 2024-03-24T12:01:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037549889999999998 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.849143+0000", 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069238360000000001 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.849381+0000", 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000237722 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.849527+0000", 2024-03-24T12:01:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000146212 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:144410 8.17 8:eb4d00be:::benchmark_data_smithi064_120708_object22384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.632849481999997, 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17982219199999999, 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 144410 2024-03-24T12:01:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666465+0000", 2024-03-24T12:01:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.615e-06 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666470+0000", 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2420000000000002e-06 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666477+0000", 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6329999999999996e-06 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666484+0000", 2024-03-24T12:01:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2450000000000003e-06 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669073+0000", 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002589755 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669197+0000", 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012349999999999999 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669550+0000", 2024-03-24T12:01:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035300000000000002 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.710796+0000", 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041246370999999997 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711483+0000", 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068668600000000005 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711540+0000", 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7515000000000001e-05 2024-03-24T12:01:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711848+0000", 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030785799999999999 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.845781+0000", 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13393276200000001 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.845858+0000", 2024-03-24T12:01:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7063000000000004e-05 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846286+0000", 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042794699999999997 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:144423 8.17 8:ecb5801d:::benchmark_data_smithi064_120708_object22397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:52.711278+0000", 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.588035351000002, 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13569683299999999, 2024-03-24T12:01:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 144423 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711278+0000", 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711278+0000", 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711279+0000", 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.265e-06 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711284+0000", 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7319999999999998e-06 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:01:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711293+0000", 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1030000000000008e-06 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711301+0000", 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2449999999999998e-06 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711897+0000", 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059516500000000004 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.712007+0000", 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110769 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.712386+0000", 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037831600000000001 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.716109+0000", 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037229670000000002 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.716136+0000", 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7500000000000001e-05 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.716177+0000", 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0615999999999999e-05 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.716748+0000", 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057089300000000005 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846770+0000", 2024-03-24T12:01:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130021991 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846833+0000", 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3149000000000001e-05 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846975+0000", 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014212200000000001 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:01:57.429 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:01:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:57.430 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:57.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:57.440 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:01:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:01:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:57.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:57.562 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:57.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:57.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:01:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:01:57.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:57 smithi193 ceph-mon[33593]: pgmap v429: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 2406/114831 objects misplaced (2.095%) 2024-03-24T12:01:57.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:57.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:57 smithi193 ceph-mon[33593]: 5.16 scrub starts 2024-03-24T12:01:57.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:57 smithi193 ceph-mon[33593]: 5.16 scrub ok 2024-03-24T12:01:57.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:57.712 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:57.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:57.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:57.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:57.963 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:58.000 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:58.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:58.102 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:58.133 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:58.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:01:58.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:58.278 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:58.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:58.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:01:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:58.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:01:58.524 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:58.555 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:58.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:58.655 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:58.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:58.687 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:58.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:01:58.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:01:58.820 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:58.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:01:58.922 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:01:58.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:01:58.963 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:58.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:01:59.065 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:01:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:59.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:59.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:01:59.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:59.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:59.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[30223]: pgmap v430: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/95097 objects misplaced (2.130%) 2024-03-24T12:01:59.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[30223]: 8.a scrub starts 2024-03-24T12:01:59.244 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T12:01:59.244 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T12:01:59.244 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[30223]: 8.c scrub starts 2024-03-24T12:01:59.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:59.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[42243]: pgmap v430: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/95097 objects misplaced (2.130%) 2024-03-24T12:01:59.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[42243]: 8.a scrub starts 2024-03-24T12:01:59.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T12:01:59.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T12:01:59.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:01:59 smithi064 ceph-mon[42243]: 8.c scrub starts 2024-03-24T12:01:59.258 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:59.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:59.360 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:59.394 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:59.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:59.495 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:01:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:59.531 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:59.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:01:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:59 smithi193 ceph-mon[33593]: pgmap v430: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/95097 objects misplaced (2.130%) 2024-03-24T12:01:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:59 smithi193 ceph-mon[33593]: 8.a scrub starts 2024-03-24T12:01:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:59 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T12:01:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:59 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T12:01:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:01:59 smithi193 ceph-mon[33593]: 8.c scrub starts 2024-03-24T12:01:59.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:01:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:59.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:01:59.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:01:59.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:01:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:01:59.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:01:59.956 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:01:59.988 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:01:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:01:59.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:00.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:00.128 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:00.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:00.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:00.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[30223]: 8.a scrub ok 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[30223]: 8.c scrub ok 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[30223]: 8.e scrub starts 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[42243]: 8.a scrub ok 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[42243]: 8.c scrub ok 2024-03-24T12:02:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:00 smithi064 ceph-mon[42243]: 8.e scrub starts 2024-03-24T12:02:00.371 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:00.407 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:00.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:00.509 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:00.545 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:00.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:00 smithi193 ceph-mon[33593]: 8.a scrub ok 2024-03-24T12:02:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:00 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T12:02:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:00 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T12:02:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:00 smithi193 ceph-mon[33593]: 8.c scrub ok 2024-03-24T12:02:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:00 smithi193 ceph-mon[33593]: 8.e scrub starts 2024-03-24T12:02:00.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:00.682 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:00.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:00.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:00.820 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:00.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:00.922 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:00.958 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:00.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:00.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:01.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:01.107 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:01.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:01.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:01.249 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:01.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:01.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:01.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:01 smithi064 ceph-mon[30223]: pgmap v431: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/95097 objects misplaced (2.130%) 2024-03-24T12:02:01.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:01 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T12:02:01.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:01 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T12:02:01.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:01 smithi064 ceph-mon[30223]: 8.e scrub ok 2024-03-24T12:02:01.385 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:01 smithi064 ceph-mon[42243]: pgmap v431: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/95097 objects misplaced (2.130%) 2024-03-24T12:02:01.385 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:01 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T12:02:01.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:01 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T12:02:01.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:01 smithi064 ceph-mon[42243]: 8.e scrub ok 2024-03-24T12:02:01.390 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:01.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:01.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:01.530 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:01.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:01 smithi193 ceph-mon[33593]: pgmap v431: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/95097 objects misplaced (2.130%) 2024-03-24T12:02:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:01 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T12:02:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:01 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T12:02:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:01 smithi193 ceph-mon[33593]: 8.e scrub ok 2024-03-24T12:02:01.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:01.669 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:01.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:01.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:01.806 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:01.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:01.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:01.960 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:01.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:02.061 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:02.091 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-24T12:02:02.091 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-03-24T12:02:02.091 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-03-24T12:02:02.091 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-03-24T12:02:02.092 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-03-24T12:02:02.092 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-03-24T12:02:02.092 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-03-24T12:02:02.092 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:02:02.092 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-03-24T12:02:02.092 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-03-24T12:02:02.092 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-03-24T12:02:02.092 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-24T12:02:02.106 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:02.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:02.208 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:02.244 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:02.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:02.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:02.387 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:02.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:02.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:02.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:02.631 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:02.663 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:02.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:02.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:02.821 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:02.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:02.883 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: pgmap v432: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/90498 objects misplaced (2.239%) 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: 6.a scrub starts 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: 6.a scrub ok 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: 4.19 scrub starts 2024-03-24T12:02:02.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[30223]: 4.19 scrub ok 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: pgmap v432: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/90498 objects misplaced (2.239%) 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: 6.a scrub starts 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: 6.a scrub ok 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: 4.19 scrub starts 2024-03-24T12:02:02.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:02 smithi064 ceph-mon[42243]: 4.19 scrub ok 2024-03-24T12:02:02.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:02.925 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:02:02.956 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:02.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:02.966 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:02:03.004 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:02:03.042 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:02:03.058 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:03.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:02:03.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: pgmap v432: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2026/90498 objects misplaced (2.239%) 2024-03-24T12:02:03.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: 6.a scrub starts 2024-03-24T12:02:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: 6.a scrub ok 2024-03-24T12:02:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T12:02:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T12:02:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:02:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:02:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: 4.19 scrub starts 2024-03-24T12:02:03.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:02 smithi193 ceph-mon[33593]: 4.19 scrub ok 2024-03-24T12:02:03.090 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:03.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:03.122 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:02:03.163 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:02:03.168 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:02:03.168 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:02:03.168 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:02:03.168 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:02:03.168 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:02:03.168 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:02:03.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:02:03.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:02:03.192 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:03.228 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:03.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:03.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:03.365 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:03.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:03.467 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:03.507 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:03.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:03.609 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:03.643 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:03.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:03.745 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:03.780 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:03.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:02:03.797 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T12:02:03.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:03.924 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:03.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:04.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:04.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:03 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/676354726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:04.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:03 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/676354726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:04.061 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:04.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:04.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:03 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/676354726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:04.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:04.202 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:04.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:04.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:04.339 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:04.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:04.392 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:04.392 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:04.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:04.478 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:04.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:04.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:04.620 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:04.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:04.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:04.757 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:04.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:04.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:04.889 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:04.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:04.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:05.038 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:05.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:05.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:04 smithi064 ceph-mon[30223]: pgmap v433: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 1630/76509 objects misplaced (2.130%) 2024-03-24T12:02:05.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:04 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T12:02:05.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:04 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T12:02:05.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:04 smithi064 ceph-mon[42243]: pgmap v433: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 1630/76509 objects misplaced (2.130%) 2024-03-24T12:02:05.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:04 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T12:02:05.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:04 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T12:02:05.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:04 smithi193 ceph-mon[33593]: pgmap v433: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 1630/76509 objects misplaced (2.130%) 2024-03-24T12:02:05.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:04 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T12:02:05.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:04 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T12:02:05.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:05.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:05.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:05.319 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:05.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:05.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:05.459 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:05.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:05.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:05.597 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:05.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:05.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:05.735 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:05.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:05.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:05.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:05.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:06.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:06.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 6.10 scrub starts 2024-03-24T12:02:06.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 6.10 scrub ok 2024-03-24T12:02:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 4.4 scrub starts 2024-03-24T12:02:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 4.4 scrub ok 2024-03-24T12:02:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 4.c scrub starts 2024-03-24T12:02:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 4.c scrub ok 2024-03-24T12:02:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T12:02:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T12:02:06.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:05 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T12:02:06.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:06.163 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:06.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:06.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:06.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 6.10 scrub starts 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 6.10 scrub ok 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 4.4 scrub starts 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 4.4 scrub ok 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 4.c scrub starts 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 4.c scrub ok 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 6.10 scrub starts 2024-03-24T12:02:06.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 6.10 scrub ok 2024-03-24T12:02:06.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 4.4 scrub starts 2024-03-24T12:02:06.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 4.4 scrub ok 2024-03-24T12:02:06.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 4.c scrub starts 2024-03-24T12:02:06.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 4.c scrub ok 2024-03-24T12:02:06.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T12:02:06.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T12:02:06.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:05 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T12:02:06.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:06.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:06.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:06.452 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:06.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:06.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:06.597 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:06.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:06.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:06.741 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:06.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:06.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:06.884 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:06.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:06.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:02:06.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:07.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[30223]: pgmap v434: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1002 B/s rd, 0 B/s wr, 1.25k op/s; 1630/76509 objects misplaced (2.130%) 2024-03-24T12:02:07.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T12:02:07.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T12:02:07.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T12:02:07.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T12:02:07.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T12:02:07.016 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[42243]: pgmap v434: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1002 B/s rd, 0 B/s wr, 1.25k op/s; 1630/76509 objects misplaced (2.130%) 2024-03-24T12:02:07.016 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T12:02:07.016 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T12:02:07.016 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T12:02:07.016 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T12:02:07.016 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:06 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T12:02:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:07.021 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:07.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:07.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:06 smithi193 ceph-mon[33593]: pgmap v434: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1002 B/s rd, 0 B/s wr, 1.25k op/s; 1630/76509 objects misplaced (2.130%) 2024-03-24T12:02:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:06 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T12:02:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:06 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T12:02:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:06 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T12:02:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:06 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T12:02:07.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:06 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T12:02:07.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:07.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:07.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:07.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:07.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:07.427 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:07.463 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:07.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:07.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:07.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:02:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:07.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:07.641 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:02:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:07.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:07.738 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:07.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:07.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:07.875 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:07.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:02:07.968 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:02:07.969 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:02:07.969 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:02:07.969 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:02:07.969 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T12:02:07.969 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T12:02:07.969 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T12:02:07.969 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:02:07.969 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:07 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:02:07.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:08.011 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:08.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:08.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:08.148 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:08.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:08.248 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:08.284 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:08.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:02:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:02:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:02:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:02:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T12:02:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T12:02:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T12:02:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:02:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:07 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:02:08.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:08.387 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:02:08.387 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:02:08.444 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:08.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:08.546 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:08.586 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:08.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:08.688 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:08.725 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:08.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:08.798 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:02:08.799 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:02:08.799 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-24T12:02:08.799 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-24T12:02:08.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:02:08.802 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:08.826 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:08.858 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:08.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:08.960 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:08.994 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:08.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:09.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:09.140 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:09.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:09.240 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:09.286 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:09.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:09.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: pgmap v436: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1242/57336 objects misplaced (2.166%) 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 8.d scrub starts 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 5.c scrub starts 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 5.c scrub ok 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 2.a scrub starts 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[30223]: 2.a scrub ok 2024-03-24T12:02:09.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: pgmap v436: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1242/57336 objects misplaced (2.166%) 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 8.d scrub starts 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 5.c scrub starts 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 5.c scrub ok 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 2.a scrub starts 2024-03-24T12:02:09.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:08 smithi064 ceph-mon[42243]: 2.a scrub ok 2024-03-24T12:02:09.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: pgmap v436: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1242/57336 objects misplaced (2.166%) 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 8.d scrub starts 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 5.c scrub starts 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 5.c scrub ok 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 2.a scrub starts 2024-03-24T12:02:09.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:08 smithi193 ceph-mon[33593]: 2.a scrub ok 2024-03-24T12:02:09.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:09.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:02:09.449 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:09.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:09.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:09.593 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:09.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:09.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:09.729 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:09.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:09.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:09.863 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:09.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:09.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:10.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:10.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:10.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:10.141 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:10.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:10.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:10.269 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:10.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[30223]: 8.d scrub ok 2024-03-24T12:02:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:02:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:02:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[30223]: 6.17 scrub starts 2024-03-24T12:02:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[30223]: 6.17 scrub ok 2024-03-24T12:02:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T12:02:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T12:02:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[42243]: 8.d scrub ok 2024-03-24T12:02:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:02:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:02:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[42243]: 6.17 scrub starts 2024-03-24T12:02:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[42243]: 6.17 scrub ok 2024-03-24T12:02:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T12:02:10.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:09 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T12:02:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:09 smithi193 ceph-mon[33593]: 8.d scrub ok 2024-03-24T12:02:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:09 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:02:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:09 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:02:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:09 smithi193 ceph-mon[33593]: 6.17 scrub starts 2024-03-24T12:02:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:09 smithi193 ceph-mon[33593]: 6.17 scrub ok 2024-03-24T12:02:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:09 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T12:02:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:09 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T12:02:10.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:10.418 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:10.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:10.519 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:10.553 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:10.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:10.655 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:10.692 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:10.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:10.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:10.832 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:10.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:10.873 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:02:10.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:10.985 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:10.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: pgmap v437: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1242/57336 objects misplaced (2.166%) 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 6.d scrub starts 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 6.d scrub ok 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T12:02:11.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1192892024' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[30223]: from='client.35225 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: pgmap v437: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1242/57336 objects misplaced (2.166%) 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 6.d scrub starts 2024-03-24T12:02:11.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T12:02:11.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T12:02:11.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 6.d scrub ok 2024-03-24T12:02:11.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T12:02:11.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T12:02:11.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T12:02:11.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T12:02:11.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1192892024' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:11.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:10 smithi064 ceph-mon[42243]: from='client.35225 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:11.088 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:11.122 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:11.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:11.223 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:11.255 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:11.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:11.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: pgmap v437: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1242/57336 objects misplaced (2.166%) 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 6.d scrub starts 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 6.d scrub ok 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1192892024' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:10 smithi193 ceph-mon[33593]: from='client.35225 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:11.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:11.395 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:11.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:11.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:11.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:11.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:11.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:11.684 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:11.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:11.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:11.823 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:11.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:11.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:11.965 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:11.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:12.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:12.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:12.208 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:12.232 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:12.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[30223]: from='client.35225 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[30223]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[42243]: from='client.35225 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:02:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[42243]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T12:02:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T12:02:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T12:02:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:12.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:11 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:02:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:11 smithi193 ceph-mon[33593]: from='client.35225 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:02:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:11 smithi193 ceph-mon[33593]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T12:02:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:11 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T12:02:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:11 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T12:02:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:12.371 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:12.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:12.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:12.509 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:12.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:12.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:12.647 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:12.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:12.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:12.782 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:12.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:12.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:12.929 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:12.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:13.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:12 smithi064 ceph-mon[30223]: pgmap v439: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 992 op/s; 1242/52686 objects misplaced (2.357%) 2024-03-24T12:02:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:12 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T12:02:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:12 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T12:02:13.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:12 smithi064 ceph-mon[42243]: pgmap v439: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 992 op/s; 1242/52686 objects misplaced (2.357%) 2024-03-24T12:02:13.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:12 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T12:02:13.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:12 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T12:02:13.069 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:13.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:13.171 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:13.206 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:13.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:13.227 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:13.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:12 smithi193 ceph-mon[33593]: pgmap v439: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 992 op/s; 1242/52686 objects misplaced (2.357%) 2024-03-24T12:02:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:12 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T12:02:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:12 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T12:02:13.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:13.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:13.464 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:13.499 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:13.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:13.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:13.660 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:13.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:13.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:13.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:13.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:13.933 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:13.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:14.034 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:14.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:13 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T12:02:14.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:13 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T12:02:14.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:13 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T12:02:14.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:13 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T12:02:14.070 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:14.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:14.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:14.213 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:14.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:14.315 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:14.342 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:13 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T12:02:14.342 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:13 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T12:02:14.349 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:14.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:14.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:14.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:14.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:14.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:14.629 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:14.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:14.730 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:14.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:02:14.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:14.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:14.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:14.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:14.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:15.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:15.051 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:15.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[30223]: pgmap v440: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 790 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 778/38514 objects misplaced (2.020%) 2024-03-24T12:02:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T12:02:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T12:02:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T12:02:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T12:02:15.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[30223]: 2.2 scrub starts 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[30223]: 2.2 scrub ok 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[42243]: pgmap v440: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 790 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 778/38514 objects misplaced (2.020%) 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[42243]: 2.2 scrub starts 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:14 smithi064 ceph-mon[42243]: 2.2 scrub ok 2024-03-24T12:02:15.153 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:15.188 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:15.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:15.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:15.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:15.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:14 smithi193 ceph-mon[33593]: pgmap v440: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 790 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 778/38514 objects misplaced (2.020%) 2024-03-24T12:02:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:14 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T12:02:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:14 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T12:02:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:14 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T12:02:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:14 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T12:02:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:14 smithi193 ceph-mon[33593]: 2.2 scrub starts 2024-03-24T12:02:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:14 smithi193 ceph-mon[33593]: 2.2 scrub ok 2024-03-24T12:02:15.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:15.473 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:15.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:15.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:15.620 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:15.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:15.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:15.768 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:15.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:15.775 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:02:15.813 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:02:15.851 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:02:15.869 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:15.899 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:15.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:15.900 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:02:15.943 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:02:15.982 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:02:16.001 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:16.028 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:02:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:16.033 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:16.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:16.067 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:02:16.067 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[30223]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[30223]: 4.6 deep-scrub starts 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3290916375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[30223]: 4.6 deep-scrub ok 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:02:16.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:02:16.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:02:16.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[42243]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T12:02:16.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[42243]: 4.6 deep-scrub starts 2024-03-24T12:02:16.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3290916375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:16.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:15 smithi064 ceph-mon[42243]: 4.6 deep-scrub ok 2024-03-24T12:02:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:02:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:02:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:02:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:02:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:02:16.072 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:02:16.072 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:02:16.072 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:02:16.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:16.160 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:16.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:16.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:16.294 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:16.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:15 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:02:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:15 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:02:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:15 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:02:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:15 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:02:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:15 smithi193 ceph-mon[33593]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T12:02:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:15 smithi193 ceph-mon[33593]: 4.6 deep-scrub starts 2024-03-24T12:02:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:15 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3290916375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:15 smithi193 ceph-mon[33593]: 4.6 deep-scrub ok 2024-03-24T12:02:16.395 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:16.427 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:16.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:16.529 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:16.556 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:16.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:16.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:16.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:16.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:16.835 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:16.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:16.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:16.993 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:16.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:17.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:17.131 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:17.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:17.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:16 smithi064 ceph-mon[30223]: pgmap v442: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 790 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 784 op/s; 778/38514 objects misplaced (2.020%) 2024-03-24T12:02:17.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:16 smithi064 ceph-mon[30223]: 4.1e scrub starts 2024-03-24T12:02:17.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:16 smithi064 ceph-mon[30223]: 4.1e scrub ok 2024-03-24T12:02:17.139 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:16 smithi064 ceph-mon[42243]: pgmap v442: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 790 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 784 op/s; 778/38514 objects misplaced (2.020%) 2024-03-24T12:02:17.140 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:16 smithi064 ceph-mon[42243]: 4.1e scrub starts 2024-03-24T12:02:17.140 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:16 smithi064 ceph-mon[42243]: 4.1e scrub ok 2024-03-24T12:02:17.232 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:17.264 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:17.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:17.285 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:16 smithi193 ceph-mon[33593]: pgmap v442: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 790 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 784 op/s; 778/38514 objects misplaced (2.020%) 2024-03-24T12:02:17.285 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:16 smithi193 ceph-mon[33593]: 4.1e scrub starts 2024-03-24T12:02:17.285 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:16 smithi193 ceph-mon[33593]: 4.1e scrub ok 2024-03-24T12:02:17.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:17.404 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:17.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:17.425 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:02:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:02:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:17.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:17.547 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:17.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:17.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:02:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:02:17.644 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:02:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-03-24T12:02:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-03-24T12:02:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-03-24T12:02:17.645 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:02:17.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:17.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:17.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:17.789 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:17.823 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:17.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:17.925 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:17.959 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:17.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:18.012 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:17 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T12:02:18.012 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T12:02:18.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:17 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T12:02:18.012 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T12:02:18.061 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:18.091 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:18.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:18.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:18.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:18.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:18.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: 8.b scrub starts 2024-03-24T12:02:18.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T12:02:18.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T12:02:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: 8.b scrub starts 2024-03-24T12:02:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T12:02:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T12:02:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:17 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T12:02:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:17 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T12:02:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:17 smithi193 ceph-mon[33593]: 8.b scrub starts 2024-03-24T12:02:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:17 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T12:02:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:17 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T12:02:18.340 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:18.371 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:18.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:18.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:18.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:18.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:18.652 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:18.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:18.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:18.793 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:18.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:18.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:18.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:18.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:19.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:02:19.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:19.117 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:19.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:19.218 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:19.264 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:19.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: pgmap v443: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 390 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 374/19314 objects misplaced (1.936%) 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: 8.b scrub ok 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: 6.1e deep-scrub starts 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[30223]: 6.1e deep-scrub ok 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: pgmap v443: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 390 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 374/19314 objects misplaced (1.936%) 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: 8.b scrub ok 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: 6.1e deep-scrub starts 2024-03-24T12:02:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:18 smithi064 ceph-mon[42243]: 6.1e deep-scrub ok 2024-03-24T12:02:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:18 smithi193 ceph-mon[33593]: pgmap v443: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 390 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 374/19314 objects misplaced (1.936%) 2024-03-24T12:02:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:18 smithi193 ceph-mon[33593]: 8.b scrub ok 2024-03-24T12:02:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:18 smithi193 ceph-mon[33593]: 6.1e deep-scrub starts 2024-03-24T12:02:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:18 smithi193 ceph-mon[33593]: 6.1e deep-scrub ok 2024-03-24T12:02:19.366 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:19.399 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:19.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:19.501 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:19.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:19.537 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:19.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:19.638 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:19.671 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:19.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:19.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:19.823 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:19.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:19.924 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:19.954 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:19.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:20.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:20.092 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:20.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:20.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:02:20.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:20.243 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:20.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:20.244 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:19 smithi064 ceph-mon[42243]: 8.f deep-scrub starts 2024-03-24T12:02:20.244 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:19 smithi064 ceph-mon[30223]: 8.f deep-scrub starts 2024-03-24T12:02:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:19 smithi193 ceph-mon[33593]: 8.f deep-scrub starts 2024-03-24T12:02:20.345 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:20.373 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:20.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:20.474 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:20.502 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:20.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:20.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:20.649 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:20.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:20.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:20.794 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:20.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:20.895 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:20.935 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:20.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:21.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:21.065 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:21.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:21.167 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:21.198 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:21.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:20 smithi193 ceph-mon[33593]: pgmap v444: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 390 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 374/19314 objects misplaced (1.936%) 2024-03-24T12:02:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:20 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T12:02:21.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:20 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T12:02:21.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:20 smithi064 ceph-mon[30223]: pgmap v444: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 390 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 374/19314 objects misplaced (1.936%) 2024-03-24T12:02:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:20 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T12:02:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:20 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T12:02:21.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:20 smithi064 ceph-mon[42243]: pgmap v444: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 390 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 374/19314 objects misplaced (1.936%) 2024-03-24T12:02:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:20 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T12:02:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:20 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T12:02:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:21.343 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:21.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:21.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:21.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:21.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:21.584 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:21.615 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:21.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:21.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:21.740 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:21.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:21.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:21.899 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:21.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:22.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:22.042 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:22.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:22.143 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 8.f deep-scrub ok 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T12:02:22.168 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: 8.f deep-scrub ok 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T12:02:22.169 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T12:02:22.175 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:22.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:22.185 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Removed 122023 objects 2024-03-24T12:02:22.185 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Clean up completed and total clean up time :111.348 2024-03-24T12:02:22.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:22.291 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:22.301 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:22.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 8.f deep-scrub ok 2024-03-24T12:02:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:02:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:02:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:02:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:02:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T12:02:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T12:02:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T12:02:22.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T12:02:22.404 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:22.429 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:22.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:22.446 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:02:22.449 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:02:22.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:22.555 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:22.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:22.656 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:22.681 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:22.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:22.782 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:22.806 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:22.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:22.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:22.908 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:22.933 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:22.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:23.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:23.059 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:23.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:23.161 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:23.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:23.185 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:23.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:23.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: pgmap v445: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 301 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 374/15015 objects misplaced (2.491%) 2024-03-24T12:02:23.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T12:02:23.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T12:02:23.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:22 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T12:02:23.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:23 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T12:02:23.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:23 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T12:02:23.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:23 smithi064 ceph-mon[42243]: pgmap v445: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 301 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 374/15015 objects misplaced (2.491%) 2024-03-24T12:02:23.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:23 smithi064 ceph-mon[42243]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T12:02:23.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:23 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T12:02:23.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:23 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T12:02:23.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:23 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T12:02:23.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:23 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T12:02:23.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:23.311 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:23.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: pgmap v445: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 301 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 374/15015 objects misplaced (2.491%) 2024-03-24T12:02:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T12:02:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T12:02:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T12:02:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T12:02:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:22 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T12:02:23.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:23.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:23.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:23.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:23.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:23.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:23.665 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:23.689 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:23.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:23.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:23.816 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:23.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:23.917 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:23.941 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:23.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:24.042 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:24.067 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:24.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:24.168 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:24.192 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:24.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:24.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:24.269 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: from='client.15399 v1:172.21.15.64:0/96528811' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:02:24.269 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:02:24.269 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:02:24.269 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:02:24.269 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: 6.b deep-scrub starts 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[42243]: 6.b deep-scrub ok 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: from='client.15399 v1:172.21.15.64:0/96528811' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:02:24.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:02:24.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T12:02:24.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T12:02:24.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T12:02:24.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: 6.b deep-scrub starts 2024-03-24T12:02:24.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:24 smithi064 ceph-mon[30223]: 6.b deep-scrub ok 2024-03-24T12:02:24.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:24.317 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:24.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: from='client.15399 v1:172.21.15.64:0/96528811' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:02:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:02:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:02:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:02:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:02:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T12:02:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T12:02:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T12:02:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: 6.b deep-scrub starts 2024-03-24T12:02:24.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:24 smithi193 ceph-mon[33593]: 6.b deep-scrub ok 2024-03-24T12:02:24.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:24.443 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:24.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:24.545 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:24.569 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:24.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:24.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:24.695 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:24.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:24.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:24.822 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:24.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:24.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:24.948 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:24.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:25.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:25.075 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:25.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:25.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:25.203 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:25.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:25.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[42243]: pgmap v448: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 34 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[42243]: 3.2 scrub starts 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[42243]: 3.2 scrub ok 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[30223]: pgmap v448: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 34 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T12:02:25.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T12:02:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[30223]: 3.2 scrub starts 2024-03-24T12:02:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:25 smithi064 ceph-mon[30223]: 3.2 scrub ok 2024-03-24T12:02:25.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:25.330 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:25.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:25 smithi193 ceph-mon[33593]: pgmap v448: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 34 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:02:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:25 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T12:02:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:25 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T12:02:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:25 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T12:02:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:25 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T12:02:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:25 smithi193 ceph-mon[33593]: 3.2 scrub starts 2024-03-24T12:02:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:25 smithi193 ceph-mon[33593]: 3.2 scrub ok 2024-03-24T12:02:25.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:25.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:25.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:25.582 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:25.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:25.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:25.713 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:25.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:25.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:25.838 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:25.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:25.939 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:25.964 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:25.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:26.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:26.088 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:26.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[30223]: 6.1d scrub starts 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[30223]: 6.1d scrub ok 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:02:26.158 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T12:02:26.159 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T12:02:26.159 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T12:02:26.159 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T12:02:26.159 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[42243]: 6.1d scrub starts 2024-03-24T12:02:26.159 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:26 smithi064 ceph-mon[42243]: 6.1d scrub ok 2024-03-24T12:02:26.181 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-03-24T12:02:22.000052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":199301494,"num_objects":3231,"num_object_clones":0,"num_object_copies":9693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":904,"num_read_kb":656,"num_write":144474,"num_write_kb":3180750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":66732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128152,"ondisk_log_size":128152,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":512,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3722760,"kb_used_data":749448,"kb_used_omap":12,"kb_used_meta":2973171,"kb_avail":746238456,"statfs":{"total":767960285184,"available":764148178944,"internally_reserved":0,"allocated":767434752,"data_stored":764233810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3044527664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,2,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-629571584,"num_objects":-9607,"num_object_clones":0,"num_object_copies":-28821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-778,"num_objects_unfound":0,"num_objects_dirty":-9607,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9608,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.682604"},"pg_stats":[{"pgid":"5.9","version":"176'9","reported_seq":523,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:15.318404+0000","last_change":"2024-03-24T12:01:58.038265+0000","last_active":"2024-03-24T12:02:15.318404+0000","last_peered":"2024-03-24T12:02:15.318404+0000","last_clean":"2024-03-24T12:02:15.318404+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:02:15.318404+0000","last_undegraded":"2024-03-24T12:02:15.318404+0000","last_fullsized":"2024-03-24T12:02:15.318404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:01:58.038146+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T11:59:08.941456+0000","last_clean_scrub_stamp":"2024-03-24T12:01:58.038146+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:25.343573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":412,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935037+0000","last_change":"2024-03-24T11:59:59.960767+0000","last_active":"2024-03-24T12:02:14.935037+0000","last_peered":"2024-03-24T12:02:14.935037+0000","last_clean":"2024-03-24T12:02:14.935037+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T12:02:14.935037+0000","last_undegraded":"2024-03-24T12:02:14.935037+0000","last_fullsized":"2024-03-24T12:02:14.935037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:59:59.957817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:59.957817+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":401,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.947216+0000","last_change":"2024-03-24T12:00:08.247939+0000","last_active":"2024-03-24T12:02:14.947216+0000","last_peered":"2024-03-24T12:02:14.947216+0000","last_clean":"2024-03-24T12:02:14.947216+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T12:02:14.947216+0000","last_undegraded":"2024-03-24T12:02:14.947216+0000","last_fullsized":"2024-03-24T12:02:14.947216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:08.247807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:00:08.247807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":379,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932883+0000","last_change":"2024-03-24T12:02:01.760487+0000","last_active":"2024-03-24T12:02:14.932883+0000","last_peered":"2024-03-24T12:02:14.932883+0000","last_clean":"2024-03-24T12:02:14.932883+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T12:02:14.932883+0000","last_undegraded":"2024-03-24T12:02:14.932883+0000","last_fullsized":"2024-03-24T12:02:14.932883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:01.760400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:02:01.760400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:02.738660+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"240'10553","reported_seq":10748,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.353516+0000","last_change":"2024-03-24T12:01:06.976154+0000","last_active":"2024-03-24T12:02:21.353516+0000","last_peered":"2024-03-24T12:02:21.353516+0000","last_clean":"2024-03-24T12:02:21.353516+0000","last_became_active":"2024-03-24T12:00:11.240925+0000","last_became_peered":"2024-03-24T12:00:11.240925+0000","last_unstale":"2024-03-24T12:02:21.353516+0000","last_undegraded":"2024-03-24T12:02:21.353516+0000","last_fullsized":"2024-03-24T12:02:21.353516+0000","mapping_epoch":170,"log_start":"194'6007","ondisk_log_start":"194'6007","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'7978","last_scrub_stamp":"2024-03-24T12:01:06.976024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:06.976024+0000","objects_scrubbed":2605,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2581,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":404,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936302+0000","last_change":"2024-03-24T12:02:09.783798+0000","last_active":"2024-03-24T12:02:14.936302+0000","last_peered":"2024-03-24T12:02:14.936302+0000","last_clean":"2024-03-24T12:02:14.936302+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:02:14.936302+0000","last_undegraded":"2024-03-24T12:02:14.936302+0000","last_fullsized":"2024-03-24T12:02:14.936302+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:02:09.783600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.783600+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:26.180718+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.934481+0000","last_change":"2024-03-24T12:01:19.951504+0000","last_active":"2024-03-24T12:02:14.934481+0000","last_peered":"2024-03-24T12:02:14.934481+0000","last_clean":"2024-03-24T12:02:14.934481+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:02:14.934481+0000","last_undegraded":"2024-03-24T12:02:14.934481+0000","last_fullsized":"2024-03-24T12:02:14.934481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:19.951388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.951388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:34.918469+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":393,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.942394+0000","last_change":"2024-03-24T11:59:53.094141+0000","last_active":"2024-03-24T12:02:14.942394+0000","last_peered":"2024-03-24T12:02:14.942394+0000","last_clean":"2024-03-24T12:02:14.942394+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T12:02:14.942394+0000","last_undegraded":"2024-03-24T12:02:14.942394+0000","last_fullsized":"2024-03-24T12:02:14.942394+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:53.093972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:53.093972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":372,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.931507+0000","last_change":"2024-03-24T11:59:59.524343+0000","last_active":"2024-03-24T12:02:14.931507+0000","last_peered":"2024-03-24T12:02:14.931507+0000","last_clean":"2024-03-24T12:02:14.931507+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T12:02:14.931507+0000","last_undegraded":"2024-03-24T12:02:14.931507+0000","last_fullsized":"2024-03-24T12:02:14.931507+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:59.523952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:59.523952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"240'10635","reported_seq":10887,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.245547+0000","last_change":"2024-03-24T12:01:15.718722+0000","last_active":"2024-03-24T12:02:17.245547+0000","last_peered":"2024-03-24T12:02:17.245547+0000","last_clean":"2024-03-24T12:02:17.245547+0000","last_became_active":"2024-03-24T12:00:11.259187+0000","last_became_peered":"2024-03-24T12:00:11.259187+0000","last_unstale":"2024-03-24T12:02:17.245547+0000","last_undegraded":"2024-03-24T12:02:17.245547+0000","last_fullsized":"2024-03-24T12:02:17.245547+0000","mapping_epoch":178,"log_start":"194'6315","ondisk_log_start":"194'6315","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'8317","last_scrub_stamp":"2024-03-24T12:01:15.718293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:15.718293+0000","objects_scrubbed":2497,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:40.931397+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":425,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.926869+0000","last_change":"2024-03-24T12:02:05.124570+0000","last_active":"2024-03-24T12:02:14.926869+0000","last_peered":"2024-03-24T12:02:14.926869+0000","last_clean":"2024-03-24T12:02:14.926869+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:02:14.926869+0000","last_undegraded":"2024-03-24T12:02:14.926869+0000","last_fullsized":"2024-03-24T12:02:14.926869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:02:05.124435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:05.124435+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:12.403233+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":401,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.944774+0000","last_change":"2024-03-24T12:00:02.219859+0000","last_active":"2024-03-24T12:02:14.944774+0000","last_peered":"2024-03-24T12:02:14.944774+0000","last_clean":"2024-03-24T12:02:14.944774+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T12:02:14.944774+0000","last_undegraded":"2024-03-24T12:02:14.944774+0000","last_fullsized":"2024-03-24T12:02:14.944774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:02.219529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:00:02.219529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":393,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.942009+0000","last_change":"2024-03-24T12:00:09.213447+0000","last_active":"2024-03-24T12:02:14.942009+0000","last_peered":"2024-03-24T12:02:14.942009+0000","last_clean":"2024-03-24T12:02:14.942009+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T12:02:14.942009+0000","last_undegraded":"2024-03-24T12:02:14.942009+0000","last_fullsized":"2024-03-24T12:02:14.942009+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:09.213304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:00:09.213304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.929962+0000","last_change":"2024-03-24T12:02:12.944517+0000","last_active":"2024-03-24T12:02:14.929962+0000","last_peered":"2024-03-24T12:02:14.929962+0000","last_clean":"2024-03-24T12:02:14.929962+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T12:02:14.929962+0000","last_undegraded":"2024-03-24T12:02:14.929962+0000","last_fullsized":"2024-03-24T12:02:14.929962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:12.944172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:02:12.944172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:16.306971+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"240'10489","reported_seq":10683,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.052465+0000","last_change":"2024-03-24T12:01:17.564112+0000","last_active":"2024-03-24T12:02:17.052465+0000","last_peered":"2024-03-24T12:02:17.052465+0000","last_clean":"2024-03-24T12:02:17.052465+0000","last_became_active":"2024-03-24T12:00:11.260765+0000","last_became_peered":"2024-03-24T12:00:11.260765+0000","last_unstale":"2024-03-24T12:02:17.052465+0000","last_undegraded":"2024-03-24T12:02:17.052465+0000","last_fullsized":"2024-03-24T12:02:17.052465+0000","mapping_epoch":170,"log_start":"182'4461","ondisk_log_start":"182'4461","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'8266","last_scrub_stamp":"2024-03-24T12:01:17.563832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:17.563832+0000","objects_scrubbed":2450,"log_size":6028,"log_dups_size":0,"ondisk_log_size":6028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:23.495532+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"116'25","reported_seq":431,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936673+0000","last_change":"2024-03-24T12:01:21.467893+0000","last_active":"2024-03-24T12:02:14.936673+0000","last_peered":"2024-03-24T12:02:14.936673+0000","last_clean":"2024-03-24T12:02:14.936673+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T12:02:14.936673+0000","last_undegraded":"2024-03-24T12:02:14.936673+0000","last_fullsized":"2024-03-24T12:02:14.936673+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'25","last_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:01:21.467704+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:27.260112+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":391,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.941784+0000","last_change":"2024-03-24T12:02:11.033710+0000","last_active":"2024-03-24T12:02:14.941784+0000","last_peered":"2024-03-24T12:02:14.941784+0000","last_clean":"2024-03-24T12:02:14.941784+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T12:02:14.941784+0000","last_undegraded":"2024-03-24T12:02:14.941784+0000","last_fullsized":"2024-03-24T12:02:14.941784+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:02:11.033556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:11.033556+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:12.977384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":144,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.937152+0000","last_change":"2024-03-24T12:02:07.910245+0000","last_active":"2024-03-24T12:02:14.937152+0000","last_peered":"2024-03-24T12:02:14.937152+0000","last_clean":"2024-03-24T12:02:14.937152+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:02:14.937152+0000","last_undegraded":"2024-03-24T12:02:14.937152+0000","last_fullsized":"2024-03-24T12:02:14.937152+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:07.910126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:07.910126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:20.578073+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":389,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.947136+0000","last_change":"2024-03-24T12:02:09.089126+0000","last_active":"2024-03-24T12:02:14.947136+0000","last_peered":"2024-03-24T12:02:14.947136+0000","last_clean":"2024-03-24T12:02:14.947136+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T12:02:14.947136+0000","last_undegraded":"2024-03-24T12:02:14.947136+0000","last_fullsized":"2024-03-24T12:02:14.947136+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:09.088728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.088728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:09.196518+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"240'10978","reported_seq":11196,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.034188+0000","last_change":"2024-03-24T12:01:18.576696+0000","last_active":"2024-03-24T12:02:17.034188+0000","last_peered":"2024-03-24T12:02:17.034188+0000","last_clean":"2024-03-24T12:02:17.034188+0000","last_became_active":"2024-03-24T12:00:11.253227+0000","last_became_peered":"2024-03-24T12:00:11.253227+0000","last_unstale":"2024-03-24T12:02:17.034188+0000","last_undegraded":"2024-03-24T12:02:17.034188+0000","last_fullsized":"2024-03-24T12:02:17.034188+0000","mapping_epoch":178,"log_start":"188'5236","ondisk_log_start":"188'5236","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'8736","last_scrub_stamp":"2024-03-24T12:01:18.576312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:18.576312+0000","objects_scrubbed":2436,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:36.290187+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11993088,"num_objects":183,"num_object_clones":0,"num_object_copies":549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2253,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":419,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935353+0000","last_change":"2024-03-24T11:59:56.028640+0000","last_active":"2024-03-24T12:02:14.935353+0000","last_peered":"2024-03-24T12:02:14.935353+0000","last_clean":"2024-03-24T12:02:14.935353+0000","last_became_active":"2024-03-24T11:59:53.779445+0000","last_became_peered":"2024-03-24T11:59:53.779445+0000","last_unstale":"2024-03-24T12:02:14.935353+0000","last_undegraded":"2024-03-24T12:02:14.935353+0000","last_fullsized":"2024-03-24T12:02:14.935353+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:56.028559+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:56.028559+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"130'15","reported_seq":415,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.937203+0000","last_change":"2024-03-24T12:02:04.883385+0000","last_active":"2024-03-24T12:02:14.937203+0000","last_peered":"2024-03-24T12:02:14.937203+0000","last_clean":"2024-03-24T12:02:14.937203+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:02:14.937203+0000","last_undegraded":"2024-03-24T12:02:14.937203+0000","last_fullsized":"2024-03-24T12:02:14.937203+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'15","last_scrub_stamp":"2024-03-24T12:02:04.883290+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:02:04.883290+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:31.402251+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":143,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935509+0000","last_change":"2024-03-24T12:02:08.418375+0000","last_active":"2024-03-24T12:02:14.935509+0000","last_peered":"2024-03-24T12:02:14.935509+0000","last_clean":"2024-03-24T12:02:14.935509+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:02:14.935509+0000","last_undegraded":"2024-03-24T12:02:14.935509+0000","last_fullsized":"2024-03-24T12:02:14.935509+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:08.418230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.418230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:11.632607+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":523,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:15.313635+0000","last_change":"2024-03-24T11:59:54.146579+0000","last_active":"2024-03-24T12:02:15.313635+0000","last_peered":"2024-03-24T12:02:15.313635+0000","last_clean":"2024-03-24T12:02:15.313635+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T12:02:15.313635+0000","last_undegraded":"2024-03-24T12:02:15.313635+0000","last_fullsized":"2024-03-24T12:02:15.313635+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:54.146359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.146359+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936679+0000","last_change":"2024-03-24T12:00:01.672184+0000","last_active":"2024-03-24T12:02:14.936679+0000","last_peered":"2024-03-24T12:02:14.936679+0000","last_clean":"2024-03-24T12:02:14.936679+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T12:02:14.936679+0000","last_undegraded":"2024-03-24T12:02:14.936679+0000","last_fullsized":"2024-03-24T12:02:14.936679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:00:01.672059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:00:01.672059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"240'10336","reported_seq":10527,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.275795+0000","last_change":"2024-03-24T12:01:16.649895+0000","last_active":"2024-03-24T12:02:17.275795+0000","last_peered":"2024-03-24T12:02:17.275795+0000","last_clean":"2024-03-24T12:02:17.275795+0000","last_became_active":"2024-03-24T12:00:11.264306+0000","last_became_peered":"2024-03-24T12:00:11.264306+0000","last_unstale":"2024-03-24T12:02:17.275795+0000","last_undegraded":"2024-03-24T12:02:17.275795+0000","last_fullsized":"2024-03-24T12:02:17.275795+0000","mapping_epoch":170,"log_start":"194'5888","ondisk_log_start":"194'5888","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'8090","last_scrub_stamp":"2024-03-24T12:01:16.649747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:16.649747+0000","objects_scrubbed":2406,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:44.248061+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":664,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.931165+0000","last_change":"2024-03-24T11:59:54.144193+0000","last_active":"2024-03-24T12:02:14.931165+0000","last_peered":"2024-03-24T12:02:14.931165+0000","last_clean":"2024-03-24T12:02:14.931165+0000","last_became_active":"2024-03-24T11:59:53.771452+0000","last_became_peered":"2024-03-24T11:59:53.771452+0000","last_unstale":"2024-03-24T12:02:14.931165+0000","last_undegraded":"2024-03-24T12:02:14.931165+0000","last_fullsized":"2024-03-24T12:02:14.931165+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.144068+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.144068+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"130'22","reported_seq":428,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935057+0000","last_change":"2024-03-24T12:02:06.275179+0000","last_active":"2024-03-24T12:02:14.935057+0000","last_peered":"2024-03-24T12:02:14.935057+0000","last_clean":"2024-03-24T12:02:14.935057+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:02:14.935057+0000","last_undegraded":"2024-03-24T12:02:14.935057+0000","last_fullsized":"2024-03-24T12:02:14.935057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'22","last_scrub_stamp":"2024-03-24T12:02:06.275037+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:02:06.275037+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:28.707471+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935805+0000","last_change":"2024-03-24T12:02:09.327241+0000","last_active":"2024-03-24T12:02:14.935805+0000","last_peered":"2024-03-24T12:02:14.935805+0000","last_clean":"2024-03-24T12:02:14.935805+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:02:14.935805+0000","last_undegraded":"2024-03-24T12:02:14.935805+0000","last_fullsized":"2024-03-24T12:02:14.935805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:09.327128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.327128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:10.522890+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":524,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:15.314994+0000","last_change":"2024-03-24T12:02:09.938055+0000","last_active":"2024-03-24T12:02:15.314994+0000","last_peered":"2024-03-24T12:02:15.314994+0000","last_clean":"2024-03-24T12:02:15.314994+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T12:02:15.314994+0000","last_undegraded":"2024-03-24T12:02:15.314994+0000","last_fullsized":"2024-03-24T12:02:15.314994+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:02:09.937913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.937913+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:37.870968+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":387,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933895+0000","last_change":"2024-03-24T12:02:09.785009+0000","last_active":"2024-03-24T12:02:14.933895+0000","last_peered":"2024-03-24T12:02:14.933895+0000","last_clean":"2024-03-24T12:02:14.933895+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T12:02:14.933895+0000","last_undegraded":"2024-03-24T12:02:14.933895+0000","last_fullsized":"2024-03-24T12:02:14.933895+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:09.784916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.784916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:18.328425+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"240'10764","reported_seq":10965,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.358950+0000","last_change":"2024-03-24T12:01:08.025876+0000","last_active":"2024-03-24T12:02:21.358950+0000","last_peered":"2024-03-24T12:02:21.358950+0000","last_clean":"2024-03-24T12:02:21.358950+0000","last_became_active":"2024-03-24T12:00:11.235608+0000","last_became_peered":"2024-03-24T12:00:11.235608+0000","last_unstale":"2024-03-24T12:02:21.358950+0000","last_undegraded":"2024-03-24T12:02:21.358950+0000","last_fullsized":"2024-03-24T12:02:21.358950+0000","mapping_epoch":178,"log_start":"195'6358","ondisk_log_start":"195'6358","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'8137","last_scrub_stamp":"2024-03-24T12:01:08.025667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:08.025667+0000","objects_scrubbed":2680,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:30.885039+0000","scrub_duration":765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2640,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":419,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932840+0000","last_change":"2024-03-24T11:59:53.991924+0000","last_active":"2024-03-24T12:02:14.932840+0000","last_peered":"2024-03-24T12:02:14.932840+0000","last_clean":"2024-03-24T12:02:14.932840+0000","last_became_active":"2024-03-24T11:59:53.771384+0000","last_became_peered":"2024-03-24T11:59:53.771384+0000","last_unstale":"2024-03-24T12:02:14.932840+0000","last_undegraded":"2024-03-24T12:02:14.932840+0000","last_fullsized":"2024-03-24T12:02:14.932840+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:53.991713+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:53.991713+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"178'16","reported_seq":411,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936474+0000","last_change":"2024-03-24T11:59:18.653135+0000","last_active":"2024-03-24T12:02:14.936474+0000","last_peered":"2024-03-24T12:02:14.936474+0000","last_clean":"2024-03-24T12:02:14.936474+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:02:14.936474+0000","last_undegraded":"2024-03-24T12:02:14.936474+0000","last_fullsized":"2024-03-24T12:02:14.936474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'16","last_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T11:59:18.653047+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930218+0000","last_change":"2024-03-24T12:01:21.034042+0000","last_active":"2024-03-24T12:02:14.930218+0000","last_peered":"2024-03-24T12:02:14.930218+0000","last_clean":"2024-03-24T12:02:14.930218+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:02:14.930218+0000","last_undegraded":"2024-03-24T12:02:14.930218+0000","last_fullsized":"2024-03-24T12:02:14.930218+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:21.033920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:01:21.033920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:34.894640+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":383,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933544+0000","last_change":"2024-03-24T11:59:52.034599+0000","last_active":"2024-03-24T12:02:14.933544+0000","last_peered":"2024-03-24T12:02:14.933544+0000","last_clean":"2024-03-24T12:02:14.933544+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T12:02:14.933544+0000","last_undegraded":"2024-03-24T12:02:14.933544+0000","last_fullsized":"2024-03-24T12:02:14.933544+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:52.034485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:52.034485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":373,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.952413+0000","last_change":"2024-03-24T11:58:55.061715+0000","last_active":"2024-03-24T12:02:14.952413+0000","last_peered":"2024-03-24T12:02:14.952413+0000","last_clean":"2024-03-24T12:02:14.952413+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T12:02:14.952413+0000","last_undegraded":"2024-03-24T12:02:14.952413+0000","last_fullsized":"2024-03-24T12:02:14.952413+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:55.061633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:55.061633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"240'11038","reported_seq":11305,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.158251+0000","last_change":"2024-03-24T12:01:14.162112+0000","last_active":"2024-03-24T12:02:21.158251+0000","last_peered":"2024-03-24T12:02:21.158251+0000","last_clean":"2024-03-24T12:02:21.158251+0000","last_became_active":"2024-03-24T12:00:11.257691+0000","last_became_peered":"2024-03-24T12:00:11.257691+0000","last_unstale":"2024-03-24T12:02:21.158251+0000","last_undegraded":"2024-03-24T12:02:21.158251+0000","last_fullsized":"2024-03-24T12:02:21.158251+0000","mapping_epoch":170,"log_start":"194'6084","ondisk_log_start":"194'6084","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'8479","last_scrub_stamp":"2024-03-24T12:01:14.161954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:14.161954+0000","objects_scrubbed":2602,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2564,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":506,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936479+0000","last_change":"2024-03-24T11:59:00.089695+0000","last_active":"2024-03-24T12:02:14.936479+0000","last_peered":"2024-03-24T12:02:14.936479+0000","last_clean":"2024-03-24T12:02:14.936479+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:02:14.936479+0000","last_undegraded":"2024-03-24T12:02:14.936479+0000","last_fullsized":"2024-03-24T12:02:14.936479+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:00.089617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:00.089617+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"88'21","reported_seq":415,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935911+0000","last_change":"2024-03-24T11:59:05.708676+0000","last_active":"2024-03-24T12:02:14.935911+0000","last_peered":"2024-03-24T12:02:14.935911+0000","last_clean":"2024-03-24T12:02:14.935911+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T12:02:14.935911+0000","last_undegraded":"2024-03-24T12:02:14.935911+0000","last_fullsized":"2024-03-24T12:02:14.935911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'21","last_scrub_stamp":"2024-03-24T11:59:05.708532+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.708532+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.927498+0000","last_change":"2024-03-24T12:00:10.038752+0000","last_active":"2024-03-24T12:02:14.927498+0000","last_peered":"2024-03-24T12:02:14.927498+0000","last_clean":"2024-03-24T12:02:14.927498+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:02:14.927498+0000","last_undegraded":"2024-03-24T12:02:14.927498+0000","last_fullsized":"2024-03-24T12:02:14.927498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:10.038672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:00:10.038672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930554+0000","last_change":"2024-03-24T12:02:02.150576+0000","last_active":"2024-03-24T12:02:14.930554+0000","last_peered":"2024-03-24T12:02:14.930554+0000","last_clean":"2024-03-24T12:02:14.930554+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T12:02:14.930554+0000","last_undegraded":"2024-03-24T12:02:14.930554+0000","last_fullsized":"2024-03-24T12:02:14.930554+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:02.150459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:02.150459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:04.297232+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"240'15351","reported_seq":15526,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.339151+0000","last_change":"2024-03-24T12:02:08.168662+0000","last_active":"2024-03-24T12:02:21.339151+0000","last_peered":"2024-03-24T12:02:21.339151+0000","last_clean":"2024-03-24T12:02:21.339151+0000","last_became_active":"2024-03-24T11:58:42.970405+0000","last_became_peered":"2024-03-24T11:58:42.970405+0000","last_unstale":"2024-03-24T12:02:21.339151+0000","last_undegraded":"2024-03-24T12:02:21.339151+0000","last_fullsized":"2024-03-24T12:02:21.339151+0000","mapping_epoch":170,"log_start":"229'10921","ondisk_log_start":"229'10921","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'14267","last_scrub_stamp":"2024-03-24T12:02:08.168034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.168034+0000","objects_scrubbed":1166,"log_size":4430,"log_dups_size":0,"ondisk_log_size":4430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:37.772145+0000","scrub_duration":385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4063256,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15351,"num_write_kb":493185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":419,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.952621+0000","last_change":"2024-03-24T11:59:54.288619+0000","last_active":"2024-03-24T12:02:14.952621+0000","last_peered":"2024-03-24T12:02:14.952621+0000","last_clean":"2024-03-24T12:02:14.952621+0000","last_became_active":"2024-03-24T11:59:53.767670+0000","last_became_peered":"2024-03-24T11:59:53.767670+0000","last_unstale":"2024-03-24T12:02:14.952621+0000","last_undegraded":"2024-03-24T12:02:14.952621+0000","last_fullsized":"2024-03-24T12:02:14.952621+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.288488+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.288488+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":383,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936010+0000","last_change":"2024-03-24T11:59:24.679291+0000","last_active":"2024-03-24T12:02:14.936010+0000","last_peered":"2024-03-24T12:02:14.936010+0000","last_clean":"2024-03-24T12:02:14.936010+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:02:14.936010+0000","last_undegraded":"2024-03-24T12:02:14.936010+0000","last_fullsized":"2024-03-24T12:02:14.936010+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.679110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"195'26","reported_seq":429,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936334+0000","last_change":"2024-03-24T12:00:09.391645+0000","last_active":"2024-03-24T12:02:14.936334+0000","last_peered":"2024-03-24T12:02:14.936334+0000","last_clean":"2024-03-24T12:02:14.936334+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:02:14.936334+0000","last_undegraded":"2024-03-24T12:02:14.936334+0000","last_fullsized":"2024-03-24T12:02:14.936334+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'26","last_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_deep_scrub":"195'26","last_deep_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_clean_scrub_stamp":"2024-03-24T12:00:09.391234+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":151,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.226252+0000","last_change":"2024-03-24T12:02:17.226291+0000","last_active":"2024-03-24T12:02:17.226252+0000","last_peered":"2024-03-24T12:02:17.226252+0000","last_clean":"2024-03-24T12:02:17.226252+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:02:17.226252+0000","last_undegraded":"2024-03-24T12:02:17.226252+0000","last_fullsized":"2024-03-24T12:02:17.226252+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:02:17.226173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:17.226173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:45.403701+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":387,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936289+0000","last_change":"2024-03-24T11:59:55.784539+0000","last_active":"2024-03-24T12:02:14.936289+0000","last_peered":"2024-03-24T12:02:14.936289+0000","last_clean":"2024-03-24T12:02:14.936289+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T12:02:14.936289+0000","last_undegraded":"2024-03-24T12:02:14.936289+0000","last_fullsized":"2024-03-24T12:02:14.936289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:55.784390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:55.784390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"240'15043","reported_seq":15222,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.065569+0000","last_change":"2024-03-24T12:01:59.442315+0000","last_active":"2024-03-24T12:02:17.065569+0000","last_peered":"2024-03-24T12:02:17.065569+0000","last_clean":"2024-03-24T12:02:17.065569+0000","last_became_active":"2024-03-24T11:58:42.911298+0000","last_became_peered":"2024-03-24T11:58:42.911298+0000","last_unstale":"2024-03-24T12:02:17.065569+0000","last_undegraded":"2024-03-24T12:02:17.065569+0000","last_fullsized":"2024-03-24T12:02:17.065569+0000","mapping_epoch":170,"log_start":"219'9876","ondisk_log_start":"219'9876","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'13622","last_scrub_stamp":"2024-03-24T12:01:59.442010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:59.442010+0000","objects_scrubbed":1816,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:17.888494+0000","scrub_duration":728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15043,"num_write_kb":493184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":419,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936277+0000","last_change":"2024-03-24T11:59:54.754584+0000","last_active":"2024-03-24T12:02:14.936277+0000","last_peered":"2024-03-24T12:02:14.936277+0000","last_clean":"2024-03-24T12:02:14.936277+0000","last_became_active":"2024-03-24T11:59:53.763541+0000","last_became_peered":"2024-03-24T11:59:53.763541+0000","last_unstale":"2024-03-24T12:02:14.936277+0000","last_undegraded":"2024-03-24T12:02:14.936277+0000","last_fullsized":"2024-03-24T12:02:14.936277+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.754447+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.754447+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":390,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.926967+0000","last_change":"2024-03-24T12:02:08.167812+0000","last_active":"2024-03-24T12:02:14.926967+0000","last_peered":"2024-03-24T12:02:14.926967+0000","last_clean":"2024-03-24T12:02:14.926967+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T12:02:14.926967+0000","last_undegraded":"2024-03-24T12:02:14.926967+0000","last_fullsized":"2024-03-24T12:02:14.926967+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:08.167536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.167536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:26.877378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":416,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936115+0000","last_change":"2024-03-24T11:59:06.717918+0000","last_active":"2024-03-24T12:02:14.936115+0000","last_peered":"2024-03-24T12:02:14.936115+0000","last_clean":"2024-03-24T12:02:14.936115+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T12:02:14.936115+0000","last_undegraded":"2024-03-24T12:02:14.936115+0000","last_fullsized":"2024-03-24T12:02:14.936115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T11:59:06.717709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.717709+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930494+0000","last_change":"2024-03-24T12:00:02.041532+0000","last_active":"2024-03-24T12:02:14.930494+0000","last_peered":"2024-03-24T12:02:14.930494+0000","last_clean":"2024-03-24T12:02:14.930494+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:02:14.930494+0000","last_undegraded":"2024-03-24T12:02:14.930494+0000","last_fullsized":"2024-03-24T12:02:14.930494+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:00:02.041192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":135,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.927833+0000","last_change":"2024-03-24T11:59:19.096886+0000","last_active":"2024-03-24T12:02:14.927833+0000","last_peered":"2024-03-24T12:02:14.927833+0000","last_clean":"2024-03-24T12:02:14.927833+0000","last_became_active":"2024-03-24T11:59:19.096322+0000","last_became_peered":"2024-03-24T11:59:19.096322+0000","last_unstale":"2024-03-24T12:02:14.927833+0000","last_undegraded":"2024-03-24T12:02:14.927833+0000","last_fullsized":"2024-03-24T12:02:14.927833+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:56.287993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:58:56.287993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"240'15151","reported_seq":15308,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.280202+0000","last_change":"2024-03-24T12:00:11.231419+0000","last_active":"2024-03-24T12:02:17.280202+0000","last_peered":"2024-03-24T12:02:17.280202+0000","last_clean":"2024-03-24T12:02:17.280202+0000","last_became_active":"2024-03-24T11:58:42.970288+0000","last_became_peered":"2024-03-24T11:58:42.970288+0000","last_unstale":"2024-03-24T12:02:17.280202+0000","last_undegraded":"2024-03-24T12:02:17.280202+0000","last_fullsized":"2024-03-24T12:02:17.280202+0000","mapping_epoch":170,"log_start":"229'10902","ondisk_log_start":"229'10902","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15151,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":425,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.927125+0000","last_change":"2024-03-24T11:59:55.936621+0000","last_active":"2024-03-24T12:02:14.927125+0000","last_peered":"2024-03-24T12:02:14.927125+0000","last_clean":"2024-03-24T12:02:14.927125+0000","last_became_active":"2024-03-24T11:59:53.771448+0000","last_became_peered":"2024-03-24T11:59:53.771448+0000","last_unstale":"2024-03-24T12:02:14.927125+0000","last_undegraded":"2024-03-24T12:02:14.927125+0000","last_fullsized":"2024-03-24T12:02:14.927125+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:55.936510+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:55.936510+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":383,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933704+0000","last_change":"2024-03-24T11:59:48.136534+0000","last_active":"2024-03-24T12:02:14.933704+0000","last_peered":"2024-03-24T12:02:14.933704+0000","last_clean":"2024-03-24T12:02:14.933704+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T12:02:14.933704+0000","last_undegraded":"2024-03-24T12:02:14.933704+0000","last_fullsized":"2024-03-24T12:02:14.933704+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:48.136407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:48.136407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"240'18","reported_seq":412,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:15.739042+0000","last_change":"2024-03-24T12:02:15.739122+0000","last_active":"2024-03-24T12:02:15.739042+0000","last_peered":"2024-03-24T12:02:15.739042+0000","last_clean":"2024-03-24T12:02:15.739042+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T12:02:15.739042+0000","last_undegraded":"2024-03-24T12:02:15.739042+0000","last_fullsized":"2024-03-24T12:02:15.739042+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:02:15.738700+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:28.384111+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":228,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.931548+0000","last_change":"2024-03-24T11:59:20.570521+0000","last_active":"2024-03-24T12:02:14.931548+0000","last_peered":"2024-03-24T12:02:14.931548+0000","last_clean":"2024-03-24T12:02:14.931548+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:02:14.931548+0000","last_undegraded":"2024-03-24T12:02:14.931548+0000","last_fullsized":"2024-03-24T12:02:14.931548+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:20.570220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:20.570220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":391,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.931217+0000","last_change":"2024-03-24T11:59:21.570352+0000","last_active":"2024-03-24T12:02:14.931217+0000","last_peered":"2024-03-24T12:02:14.931217+0000","last_clean":"2024-03-24T12:02:14.931217+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:02:14.931217+0000","last_undegraded":"2024-03-24T12:02:14.931217+0000","last_fullsized":"2024-03-24T12:02:14.931217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.570115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.570115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933303+0000","last_change":"2024-03-24T12:01:20.958289+0000","last_active":"2024-03-24T12:02:14.933303+0000","last_peered":"2024-03-24T12:02:14.933303+0000","last_clean":"2024-03-24T12:02:14.933303+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T12:02:14.933303+0000","last_undegraded":"2024-03-24T12:02:14.933303+0000","last_fullsized":"2024-03-24T12:02:14.933303+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:20.958158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:01:20.958158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:46.260114+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.a","version":"240'15252","reported_seq":15437,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.910028+0000","last_change":"2024-03-24T12:01:58.734915+0000","last_active":"2024-03-24T12:02:21.910028+0000","last_peered":"2024-03-24T12:02:21.910028+0000","last_clean":"2024-03-24T12:02:21.910028+0000","last_became_active":"2024-03-24T11:59:19.098023+0000","last_became_peered":"2024-03-24T11:59:19.098023+0000","last_unstale":"2024-03-24T12:02:21.910028+0000","last_undegraded":"2024-03-24T12:02:21.910028+0000","last_fullsized":"2024-03-24T12:02:21.910028+0000","mapping_epoch":178,"log_start":"217'9745","ondisk_log_start":"217'9745","created":170,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'13459","last_scrub_stamp":"2024-03-24T12:01:58.734521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:58.734521+0000","objects_scrubbed":1835,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:22.994413+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15252,"num_write_kb":488704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":425,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.926983+0000","last_change":"2024-03-24T11:59:53.931656+0000","last_active":"2024-03-24T12:02:14.926983+0000","last_peered":"2024-03-24T12:02:14.926983+0000","last_clean":"2024-03-24T12:02:14.926983+0000","last_became_active":"2024-03-24T11:59:53.768899+0000","last_became_peered":"2024-03-24T11:59:53.768899+0000","last_unstale":"2024-03-24T12:02:14.926983+0000","last_undegraded":"2024-03-24T12:02:14.926983+0000","last_fullsized":"2024-03-24T12:02:14.926983+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:53.931538+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:53.931538+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":392,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932740+0000","last_change":"2024-03-24T12:02:06.076507+0000","last_active":"2024-03-24T12:02:14.932740+0000","last_peered":"2024-03-24T12:02:14.932740+0000","last_clean":"2024-03-24T12:02:14.932740+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T12:02:14.932740+0000","last_undegraded":"2024-03-24T12:02:14.932740+0000","last_fullsized":"2024-03-24T12:02:14.932740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:06.076284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:02:06.076284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:07.238413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":538,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:15.315246+0000","last_change":"2024-03-24T11:59:21.898682+0000","last_active":"2024-03-24T12:02:15.315246+0000","last_peered":"2024-03-24T12:02:15.315246+0000","last_clean":"2024-03-24T12:02:15.315246+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:02:15.315246+0000","last_undegraded":"2024-03-24T12:02:15.315246+0000","last_fullsized":"2024-03-24T12:02:15.315246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T11:59:21.898325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.898325+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":657,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930126+0000","last_change":"2024-03-24T11:59:50.052718+0000","last_active":"2024-03-24T12:02:14.930126+0000","last_peered":"2024-03-24T12:02:14.930126+0000","last_clean":"2024-03-24T12:02:14.930126+0000","last_became_active":"2024-03-24T11:59:49.085087+0000","last_became_peered":"2024-03-24T11:59:49.085087+0000","last_unstale":"2024-03-24T12:02:14.930126+0000","last_undegraded":"2024-03-24T12:02:14.930126+0000","last_fullsized":"2024-03-24T12:02:14.930126+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:50.052605+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:50.052605+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":391,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932372+0000","last_change":"2024-03-24T11:59:51.078512+0000","last_active":"2024-03-24T12:02:14.932372+0000","last_peered":"2024-03-24T12:02:14.932372+0000","last_clean":"2024-03-24T12:02:14.932372+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T12:02:14.932372+0000","last_undegraded":"2024-03-24T12:02:14.932372+0000","last_fullsized":"2024-03-24T12:02:14.932372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:51.078392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:51.078392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935408+0000","last_change":"2024-03-24T11:58:57.101127+0000","last_active":"2024-03-24T12:02:14.935408+0000","last_peered":"2024-03-24T12:02:14.935408+0000","last_clean":"2024-03-24T12:02:14.935408+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T12:02:14.935408+0000","last_undegraded":"2024-03-24T12:02:14.935408+0000","last_fullsized":"2024-03-24T12:02:14.935408+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T11:58:57.101013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:58:57.101013+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":379,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933674+0000","last_change":"2024-03-24T11:59:24.028954+0000","last_active":"2024-03-24T12:02:14.933674+0000","last_peered":"2024-03-24T12:02:14.933674+0000","last_clean":"2024-03-24T12:02:14.933674+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T12:02:14.933674+0000","last_undegraded":"2024-03-24T12:02:14.933674+0000","last_fullsized":"2024-03-24T12:02:14.933674+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.028868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.028868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"240'11010","reported_seq":11267,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.187524+0000","last_change":"2024-03-24T12:01:19.661270+0000","last_active":"2024-03-24T12:02:21.187524+0000","last_peered":"2024-03-24T12:02:21.187524+0000","last_clean":"2024-03-24T12:02:21.187524+0000","last_became_active":"2024-03-24T12:00:11.250460+0000","last_became_peered":"2024-03-24T12:00:11.250460+0000","last_unstale":"2024-03-24T12:02:21.187524+0000","last_undegraded":"2024-03-24T12:02:21.187524+0000","last_fullsized":"2024-03-24T12:02:21.187524+0000","mapping_epoch":170,"log_start":"186'5185","ondisk_log_start":"186'5185","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'8567","last_scrub_stamp":"2024-03-24T12:01:19.661016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.661016+0000","objects_scrubbed":2497,"log_size":5825,"log_dups_size":0,"ondisk_log_size":5825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:32.820655+0000","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":401,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.947431+0000","last_change":"2024-03-24T12:02:14.030414+0000","last_active":"2024-03-24T12:02:14.947431+0000","last_peered":"2024-03-24T12:02:14.947431+0000","last_clean":"2024-03-24T12:02:14.947431+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T12:02:14.947431+0000","last_undegraded":"2024-03-24T12:02:14.947431+0000","last_fullsized":"2024-03-24T12:02:14.947431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:14.030283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T12:02:14.030283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:41.766239+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":423,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.934955+0000","last_change":"2024-03-24T12:02:14.278544+0000","last_active":"2024-03-24T12:02:14.934955+0000","last_peered":"2024-03-24T12:02:14.934955+0000","last_clean":"2024-03-24T12:02:14.934955+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T12:02:14.934955+0000","last_undegraded":"2024-03-24T12:02:14.934955+0000","last_fullsized":"2024-03-24T12:02:14.934955+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-03-24T12:02:14.278296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:14.278296+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:31.287935+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":401,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.944428+0000","last_change":"2024-03-24T11:59:07.080907+0000","last_active":"2024-03-24T12:02:14.944428+0000","last_peered":"2024-03-24T12:02:14.944428+0000","last_clean":"2024-03-24T12:02:14.944428+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T12:02:14.944428+0000","last_undegraded":"2024-03-24T12:02:14.944428+0000","last_fullsized":"2024-03-24T12:02:14.944428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:07.080763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.080763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":367,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:20.582119+0000","last_change":"2024-03-24T11:59:22.979714+0000","last_active":"2024-03-24T12:02:20.582119+0000","last_peered":"2024-03-24T12:02:20.582119+0000","last_clean":"2024-03-24T12:02:20.582119+0000","last_became_active":"2024-03-24T11:59:19.095838+0000","last_became_peered":"2024-03-24T11:59:19.095838+0000","last_unstale":"2024-03-24T12:02:20.582119+0000","last_undegraded":"2024-03-24T12:02:20.582119+0000","last_fullsized":"2024-03-24T12:02:20.582119+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:59:22.978824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.978824+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":189,"num_read_kb":184,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:20.995868+0000","last_change":"2024-03-24T12:02:20.995891+0000","last_active":"2024-03-24T12:02:20.995868+0000","last_peered":"2024-03-24T12:02:20.995868+0000","last_clean":"2024-03-24T12:02:20.995868+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:02:20.995868+0000","last_undegraded":"2024-03-24T12:02:20.995868+0000","last_fullsized":"2024-03-24T12:02:20.995868+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:20.995809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:20.995809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:40.357955+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"240'14709","reported_seq":14863,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.264519+0000","last_change":"2024-03-24T11:58:42.817551+0000","last_active":"2024-03-24T12:02:17.264519+0000","last_peered":"2024-03-24T12:02:17.264519+0000","last_clean":"2024-03-24T12:02:17.264519+0000","last_became_active":"2024-03-24T11:58:42.817154+0000","last_became_peered":"2024-03-24T11:58:42.817154+0000","last_unstale":"2024-03-24T12:02:17.264519+0000","last_undegraded":"2024-03-24T12:02:17.264519+0000","last_fullsized":"2024-03-24T12:02:17.264519+0000","mapping_epoch":170,"log_start":"228'10429","ondisk_log_start":"228'10429","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14709,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":664,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.931102+0000","last_change":"2024-03-24T11:59:55.149768+0000","last_active":"2024-03-24T12:02:14.931102+0000","last_peered":"2024-03-24T12:02:14.931102+0000","last_clean":"2024-03-24T12:02:14.931102+0000","last_became_active":"2024-03-24T11:59:53.778968+0000","last_became_peered":"2024-03-24T11:59:53.778968+0000","last_unstale":"2024-03-24T12:02:14.931102+0000","last_undegraded":"2024-03-24T12:02:14.931102+0000","last_fullsized":"2024-03-24T12:02:14.931102+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:55.149680+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:55.149680+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":376,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.944603+0000","last_change":"2024-03-24T11:59:14.667905+0000","last_active":"2024-03-24T12:02:14.944603+0000","last_peered":"2024-03-24T12:02:14.944603+0000","last_clean":"2024-03-24T12:02:14.944603+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:02:14.944603+0000","last_undegraded":"2024-03-24T12:02:14.944603+0000","last_fullsized":"2024-03-24T12:02:14.944603+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:14.667523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.667523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":509,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:15.316275+0000","last_change":"2024-03-24T12:02:04.818118+0000","last_active":"2024-03-24T12:02:15.316275+0000","last_peered":"2024-03-24T12:02:15.316275+0000","last_clean":"2024-03-24T12:02:15.316275+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T12:02:15.316275+0000","last_undegraded":"2024-03-24T12:02:15.316275+0000","last_fullsized":"2024-03-24T12:02:15.316275+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-03-24T12:02:04.818018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:04.818018+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:19.869690+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":399,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932973+0000","last_change":"2024-03-24T12:02:14.810673+0000","last_active":"2024-03-24T12:02:14.932973+0000","last_peered":"2024-03-24T12:02:14.932973+0000","last_clean":"2024-03-24T12:02:14.932973+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T12:02:14.932973+0000","last_undegraded":"2024-03-24T12:02:14.932973+0000","last_fullsized":"2024-03-24T12:02:14.932973+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:14.810556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:02:14.810556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:44.551255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":381,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.943629+0000","last_change":"2024-03-24T12:02:06.504143+0000","last_active":"2024-03-24T12:02:14.943629+0000","last_peered":"2024-03-24T12:02:14.943629+0000","last_clean":"2024-03-24T12:02:14.943629+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:02:14.943629+0000","last_undegraded":"2024-03-24T12:02:14.943629+0000","last_fullsized":"2024-03-24T12:02:14.943629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:06.504045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:06.504045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:26.888906+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"240'11055","reported_seq":11254,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.335728+0000","last_change":"2024-03-24T12:01:08.916804+0000","last_active":"2024-03-24T12:02:21.335728+0000","last_peered":"2024-03-24T12:02:21.335728+0000","last_clean":"2024-03-24T12:02:21.335728+0000","last_became_active":"2024-03-24T12:00:11.256297+0000","last_became_peered":"2024-03-24T12:00:11.256297+0000","last_unstale":"2024-03-24T12:02:21.335728+0000","last_undegraded":"2024-03-24T12:02:21.335728+0000","last_fullsized":"2024-03-24T12:02:21.335728+0000","mapping_epoch":170,"log_start":"197'6658","ondisk_log_start":"197'6658","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'8371","last_scrub_stamp":"2024-03-24T12:01:08.915676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:08.915676+0000","objects_scrubbed":2724,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:38.623471+0000","scrub_duration":672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1835008,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":419,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932528+0000","last_change":"2024-03-24T11:59:54.989730+0000","last_active":"2024-03-24T12:02:14.932528+0000","last_peered":"2024-03-24T12:02:14.932528+0000","last_clean":"2024-03-24T12:02:14.932528+0000","last_became_active":"2024-03-24T11:59:53.773904+0000","last_became_peered":"2024-03-24T11:59:53.773904+0000","last_unstale":"2024-03-24T12:02:14.932528+0000","last_undegraded":"2024-03-24T12:02:14.932528+0000","last_fullsized":"2024-03-24T12:02:14.932528+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.989619+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.989619+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":513,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936446+0000","last_change":"2024-03-24T11:59:51.517974+0000","last_active":"2024-03-24T12:02:14.936446+0000","last_peered":"2024-03-24T12:02:14.936446+0000","last_clean":"2024-03-24T12:02:14.936446+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T12:02:14.936446+0000","last_undegraded":"2024-03-24T12:02:14.936446+0000","last_fullsized":"2024-03-24T12:02:14.936446+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:51.517833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:51.517833+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":416,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932780+0000","last_change":"2024-03-24T12:02:08.326578+0000","last_active":"2024-03-24T12:02:14.932780+0000","last_peered":"2024-03-24T12:02:14.932780+0000","last_clean":"2024-03-24T12:02:14.932780+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T12:02:14.932780+0000","last_undegraded":"2024-03-24T12:02:14.932780+0000","last_fullsized":"2024-03-24T12:02:14.932780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:02:08.325831+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.325831+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:25.659060+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936431+0000","last_change":"2024-03-24T11:59:19.117186+0000","last_active":"2024-03-24T12:02:14.936431+0000","last_peered":"2024-03-24T12:02:14.936431+0000","last_clean":"2024-03-24T12:02:14.936431+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:02:14.936431+0000","last_undegraded":"2024-03-24T12:02:14.936431+0000","last_fullsized":"2024-03-24T12:02:14.936431+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:13.313625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T11:59:13.313625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":372,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.931129+0000","last_change":"2024-03-24T11:59:18.622093+0000","last_active":"2024-03-24T12:02:14.931129+0000","last_peered":"2024-03-24T12:02:14.931129+0000","last_clean":"2024-03-24T12:02:14.931129+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T12:02:14.931129+0000","last_undegraded":"2024-03-24T12:02:14.931129+0000","last_fullsized":"2024-03-24T12:02:14.931129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T11:59:18.621992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"240'14820","reported_seq":15001,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.056604+0000","last_change":"2024-03-24T12:02:00.481611+0000","last_active":"2024-03-24T12:02:17.056604+0000","last_peered":"2024-03-24T12:02:17.056604+0000","last_clean":"2024-03-24T12:02:17.056604+0000","last_became_active":"2024-03-24T11:58:42.972967+0000","last_became_peered":"2024-03-24T11:58:42.972967+0000","last_unstale":"2024-03-24T12:02:17.056604+0000","last_undegraded":"2024-03-24T12:02:17.056604+0000","last_fullsized":"2024-03-24T12:02:17.056604+0000","mapping_epoch":170,"log_start":"218'9663","ondisk_log_start":"218'9663","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'13497","last_scrub_stamp":"2024-03-24T12:02:00.481380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:02:00.481380+0000","objects_scrubbed":1717,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:03.742164+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23724032,"num_objects":362,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14820,"num_write_kb":485824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":425,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.927334+0000","last_change":"2024-03-24T11:59:54.973664+0000","last_active":"2024-03-24T12:02:14.927334+0000","last_peered":"2024-03-24T12:02:14.927334+0000","last_clean":"2024-03-24T12:02:14.927334+0000","last_became_active":"2024-03-24T11:59:53.769258+0000","last_became_peered":"2024-03-24T11:59:53.769258+0000","last_unstale":"2024-03-24T12:02:14.927334+0000","last_undegraded":"2024-03-24T12:02:14.927334+0000","last_fullsized":"2024-03-24T12:02:14.927334+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.973532+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.973532+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":393,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.941856+0000","last_change":"2024-03-24T12:02:07.096552+0000","last_active":"2024-03-24T12:02:14.941856+0000","last_peered":"2024-03-24T12:02:14.941856+0000","last_clean":"2024-03-24T12:02:14.941856+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:02:14.941856+0000","last_undegraded":"2024-03-24T12:02:14.941856+0000","last_fullsized":"2024-03-24T12:02:14.941856+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:07.096421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:07.096421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:36.060668+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":436,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935930+0000","last_change":"2024-03-24T11:59:16.735826+0000","last_active":"2024-03-24T12:02:14.935930+0000","last_peered":"2024-03-24T12:02:14.935930+0000","last_clean":"2024-03-24T12:02:14.935930+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:02:14.935930+0000","last_undegraded":"2024-03-24T12:02:14.935930+0000","last_fullsized":"2024-03-24T12:02:14.935930+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:59:16.735674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.735674+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":411,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936602+0000","last_change":"2024-03-24T12:01:58.883035+0000","last_active":"2024-03-24T12:02:14.936602+0000","last_peered":"2024-03-24T12:02:14.936602+0000","last_clean":"2024-03-24T12:02:14.936602+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:02:14.936602+0000","last_undegraded":"2024-03-24T12:02:14.936602+0000","last_fullsized":"2024-03-24T12:02:14.936602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:01:58.882896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:01:58.882896+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:17.108850+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":373,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.944549+0000","last_change":"2024-03-24T11:59:03.131422+0000","last_active":"2024-03-24T12:02:14.944549+0000","last_peered":"2024-03-24T12:02:14.944549+0000","last_clean":"2024-03-24T12:02:14.944549+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:02:14.944549+0000","last_undegraded":"2024-03-24T12:02:14.944549+0000","last_fullsized":"2024-03-24T12:02:14.944549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:03.131309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:03.131309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"240'10910","reported_seq":11173,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.195130+0000","last_change":"2024-03-24T12:01:18.444278+0000","last_active":"2024-03-24T12:02:21.195130+0000","last_peered":"2024-03-24T12:02:21.195130+0000","last_clean":"2024-03-24T12:02:21.195130+0000","last_became_active":"2024-03-24T12:00:11.263496+0000","last_became_peered":"2024-03-24T12:00:11.263496+0000","last_unstale":"2024-03-24T12:02:21.195130+0000","last_undegraded":"2024-03-24T12:02:21.195130+0000","last_fullsized":"2024-03-24T12:02:21.195130+0000","mapping_epoch":170,"log_start":"184'4995","ondisk_log_start":"184'4995","created":170,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'8465","last_scrub_stamp":"2024-03-24T12:01:18.444128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:18.444128+0000","objects_scrubbed":2485,"log_size":5915,"log_dups_size":0,"ondisk_log_size":5915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:48.226210+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"115'13","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.934095+0000","last_change":"2024-03-24T11:58:55.576054+0000","last_active":"2024-03-24T12:02:14.934095+0000","last_peered":"2024-03-24T12:02:14.934095+0000","last_clean":"2024-03-24T12:02:14.934095+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:02:14.934095+0000","last_undegraded":"2024-03-24T12:02:14.934095+0000","last_fullsized":"2024-03-24T12:02:14.934095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'13","last_scrub_stamp":"2024-03-24T11:58:55.575897+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T11:58:55.575897+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":401,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.941761+0000","last_change":"2024-03-24T12:02:03.999738+0000","last_active":"2024-03-24T12:02:14.941761+0000","last_peered":"2024-03-24T12:02:14.941761+0000","last_clean":"2024-03-24T12:02:14.941761+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T12:02:14.941761+0000","last_undegraded":"2024-03-24T12:02:14.941761+0000","last_fullsized":"2024-03-24T12:02:14.941761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:03.999630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:03.999630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:29.107834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":377,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.949434+0000","last_change":"2024-03-24T11:59:14.960274+0000","last_active":"2024-03-24T12:02:14.949434+0000","last_peered":"2024-03-24T12:02:14.949434+0000","last_clean":"2024-03-24T12:02:14.949434+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:02:14.949434+0000","last_undegraded":"2024-03-24T12:02:14.949434+0000","last_fullsized":"2024-03-24T12:02:14.949434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:14.960108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.960108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":381,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.944567+0000","last_change":"2024-03-24T11:59:23.956872+0000","last_active":"2024-03-24T12:02:14.944567+0000","last_peered":"2024-03-24T12:02:14.944567+0000","last_clean":"2024-03-24T12:02:14.944567+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T12:02:14.944567+0000","last_undegraded":"2024-03-24T12:02:14.944567+0000","last_fullsized":"2024-03-24T12:02:14.944567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:23.956595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:23.956595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":398,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.949348+0000","last_change":"2024-03-24T11:59:23.086796+0000","last_active":"2024-03-24T12:02:14.949348+0000","last_peered":"2024-03-24T12:02:14.949348+0000","last_clean":"2024-03-24T12:02:14.949348+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:02:14.949348+0000","last_undegraded":"2024-03-24T12:02:14.949348+0000","last_fullsized":"2024-03-24T12:02:14.949348+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:59:23.086667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:23.086667+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":398,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.928147+0000","last_change":"2024-03-24T11:59:48.877101+0000","last_active":"2024-03-24T12:02:14.928147+0000","last_peered":"2024-03-24T12:02:14.928147+0000","last_clean":"2024-03-24T12:02:14.928147+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:02:14.928147+0000","last_undegraded":"2024-03-24T12:02:14.928147+0000","last_fullsized":"2024-03-24T12:02:14.928147+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:48.876979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T11:59:48.876979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":383,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933253+0000","last_change":"2024-03-24T12:01:56.816948+0000","last_active":"2024-03-24T12:02:14.933253+0000","last_peered":"2024-03-24T12:02:14.933253+0000","last_clean":"2024-03-24T12:02:14.933253+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T12:02:14.933253+0000","last_undegraded":"2024-03-24T12:02:14.933253+0000","last_fullsized":"2024-03-24T12:02:14.933253+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:56.816694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:01:56.816694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:24.786941+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930263+0000","last_change":"2024-03-24T12:02:10.050207+0000","last_active":"2024-03-24T12:02:14.930263+0000","last_peered":"2024-03-24T12:02:14.930263+0000","last_clean":"2024-03-24T12:02:14.930263+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:02:14.930263+0000","last_undegraded":"2024-03-24T12:02:14.930263+0000","last_fullsized":"2024-03-24T12:02:14.930263+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:10.050104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:10.050104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:12.810986+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"240'11085","reported_seq":18568,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.251780+0000","last_change":"2024-03-24T12:01:14.741986+0000","last_active":"2024-03-24T12:02:21.251780+0000","last_peered":"2024-03-24T12:02:21.251780+0000","last_clean":"2024-03-24T12:02:21.251780+0000","last_became_active":"2024-03-24T12:00:40.902278+0000","last_became_peered":"2024-03-24T12:00:40.902278+0000","last_unstale":"2024-03-24T12:02:21.251780+0000","last_undegraded":"2024-03-24T12:02:21.251780+0000","last_fullsized":"2024-03-24T12:02:21.251780+0000","mapping_epoch":206,"log_start":"194'6131","ondisk_log_start":"194'6131","created":196,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":5,"last_scrub":"223'8557","last_scrub_stamp":"2024-03-24T12:01:14.741839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:14.741839+0000","objects_scrubbed":2561,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:35.761593+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"94'15","reported_seq":409,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.952514+0000","last_change":"2024-03-24T11:59:57.333063+0000","last_active":"2024-03-24T12:02:14.952514+0000","last_peered":"2024-03-24T12:02:14.952514+0000","last_clean":"2024-03-24T12:02:14.952514+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:02:14.952514+0000","last_undegraded":"2024-03-24T12:02:14.952514+0000","last_fullsized":"2024-03-24T12:02:14.952514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:59:57.331734+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:59:57.331734+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":391,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933794+0000","last_change":"2024-03-24T11:59:22.001137+0000","last_active":"2024-03-24T12:02:14.933794+0000","last_peered":"2024-03-24T12:02:14.933794+0000","last_clean":"2024-03-24T12:02:14.933794+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T12:02:14.933794+0000","last_undegraded":"2024-03-24T12:02:14.933794+0000","last_fullsized":"2024-03-24T12:02:14.933794+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.000913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.000913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":521,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:15.313600+0000","last_change":"2024-03-24T12:01:16.301972+0000","last_active":"2024-03-24T12:02:15.313600+0000","last_peered":"2024-03-24T12:02:15.313600+0000","last_clean":"2024-03-24T12:02:15.313600+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T12:02:15.313600+0000","last_undegraded":"2024-03-24T12:02:15.313600+0000","last_fullsized":"2024-03-24T12:02:15.313600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:01:16.301884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:01:16.301884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:27.505394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":372,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933879+0000","last_change":"2024-03-24T11:59:16.675122+0000","last_active":"2024-03-24T12:02:14.933879+0000","last_peered":"2024-03-24T12:02:14.933879+0000","last_clean":"2024-03-24T12:02:14.933879+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:02:14.933879+0000","last_undegraded":"2024-03-24T12:02:14.933879+0000","last_fullsized":"2024-03-24T12:02:14.933879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:16.673746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.673746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"240'10855","reported_seq":17468,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.334495+0000","last_change":"2024-03-24T12:01:09.849329+0000","last_active":"2024-03-24T12:02:21.334495+0000","last_peered":"2024-03-24T12:02:21.334495+0000","last_clean":"2024-03-24T12:02:21.334495+0000","last_became_active":"2024-03-24T12:00:55.103200+0000","last_became_peered":"2024-03-24T12:00:55.103200+0000","last_unstale":"2024-03-24T12:02:21.334495+0000","last_undegraded":"2024-03-24T12:02:21.334495+0000","last_fullsized":"2024-03-24T12:02:21.334495+0000","mapping_epoch":214,"log_start":"195'6453","ondisk_log_start":"195'6453","created":196,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"221'8301","last_scrub_stamp":"2024-03-24T12:01:09.848272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:09.848272+0000","objects_scrubbed":2597,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:29.098544+0000","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":403,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933593+0000","last_change":"2024-03-24T11:59:19.016347+0000","last_active":"2024-03-24T12:02:14.933593+0000","last_peered":"2024-03-24T12:02:14.933593+0000","last_clean":"2024-03-24T12:02:14.933593+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T12:02:14.933593+0000","last_undegraded":"2024-03-24T12:02:14.933593+0000","last_fullsized":"2024-03-24T12:02:14.933593+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T11:59:19.016233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:19.016233+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936213+0000","last_change":"2024-03-24T11:59:22.532876+0000","last_active":"2024-03-24T12:02:14.936213+0000","last_peered":"2024-03-24T12:02:14.936213+0000","last_clean":"2024-03-24T12:02:14.936213+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T12:02:14.936213+0000","last_undegraded":"2024-03-24T12:02:14.936213+0000","last_fullsized":"2024-03-24T12:02:14.936213+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.532751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.532751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":514,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.952828+0000","last_change":"2024-03-24T11:59:24.105607+0000","last_active":"2024-03-24T12:02:14.952828+0000","last_peered":"2024-03-24T12:02:14.952828+0000","last_clean":"2024-03-24T12:02:14.952828+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T12:02:14.952828+0000","last_undegraded":"2024-03-24T12:02:14.952828+0000","last_fullsized":"2024-03-24T12:02:14.952828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:24.105308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.105308+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":387,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936710+0000","last_change":"2024-03-24T12:02:08.957308+0000","last_active":"2024-03-24T12:02:14.936710+0000","last_peered":"2024-03-24T12:02:14.936710+0000","last_clean":"2024-03-24T12:02:14.936710+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T12:02:14.936710+0000","last_undegraded":"2024-03-24T12:02:14.936710+0000","last_fullsized":"2024-03-24T12:02:14.936710+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:08.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:37.411715+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":178,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.926775+0000","last_change":"2024-03-24T12:02:14.181039+0000","last_active":"2024-03-24T12:02:14.926775+0000","last_peered":"2024-03-24T12:02:14.926775+0000","last_clean":"2024-03-24T12:02:14.926775+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:02:14.926775+0000","last_undegraded":"2024-03-24T12:02:14.926775+0000","last_fullsized":"2024-03-24T12:02:14.926775+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:02:14.180895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:14.180895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:19.749588+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":398,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.926656+0000","last_change":"2024-03-24T11:59:24.868399+0000","last_active":"2024-03-24T12:02:14.926656+0000","last_peered":"2024-03-24T12:02:14.926656+0000","last_clean":"2024-03-24T12:02:14.926656+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T12:02:14.926656+0000","last_undegraded":"2024-03-24T12:02:14.926656+0000","last_fullsized":"2024-03-24T12:02:14.926656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.868290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.868290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":385,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.944460+0000","last_change":"2024-03-24T12:01:19.475234+0000","last_active":"2024-03-24T12:02:14.944460+0000","last_peered":"2024-03-24T12:02:14.944460+0000","last_clean":"2024-03-24T12:02:14.944460+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T12:02:14.944460+0000","last_undegraded":"2024-03-24T12:02:14.944460+0000","last_fullsized":"2024-03-24T12:02:14.944460+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:19.474426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.474426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:45.921260+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":380,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930949+0000","last_change":"2024-03-24T12:02:04.803951+0000","last_active":"2024-03-24T12:02:14.930949+0000","last_peered":"2024-03-24T12:02:14.930949+0000","last_clean":"2024-03-24T12:02:14.930949+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T12:02:14.930949+0000","last_undegraded":"2024-03-24T12:02:14.930949+0000","last_fullsized":"2024-03-24T12:02:14.930949+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:04.803578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T12:02:04.803578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:28.836025+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":406,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936252+0000","last_change":"2024-03-24T11:59:49.721576+0000","last_active":"2024-03-24T12:02:14.936252+0000","last_peered":"2024-03-24T12:02:14.936252+0000","last_clean":"2024-03-24T12:02:14.936252+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:02:14.936252+0000","last_undegraded":"2024-03-24T12:02:14.936252+0000","last_fullsized":"2024-03-24T12:02:14.936252+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:49.721488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:49.721488+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.934316+0000","last_change":"2024-03-24T12:01:16.997174+0000","last_active":"2024-03-24T12:02:14.934316+0000","last_peered":"2024-03-24T12:02:14.934316+0000","last_clean":"2024-03-24T12:02:14.934316+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T12:02:14.934316+0000","last_undegraded":"2024-03-24T12:02:14.934316+0000","last_fullsized":"2024-03-24T12:02:14.934316+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:16.997067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:01:16.997067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:23.120606+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":144,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933723+0000","last_change":"2024-03-24T11:59:58.999301+0000","last_active":"2024-03-24T12:02:14.933723+0000","last_peered":"2024-03-24T12:02:14.933723+0000","last_clean":"2024-03-24T12:02:14.933723+0000","last_became_active":"2024-03-24T11:59:19.099779+0000","last_became_peered":"2024-03-24T11:59:19.099779+0000","last_unstale":"2024-03-24T12:02:14.933723+0000","last_undegraded":"2024-03-24T12:02:14.933723+0000","last_fullsized":"2024-03-24T12:02:14.933723+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:58.998975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:59:58.998975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":373,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935361+0000","last_change":"2024-03-24T11:59:08.970434+0000","last_active":"2024-03-24T12:02:14.935361+0000","last_peered":"2024-03-24T12:02:14.935361+0000","last_clean":"2024-03-24T12:02:14.935361+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T12:02:14.935361+0000","last_undegraded":"2024-03-24T12:02:14.935361+0000","last_fullsized":"2024-03-24T12:02:14.935361+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:08.970296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.970296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"180'6","reported_seq":397,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936167+0000","last_change":"2024-03-24T11:59:25.157079+0000","last_active":"2024-03-24T12:02:14.936167+0000","last_peered":"2024-03-24T12:02:14.936167+0000","last_clean":"2024-03-24T12:02:14.936167+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:02:14.936167+0000","last_undegraded":"2024-03-24T12:02:14.936167+0000","last_fullsized":"2024-03-24T12:02:14.936167+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'6","last_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T11:59:25.156553+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":391,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933406+0000","last_change":"2024-03-24T12:01:20.259860+0000","last_active":"2024-03-24T12:02:14.933406+0000","last_peered":"2024-03-24T12:02:14.933406+0000","last_clean":"2024-03-24T12:02:14.933406+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T12:02:14.933406+0000","last_undegraded":"2024-03-24T12:02:14.933406+0000","last_fullsized":"2024-03-24T12:02:14.933406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:20.259734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:01:20.259734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:25.698238+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":390,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.928103+0000","last_change":"2024-03-24T12:00:07.037958+0000","last_active":"2024-03-24T12:02:14.928103+0000","last_peered":"2024-03-24T12:02:14.928103+0000","last_clean":"2024-03-24T12:02:14.928103+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:02:14.928103+0000","last_undegraded":"2024-03-24T12:02:14.928103+0000","last_fullsized":"2024-03-24T12:02:14.928103+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:07.037878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:00:07.037878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":380,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.931196+0000","last_change":"2024-03-24T12:00:01.072539+0000","last_active":"2024-03-24T12:02:14.931196+0000","last_peered":"2024-03-24T12:02:14.931196+0000","last_clean":"2024-03-24T12:02:14.931196+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:02:14.931196+0000","last_undegraded":"2024-03-24T12:02:14.931196+0000","last_fullsized":"2024-03-24T12:02:14.931196+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:01.072391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:00:01.072391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936298+0000","last_change":"2024-03-24T11:59:07.046859+0000","last_active":"2024-03-24T12:02:14.936298+0000","last_peered":"2024-03-24T12:02:14.936298+0000","last_clean":"2024-03-24T12:02:14.936298+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:02:14.936298+0000","last_undegraded":"2024-03-24T12:02:14.936298+0000","last_fullsized":"2024-03-24T12:02:14.936298+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:07.046695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.046695+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":390,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.927450+0000","last_change":"2024-03-24T11:59:04.798046+0000","last_active":"2024-03-24T12:02:14.927450+0000","last_peered":"2024-03-24T12:02:14.927450+0000","last_clean":"2024-03-24T12:02:14.927450+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T12:02:14.927450+0000","last_undegraded":"2024-03-24T12:02:14.927450+0000","last_fullsized":"2024-03-24T12:02:14.927450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:04.797928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:04.797928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930447+0000","last_change":"2024-03-24T12:01:19.064543+0000","last_active":"2024-03-24T12:02:14.930447+0000","last_peered":"2024-03-24T12:02:14.930447+0000","last_clean":"2024-03-24T12:02:14.930447+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:02:14.930447+0000","last_undegraded":"2024-03-24T12:02:14.930447+0000","last_fullsized":"2024-03-24T12:02:14.930447+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:19.064391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.064391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:21.020399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":373,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.944649+0000","last_change":"2024-03-24T11:59:06.058871+0000","last_active":"2024-03-24T12:02:14.944649+0000","last_peered":"2024-03-24T12:02:14.944649+0000","last_clean":"2024-03-24T12:02:14.944649+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:02:14.944649+0000","last_undegraded":"2024-03-24T12:02:14.944649+0000","last_fullsized":"2024-03-24T12:02:14.944649+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:06.058633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.058633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"93'15","reported_seq":408,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:16.771571+0000","last_change":"2024-03-24T12:02:16.773052+0000","last_active":"2024-03-24T12:02:16.771571+0000","last_peered":"2024-03-24T12:02:16.771571+0000","last_clean":"2024-03-24T12:02:16.771571+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:02:16.771571+0000","last_undegraded":"2024-03-24T12:02:16.771571+0000","last_fullsized":"2024-03-24T12:02:16.771571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T12:02:16.771488+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:02:16.771488+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:36.015402+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":391,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935267+0000","last_change":"2024-03-24T11:59:04.929757+0000","last_active":"2024-03-24T12:02:14.935267+0000","last_peered":"2024-03-24T12:02:14.935267+0000","last_clean":"2024-03-24T12:02:14.935267+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T12:02:14.935267+0000","last_undegraded":"2024-03-24T12:02:14.935267+0000","last_fullsized":"2024-03-24T12:02:14.935267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:04.929640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:04.929640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":515,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:15.315067+0000","last_change":"2024-03-24T11:59:16.869955+0000","last_active":"2024-03-24T12:02:15.315067+0000","last_peered":"2024-03-24T12:02:15.315067+0000","last_clean":"2024-03-24T12:02:15.315067+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:02:15.315067+0000","last_undegraded":"2024-03-24T12:02:15.315067+0000","last_fullsized":"2024-03-24T12:02:15.315067+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:16.869590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.869590+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":390,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933530+0000","last_change":"2024-03-24T12:00:07.070413+0000","last_active":"2024-03-24T12:02:14.933530+0000","last_peered":"2024-03-24T12:02:14.933530+0000","last_clean":"2024-03-24T12:02:14.933530+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T12:02:14.933530+0000","last_undegraded":"2024-03-24T12:02:14.933530+0000","last_fullsized":"2024-03-24T12:02:14.933530+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:00:07.070250+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"240'11072","reported_seq":22668,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.175016+0000","last_change":"2024-03-24T12:01:17.649522+0000","last_active":"2024-03-24T12:02:21.175016+0000","last_peered":"2024-03-24T12:02:21.175016+0000","last_clean":"2024-03-24T12:02:21.175016+0000","last_became_active":"2024-03-24T12:01:16.827583+0000","last_became_peered":"2024-03-24T12:01:16.827583+0000","last_unstale":"2024-03-24T12:02:21.175016+0000","last_undegraded":"2024-03-24T12:02:21.175016+0000","last_fullsized":"2024-03-24T12:02:21.175016+0000","mapping_epoch":225,"log_start":"185'5155","ondisk_log_start":"185'5155","created":196,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"226'8634","last_scrub_stamp":"2024-03-24T12:01:17.649213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:17.649213+0000","objects_scrubbed":2479,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:29.006892+0000","scrub_duration":612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":421,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.927614+0000","last_change":"2024-03-24T12:02:12.161719+0000","last_active":"2024-03-24T12:02:14.927614+0000","last_peered":"2024-03-24T12:02:14.927614+0000","last_clean":"2024-03-24T12:02:14.927614+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T12:02:14.927614+0000","last_undegraded":"2024-03-24T12:02:14.927614+0000","last_fullsized":"2024-03-24T12:02:14.927614+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:02:12.161589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:12.161589+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:19.376501+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":386,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935522+0000","last_change":"2024-03-24T11:59:09.953682+0000","last_active":"2024-03-24T12:02:14.935522+0000","last_peered":"2024-03-24T12:02:14.935522+0000","last_clean":"2024-03-24T12:02:14.935522+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:02:14.935522+0000","last_undegraded":"2024-03-24T12:02:14.935522+0000","last_fullsized":"2024-03-24T12:02:14.935522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:59:09.953549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.953549+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.934407+0000","last_change":"2024-03-24T12:00:02.387977+0000","last_active":"2024-03-24T12:02:14.934407+0000","last_peered":"2024-03-24T12:02:14.934407+0000","last_clean":"2024-03-24T12:02:14.934407+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T12:02:14.934407+0000","last_undegraded":"2024-03-24T12:02:14.934407+0000","last_fullsized":"2024-03-24T12:02:14.934407+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:02.387837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:00:02.387837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":135,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.933929+0000","last_change":"2024-03-24T11:59:19.099058+0000","last_active":"2024-03-24T12:02:14.933929+0000","last_peered":"2024-03-24T12:02:14.933929+0000","last_clean":"2024-03-24T12:02:14.933929+0000","last_became_active":"2024-03-24T11:59:19.098557+0000","last_became_peered":"2024-03-24T11:59:19.098557+0000","last_unstale":"2024-03-24T12:02:14.933929+0000","last_undegraded":"2024-03-24T12:02:14.933929+0000","last_fullsized":"2024-03-24T12:02:14.933929+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:05.394143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:05.394143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"240'10677","reported_seq":19462,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.202247+0000","last_change":"2024-03-24T12:01:19.381593+0000","last_active":"2024-03-24T12:02:17.202247+0000","last_peered":"2024-03-24T12:02:17.202247+0000","last_clean":"2024-03-24T12:02:17.202247+0000","last_became_active":"2024-03-24T12:01:15.854244+0000","last_became_peered":"2024-03-24T12:01:15.854244+0000","last_unstale":"2024-03-24T12:02:17.202247+0000","last_undegraded":"2024-03-24T12:02:17.202247+0000","last_fullsized":"2024-03-24T12:02:17.202247+0000","mapping_epoch":224,"log_start":"195'6353","ondisk_log_start":"195'6353","created":196,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"227'8475","last_scrub_stamp":"2024-03-24T12:01:19.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.381248+0000","objects_scrubbed":2386,"log_size":4324,"log_dups_size":0,"ondisk_log_size":4324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:47.071196+0000","scrub_duration":893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2215,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"115'22","reported_seq":429,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.945078+0000","last_change":"2024-03-24T12:02:01.949233+0000","last_active":"2024-03-24T12:02:14.945078+0000","last_peered":"2024-03-24T12:02:14.945078+0000","last_clean":"2024-03-24T12:02:14.945078+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T12:02:14.945078+0000","last_undegraded":"2024-03-24T12:02:14.945078+0000","last_fullsized":"2024-03-24T12:02:14.945078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'22","last_scrub_stamp":"2024-03-24T12:02:01.948968+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:02:01.948968+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:11.553113+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":398,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.926924+0000","last_change":"2024-03-24T11:59:20.890264+0000","last_active":"2024-03-24T12:02:14.926924+0000","last_peered":"2024-03-24T12:02:14.926924+0000","last_clean":"2024-03-24T12:02:14.926924+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:02:14.926924+0000","last_undegraded":"2024-03-24T12:02:14.926924+0000","last_fullsized":"2024-03-24T12:02:14.926924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_clean_scrub_stamp":"2024-03-24T11:59:20.890106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":144,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936878+0000","last_change":"2024-03-24T11:59:21.674710+0000","last_active":"2024-03-24T12:02:14.936878+0000","last_peered":"2024-03-24T12:02:14.936878+0000","last_clean":"2024-03-24T12:02:14.936878+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:02:14.936878+0000","last_undegraded":"2024-03-24T12:02:14.936878+0000","last_fullsized":"2024-03-24T12:02:14.936878+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.674636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.674636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":387,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:18.004506+0000","last_change":"2024-03-24T12:02:18.004526+0000","last_active":"2024-03-24T12:02:18.004506+0000","last_peered":"2024-03-24T12:02:18.004506+0000","last_clean":"2024-03-24T12:02:18.004506+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T12:02:18.004506+0000","last_undegraded":"2024-03-24T12:02:18.004506+0000","last_fullsized":"2024-03-24T12:02:18.004506+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:18.004438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:22.363231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.10","version":"240'10702","reported_seq":10885,"reported_epoch":240,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-24T12:02:21.490035+0000","last_change":"2024-03-24T12:02:13.855392+0000","last_active":"2024-03-24T12:02:21.490035+0000","last_peered":"2024-03-24T12:02:21.490035+0000","last_clean":"2024-03-24T12:00:11.733655+0000","last_became_active":"2024-03-24T12:00:13.820081+0000","last_became_peered":"2024-03-24T12:00:13.820081+0000","last_unstale":"2024-03-24T12:02:21.490035+0000","last_undegraded":"2024-03-24T12:02:21.490035+0000","last_fullsized":"2024-03-24T12:02:21.490035+0000","mapping_epoch":199,"log_start":"194'5721","ondisk_log_start":"194'5721","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T11:58:41.798922+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7545,"num_write_kb":241376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[1,5,3],"avail_no_missing":["1","3","4","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"93'17","reported_seq":426,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935938+0000","last_change":"2024-03-24T11:58:53.063331+0000","last_active":"2024-03-24T12:02:14.935938+0000","last_peered":"2024-03-24T12:02:14.935938+0000","last_clean":"2024-03-24T12:02:14.935938+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:02:14.935938+0000","last_undegraded":"2024-03-24T12:02:14.935938+0000","last_fullsized":"2024-03-24T12:02:14.935938+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-03-24T11:58:53.063227+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-03-24T11:53:44.635073+0000","last_clean_scrub_stamp":"2024-03-24T11:58:53.063227+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":399,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.947917+0000","last_change":"2024-03-24T12:00:05.346648+0000","last_active":"2024-03-24T12:02:14.947917+0000","last_peered":"2024-03-24T12:02:14.947917+0000","last_clean":"2024-03-24T12:02:14.947917+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:02:14.947917+0000","last_undegraded":"2024-03-24T12:02:14.947917+0000","last_fullsized":"2024-03-24T12:02:14.947917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:00:05.346515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:00:05.346515+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":383,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.937322+0000","last_change":"2024-03-24T11:59:22.719982+0000","last_active":"2024-03-24T12:02:14.937322+0000","last_peered":"2024-03-24T12:02:14.937322+0000","last_clean":"2024-03-24T12:02:14.937322+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T12:02:14.937322+0000","last_undegraded":"2024-03-24T12:02:14.937322+0000","last_fullsized":"2024-03-24T12:02:14.937322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.719851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.719851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":381,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.949253+0000","last_change":"2024-03-24T11:59:21.206602+0000","last_active":"2024-03-24T12:02:14.949253+0000","last_peered":"2024-03-24T12:02:14.949253+0000","last_clean":"2024-03-24T12:02:14.949253+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T12:02:14.949253+0000","last_undegraded":"2024-03-24T12:02:14.949253+0000","last_fullsized":"2024-03-24T12:02:14.949253+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.206482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.206482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"240'11141","reported_seq":25224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.328528+0000","last_change":"2024-03-24T12:01:10.525741+0000","last_active":"2024-03-24T12:02:21.328528+0000","last_peered":"2024-03-24T12:02:21.328528+0000","last_clean":"2024-03-24T12:02:21.328528+0000","last_became_active":"2024-03-24T12:00:54.110320+0000","last_became_peered":"2024-03-24T12:00:54.110320+0000","last_unstale":"2024-03-24T12:02:21.328528+0000","last_undegraded":"2024-03-24T12:02:21.328528+0000","last_fullsized":"2024-03-24T12:02:21.328528+0000","mapping_epoch":213,"log_start":"195'6701","ondisk_log_start":"195'6701","created":196,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":5,"last_scrub":"221'8508","last_scrub_stamp":"2024-03-24T12:01:10.525599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:10.525599+0000","objects_scrubbed":2660,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:26.497892+0000","scrub_duration":322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":411,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936144+0000","last_change":"2024-03-24T12:00:03.691490+0000","last_active":"2024-03-24T12:02:14.936144+0000","last_peered":"2024-03-24T12:02:14.936144+0000","last_clean":"2024-03-24T12:02:14.936144+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:02:14.936144+0000","last_undegraded":"2024-03-24T12:02:14.936144+0000","last_fullsized":"2024-03-24T12:02:14.936144+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:00:03.691210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:00:03.691210+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":403,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932577+0000","last_change":"2024-03-24T12:02:07.041906+0000","last_active":"2024-03-24T12:02:14.932577+0000","last_peered":"2024-03-24T12:02:14.932577+0000","last_clean":"2024-03-24T12:02:14.932577+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T12:02:14.932577+0000","last_undegraded":"2024-03-24T12:02:14.932577+0000","last_fullsized":"2024-03-24T12:02:14.932577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:07.041777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:07.041777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:12.080752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":383,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932325+0000","last_change":"2024-03-24T12:01:14.211440+0000","last_active":"2024-03-24T12:02:14.932325+0000","last_peered":"2024-03-24T12:02:14.932325+0000","last_clean":"2024-03-24T12:02:14.932325+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T12:02:14.932325+0000","last_undegraded":"2024-03-24T12:02:14.932325+0000","last_fullsized":"2024-03-24T12:02:14.932325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:14.211332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:01:14.211332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:32.270180+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":372,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.931632+0000","last_change":"2024-03-24T12:02:05.800911+0000","last_active":"2024-03-24T12:02:14.931632+0000","last_peered":"2024-03-24T12:02:14.931632+0000","last_clean":"2024-03-24T12:02:14.931632+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:02:14.931632+0000","last_undegraded":"2024-03-24T12:02:14.931632+0000","last_fullsized":"2024-03-24T12:02:14.931632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:05.800774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:05.800774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:17.921551+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"240'12640","reported_seq":25863,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.241688+0000","last_change":"2024-03-24T12:01:30.287391+0000","last_active":"2024-03-24T12:02:17.241688+0000","last_peered":"2024-03-24T12:02:17.241688+0000","last_clean":"2024-03-24T12:02:17.241688+0000","last_became_active":"2024-03-24T12:01:06.418930+0000","last_became_peered":"2024-03-24T12:01:06.418930+0000","last_unstale":"2024-03-24T12:02:17.241688+0000","last_undegraded":"2024-03-24T12:02:17.241688+0000","last_fullsized":"2024-03-24T12:02:17.241688+0000","mapping_epoch":219,"log_start":"226'8397","ondisk_log_start":"226'8397","created":196,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":5,"last_scrub":"230'10807","last_scrub_stamp":"2024-03-24T12:01:30.287167+0000","last_deep_scrub":"230'10807","last_deep_scrub_stamp":"2024-03-24T12:01:30.287167+0000","last_clean_scrub_stamp":"2024-03-24T12:01:30.287167+0000","objects_scrubbed":2210,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":13,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:30.559180+0000","scrub_duration":12865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":407,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936405+0000","last_change":"2024-03-24T12:01:19.495169+0000","last_active":"2024-03-24T12:02:14.936405+0000","last_peered":"2024-03-24T12:02:14.936405+0000","last_clean":"2024-03-24T12:02:14.936405+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:02:14.936405+0000","last_undegraded":"2024-03-24T12:02:14.936405+0000","last_fullsized":"2024-03-24T12:02:14.936405+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T12:01:19.495033+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.495033+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:24.308680+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930387+0000","last_change":"2024-03-24T12:00:03.053959+0000","last_active":"2024-03-24T12:02:14.930387+0000","last_peered":"2024-03-24T12:02:14.930387+0000","last_clean":"2024-03-24T12:02:14.930387+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:02:14.930387+0000","last_undegraded":"2024-03-24T12:02:14.930387+0000","last_fullsized":"2024-03-24T12:02:14.930387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:00:03.053839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":392,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.930997+0000","last_change":"2024-03-24T12:02:08.062917+0000","last_active":"2024-03-24T12:02:14.930997+0000","last_peered":"2024-03-24T12:02:14.930997+0000","last_clean":"2024-03-24T12:02:14.930997+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:02:14.930997+0000","last_undegraded":"2024-03-24T12:02:14.930997+0000","last_fullsized":"2024-03-24T12:02:14.930997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:08.062779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.062779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:08.263500+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":387,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.932683+0000","last_change":"2024-03-24T12:01:21.229722+0000","last_active":"2024-03-24T12:02:14.932683+0000","last_peered":"2024-03-24T12:02:14.932683+0000","last_clean":"2024-03-24T12:02:14.932683+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T12:02:14.932683+0000","last_undegraded":"2024-03-24T12:02:14.932683+0000","last_fullsized":"2024-03-24T12:02:14.932683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:21.229585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:01:21.229585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:38.432494+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.17","version":"240'10878","reported_seq":19820,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.273817+0000","last_change":"2024-03-24T12:01:20.531013+0000","last_active":"2024-03-24T12:02:21.273817+0000","last_peered":"2024-03-24T12:02:21.273817+0000","last_clean":"2024-03-24T12:02:21.273817+0000","last_became_active":"2024-03-24T12:01:14.849548+0000","last_became_peered":"2024-03-24T12:01:14.849548+0000","last_unstale":"2024-03-24T12:02:21.273817+0000","last_undegraded":"2024-03-24T12:02:21.273817+0000","last_fullsized":"2024-03-24T12:02:21.273817+0000","mapping_epoch":223,"log_start":"201'6668","ondisk_log_start":"201'6668","created":196,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":5,"last_scrub":"228'8486","last_scrub_stamp":"2024-03-24T12:01:20.530700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:20.530700+0000","objects_scrubbed":2429,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:24.804316+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2399,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"176'13","reported_seq":407,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935625+0000","last_change":"2024-03-24T11:59:14.935407+0000","last_active":"2024-03-24T12:02:14.935625+0000","last_peered":"2024-03-24T12:02:14.935625+0000","last_clean":"2024-03-24T12:02:14.935625+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:02:14.935625+0000","last_undegraded":"2024-03-24T12:02:14.935625+0000","last_fullsized":"2024-03-24T12:02:14.935625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'13","last_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.935318+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.943582+0000","last_change":"2024-03-24T12:00:06.381223+0000","last_active":"2024-03-24T12:02:14.943582+0000","last_peered":"2024-03-24T12:02:14.943582+0000","last_clean":"2024-03-24T12:02:14.943582+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T12:02:14.943582+0000","last_undegraded":"2024-03-24T12:02:14.943582+0000","last_fullsized":"2024-03-24T12:02:14.943582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:00:06.381099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":144,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:17.838619+0000","last_change":"2024-03-24T12:02:17.838660+0000","last_active":"2024-03-24T12:02:17.838619+0000","last_peered":"2024-03-24T12:02:17.838619+0000","last_clean":"2024-03-24T12:02:17.838619+0000","last_became_active":"2024-03-24T11:59:19.117244+0000","last_became_peered":"2024-03-24T11:59:19.117244+0000","last_unstale":"2024-03-24T12:02:17.838619+0000","last_undegraded":"2024-03-24T12:02:17.838619+0000","last_fullsized":"2024-03-24T12:02:17.838619+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:17.838539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:17.838539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:19.996793+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":387,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936115+0000","last_change":"2024-03-24T12:01:59.894957+0000","last_active":"2024-03-24T12:02:14.936115+0000","last_peered":"2024-03-24T12:02:14.936115+0000","last_clean":"2024-03-24T12:02:14.936115+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T12:02:14.936115+0000","last_undegraded":"2024-03-24T12:02:14.936115+0000","last_fullsized":"2024-03-24T12:02:14.936115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:59.894840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:01:59.894840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:21.806890+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.14","version":"240'10675","reported_seq":16755,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.176732+0000","last_change":"2024-03-24T12:01:12.206892+0000","last_active":"2024-03-24T12:02:21.176732+0000","last_peered":"2024-03-24T12:02:21.176732+0000","last_clean":"2024-03-24T12:02:21.176732+0000","last_became_active":"2024-03-24T12:01:10.805577+0000","last_became_peered":"2024-03-24T12:01:10.805577+0000","last_unstale":"2024-03-24T12:02:21.176732+0000","last_undegraded":"2024-03-24T12:02:21.176732+0000","last_fullsized":"2024-03-24T12:02:21.176732+0000","mapping_epoch":221,"log_start":"192'5589","ondisk_log_start":"192'5589","created":196,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":5,"last_scrub":"222'8196","last_scrub_stamp":"2024-03-24T12:01:12.206553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:12.206553+0000","objects_scrubbed":2537,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:27.226435+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2487,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"117'22","reported_seq":421,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.936400+0000","last_change":"2024-03-24T12:02:02.488565+0000","last_active":"2024-03-24T12:02:14.936400+0000","last_peered":"2024-03-24T12:02:14.936400+0000","last_clean":"2024-03-24T12:02:14.936400+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T12:02:14.936400+0000","last_undegraded":"2024-03-24T12:02:14.936400+0000","last_fullsized":"2024-03-24T12:02:14.936400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'22","last_scrub_stamp":"2024-03-24T12:02:02.488418+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:02:02.488418+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:32.060704+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.947993+0000","last_change":"2024-03-24T11:59:50.555909+0000","last_active":"2024-03-24T12:02:14.947993+0000","last_peered":"2024-03-24T12:02:14.947993+0000","last_clean":"2024-03-24T12:02:14.947993+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:02:14.947993+0000","last_undegraded":"2024-03-24T12:02:14.947993+0000","last_fullsized":"2024-03-24T12:02:14.947993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:50.555718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T11:59:50.555718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":391,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.935785+0000","last_change":"2024-03-24T11:59:17.681747+0000","last_active":"2024-03-24T12:02:14.935785+0000","last_peered":"2024-03-24T12:02:14.935785+0000","last_clean":"2024-03-24T12:02:14.935785+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T12:02:14.935785+0000","last_undegraded":"2024-03-24T12:02:14.935785+0000","last_fullsized":"2024-03-24T12:02:14.935785+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:17.681628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:59:17.681628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":372,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:14.938664+0000","last_change":"2024-03-24T11:59:02.144257+0000","last_active":"2024-03-24T12:02:14.938664+0000","last_peered":"2024-03-24T12:02:14.938664+0000","last_clean":"2024-03-24T12:02:14.938664+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:02:14.938664+0000","last_undegraded":"2024-03-24T12:02:14.938664+0000","last_fullsized":"2024-03-24T12:02:14.938664+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:02.144154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:02.144154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"240'10812","reported_seq":17091,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-24T12:02:21.266974+0000","last_change":"2024-03-24T12:01:17.589883+0000","last_active":"2024-03-24T12:02:21.266974+0000","last_peered":"2024-03-24T12:02:21.266974+0000","last_clean":"2024-03-24T12:02:21.266974+0000","last_became_active":"2024-03-24T12:01:05.481286+0000","last_became_peered":"2024-03-24T12:01:05.481286+0000","last_unstale":"2024-03-24T12:02:21.266974+0000","last_undegraded":"2024-03-24T12:02:21.266974+0000","last_fullsized":"2024-03-24T12:02:21.266974+0000","mapping_epoch":218,"log_start":"200'6585","ondisk_log_start":"200'6585","created":196,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"226'8431","last_scrub_stamp":"2024-03-24T12:01:17.589695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:41.798922+0000","last_clean_scrub_stamp":"2024-03-24T12:01:17.589695+0000","objects_scrubbed":2429,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:38.215292+0000","scrub_duration":583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2391,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":198705176,"num_objects":3033,"num_object_clones":0,"num_object_copies":9099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144038,"num_write_kb":3180705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":619786240,"data_stored":619774024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126155,"ondisk_log_size":126155,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":189,"num_read_kb":184,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":486,"ondisk_log_size":486,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005181,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":445628,"kb_used_data":93748,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93299524,"statfs":{"total":95995035648,"available":95538712576,"internally_reserved":0,"allocated":95997952,"data_stored":95603125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":360315338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332228,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":382964,"kb_used_data":40812,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93362188,"statfs":{"total":95995035648,"available":95602880512,"internally_reserved":0,"allocated":41791488,"data_stored":41404647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575004,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384512,"kb_used_data":32632,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93360640,"statfs":{"total":95995035648,"available":95601295360,"internally_reserved":0,"allocated":33415168,"data_stored":33015618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738529,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546936,"kb_used_data":140848,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93198216,"statfs":{"total":95995035648,"available":95434973184,"internally_reserved":0,"allocated":144228352,"data_stored":143836464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411480,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":432260,"kb_used_data":35964,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93312892,"statfs":{"total":95995035648,"available":95552401408,"internally_reserved":0,"allocated":36827136,"data_stored":36424478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117137,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641812,"kb_used_data":223308,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93103340,"statfs":{"total":95995035648,"available":95337820160,"internally_reserved":0,"allocated":228667392,"data_stored":228248756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920909,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":378436,"kb_used_data":32124,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93366716,"statfs":{"total":95995035648,"available":95607517184,"internally_reserved":0,"allocated":32894976,"data_stored":32491365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659272,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":510212,"kb_used_data":150012,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93234940,"statfs":{"total":95995035648,"available":95472578560,"internally_reserved":0,"allocated":153612288,"data_stored":153209357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135397376,"data_stored":135397376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21889024,"data_stored":21889024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9244672,"data_stored":9240600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":197001216,"data_stored":197001216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":23134208,"data_stored":23134208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":128323584,"data_stored":128319512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20250624,"data_stored":20250624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":84545536,"data_stored":84541464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:02:26.188 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-03-24T12:02:26.188 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-24T12:02:26.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-24T12:02:26.189 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:26.217 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:26.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:26.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:26 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:02:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:26 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:02:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:26 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T12:02:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:26 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T12:02:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:26 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T12:02:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:26 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T12:02:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:26 smithi193 ceph-mon[33593]: 6.1d scrub starts 2024-03-24T12:02:26.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:26 smithi193 ceph-mon[33593]: 6.1d scrub ok 2024-03-24T12:02:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:26.344 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:26.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:26.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:26.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:26.573 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:26.601 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:26.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:26.703 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:26.727 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:26.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:26.768 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:02:26.768 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:02:26.768 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:02:26.768 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:02:26.768 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:02:26.768 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:02:26.768 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:02:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:02:26.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:26.857 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:26.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:26.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:26.984 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:26.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:27.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:27.109 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:27.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:27.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:27.240 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:27.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:27.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: pgmap v449: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 34 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 713 op/s 2024-03-24T12:02:27.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: 2.14 deep-scrub starts 2024-03-24T12:02:27.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: 2.14 deep-scrub ok 2024-03-24T12:02:27.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: 4.1d deep-scrub starts 2024-03-24T12:02:27.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: 4.1d deep-scrub ok 2024-03-24T12:02:27.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:27.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1384347477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:27.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T12:02:27.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T12:02:27.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: 2.12 scrub starts 2024-03-24T12:02:27.291 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-mon[33593]: 2.12 scrub ok 2024-03-24T12:02:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: pgmap v449: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 34 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 713 op/s 2024-03-24T12:02:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: 2.14 deep-scrub starts 2024-03-24T12:02:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: 2.14 deep-scrub ok 2024-03-24T12:02:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: 4.1d deep-scrub starts 2024-03-24T12:02:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: 4.1d deep-scrub ok 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1384347477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: 2.12 scrub starts 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[42243]: 2.12 scrub ok 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: pgmap v449: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 34 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 713 op/s 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: 2.14 deep-scrub starts 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: 2.14 deep-scrub ok 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: 4.1d deep-scrub starts 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: 4.1d deep-scrub ok 2024-03-24T12:02:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:27.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1384347477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:27.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T12:02:27.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T12:02:27.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: 2.12 scrub starts 2024-03-24T12:02:27.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-mon[30223]: 2.12 scrub ok 2024-03-24T12:02:27.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:27.368 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:27.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:27.470 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:27.497 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:27.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:27.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:02:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:02:27.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:27.622 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:27.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:27.641 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:02:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:02:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:27.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:27.752 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:27.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:27.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:27.878 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:27.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:27.979 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:28.007 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:28.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:28.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:28.137 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:28.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:28.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:28.262 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:28.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:28.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T12:02:28.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T12:02:28.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T12:02:28.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T12:02:28.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[30223]: 2.18 scrub starts 2024-03-24T12:02:28.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[30223]: 2.18 scrub ok 2024-03-24T12:02:28.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T12:02:28.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T12:02:28.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T12:02:28.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T12:02:28.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[42243]: 2.18 scrub starts 2024-03-24T12:02:28.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:28 smithi064 ceph-mon[42243]: 2.18 scrub ok 2024-03-24T12:02:28.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:28 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T12:02:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:28 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T12:02:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:28 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T12:02:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:28 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T12:02:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:28 smithi193 ceph-mon[33593]: 2.18 scrub starts 2024-03-24T12:02:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:28 smithi193 ceph-mon[33593]: 2.18 scrub ok 2024-03-24T12:02:28.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:28.390 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:28.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:28.492 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:28.520 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:28.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:28.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:28.645 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:28.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:28.708 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:28.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:28.769 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:28.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:28.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:28.871 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:28.895 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:28.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:28.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:29.020 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:29.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:29.032 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:02:29.122 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:29.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:29.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:29.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:29.279 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:29.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:29.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:29.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:29.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: pgmap v450: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 873 op/s 2024-03-24T12:02:29.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: 1.9 scrub starts 2024-03-24T12:02:29.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: 1.9 scrub ok 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3101681266' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: pgmap v450: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 873 op/s 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: 1.9 scrub starts 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: 1.9 scrub ok 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T12:02:29.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T12:02:29.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T12:02:29.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T12:02:29.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:29 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3101681266' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:02:29.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:29.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: pgmap v450: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 873 op/s 2024-03-24T12:02:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: 1.9 scrub starts 2024-03-24T12:02:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: 1.9 scrub ok 2024-03-24T12:02:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T12:02:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T12:02:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T12:02:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T12:02:29.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:29 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3101681266' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:02:29.381 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:29.408 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:29.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:29.510 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:29.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:02:29.536 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:29.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:29.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:29.661 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:29.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:29.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:29.790 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:29.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:29.892 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:29.919 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:29.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:30.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:30.047 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:30.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:30.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:30.173 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:30.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:30.275 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3101681266' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[30223]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3101681266' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:02:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:30 smithi064 ceph-mon[42243]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T12:02:30.303 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:30.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:30 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T12:02:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:30 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T12:02:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:30 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:30 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3101681266' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:02:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:30 smithi193 ceph-mon[33593]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T12:02:30.405 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:30.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:30.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:30.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:30.562 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:30.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:30.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:30.688 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:30.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:30.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:30.806 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:30.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:30.917 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:30.945 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:30.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:31.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:31.070 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:31.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:31.163 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:31 smithi064 ceph-mon[30223]: pgmap v452: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 515 op/s 2024-03-24T12:02:31.163 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:31 smithi064 ceph-mon[42243]: pgmap v452: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 515 op/s 2024-03-24T12:02:31.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:31.189 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:02:31.189 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:02:31.195 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:31.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:31.296 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:31.324 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:31.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:31 smithi193 ceph-mon[33593]: pgmap v452: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 515 op/s 2024-03-24T12:02:31.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:31.453 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:31.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:31.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:31.582 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:31.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:31.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:31.713 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:31.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:31.814 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:31.841 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:31.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:31.915 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:02:31.915 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:02:31.942 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:31.967 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:31.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:32.069 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:32.096 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:32.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:32.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:32.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:32.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:32.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:32.347 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:32.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:32.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:02:32.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:32.476 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:32.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:32.578 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:32.607 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:32.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:32.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:32.732 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:32.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:32.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:32.858 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:32.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:32.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:32.986 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:32.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:33.053 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_1' removed 2024-03-24T12:02:33.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:33.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:33.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:33.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:33 smithi064 ceph-mon[30223]: pgmap v453: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 154 op/s 2024-03-24T12:02:33.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:33 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1259428415' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:02:33.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:33 smithi064 ceph-mon[30223]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:02:33.263 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:33 smithi064 ceph-mon[42243]: pgmap v453: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 154 op/s 2024-03-24T12:02:33.263 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:33 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1259428415' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:02:33.263 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:33 smithi064 ceph-mon[42243]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:02:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:33.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:33.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:33 smithi193 ceph-mon[33593]: pgmap v453: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 154 op/s 2024-03-24T12:02:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:33 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1259428415' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:02:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:33 smithi193 ceph-mon[33593]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:02:33.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:33.395 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:33.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:33.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:33.521 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:33.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:33.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:33.647 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:33.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:33.749 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:33.772 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:33.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:33.874 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:33.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:33.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:33.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:34.024 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:34.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:34.125 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:34.149 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:34.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:34.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:34.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:34 smithi064 ceph-mon[30223]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:02:34.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:34 smithi064 ceph-mon[30223]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T12:02:34.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:34 smithi064 ceph-mon[42243]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:02:34.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:34 smithi064 ceph-mon[42243]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T12:02:34.277 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:34.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:34.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:34 smithi193 ceph-mon[33593]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:02:34.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:34 smithi193 ceph-mon[33593]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T12:02:34.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:34.402 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:34.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:34.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:34.527 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:34.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:34.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:34.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:34.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:34.754 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:34.778 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:34.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:34.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:34.906 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:34.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:34.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:02:35.007 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:35.033 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:35.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:35.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:35.138 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:35 smithi064 ceph-mon[30223]: pgmap v455: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-24T12:02:35.138 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:35 smithi064 ceph-mon[30223]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:02:35.138 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:35 smithi064 ceph-mon[42243]: pgmap v455: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-24T12:02:35.138 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:35 smithi064 ceph-mon[42243]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:02:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:35.162 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:35.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:35.171 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:02:35.193 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:02:35.212 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:02:35.233 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:02:35.259 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:02:35.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:35.280 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:02:35.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:35.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:35.300 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:02:35.320 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:02:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:02:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:02:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:02:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:02:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:02:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:02:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:02:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:02:35.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:35 smithi193 ceph-mon[33593]: pgmap v455: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-24T12:02:35.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:35 smithi193 ceph-mon[33593]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:02:35.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:35.415 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:35.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:35.517 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:35.542 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:35.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:35.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:35.668 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:35.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:35.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:35.776 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T12:02:35.777 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T12:02:35.777 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-24T12:02:35.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-24T12:02:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:35.794 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:35.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:35.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:35.919 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:35.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:36.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:36.047 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:36.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:36.149 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:36.175 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:36.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:36.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:36.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:36 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2345427565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:36.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:36 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2345427565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:36.302 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:36.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:36.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:36 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2345427565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:36.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:36.429 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:36.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:36.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:36.531 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:36.542 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:36.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:36.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:36.581 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:36.662 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:36.690 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:36.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:36.792 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:36.820 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:36.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:36.922 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:36.950 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:36.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:37.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:37.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:37 smithi064 ceph-mon[42243]: pgmap v456: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:02:37.076 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:37 smithi064 ceph-mon[30223]: pgmap v456: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:02:37.077 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:37.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:37.179 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:37.205 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:37.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:37.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:37.332 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:37.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:37.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:37 smithi193 ceph-mon[33593]: pgmap v456: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:02:37.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:02:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:02:37.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:37.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:37.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:37.557 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:37.585 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:37.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:37.628 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:02:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:02:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:37.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:37.713 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:37.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:37.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:37.839 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:37.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:37.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:37.965 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:37.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:38.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:38.094 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:38.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:38.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:38.222 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:38.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:38.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:38.346 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:38.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:38.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:38.475 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:38.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:38.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:02:38.577 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:38.604 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:38.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:38.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:38.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:38.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:38.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:38.858 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:38.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:38.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:38.984 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:38.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:39.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:39.110 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:39.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:39.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:39.235 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:39.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:39.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:39 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:39.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:39 smithi064 ceph-mon[30223]: pgmap v457: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:02:39.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:39 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:39.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:39 smithi064 ceph-mon[42243]: pgmap v457: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:02:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:39 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:39 smithi193 ceph-mon[33593]: pgmap v457: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:02:39.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:39.365 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:39.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:39.467 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:39.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:39.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:39.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:39.622 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:39.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:39.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:39.750 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:39.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:39.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:39.878 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:39.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:39.979 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:40.003 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:40.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:40.105 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:40.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:40.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:40.144 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:40.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:40.256 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:40.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:40.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:40.382 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:40.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:40.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:40.510 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:40.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:40.611 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:40.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:40.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:40.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:40.765 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:40.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:40.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:40.891 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:40.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:40.990 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:02:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-03-24T12:02:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-03-24T12:02:40.991 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:02:40.992 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:41.020 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:41.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:41.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:41.145 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:41.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:41.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:41.274 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:41.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:41.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:41.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:41 smithi193 ceph-mon[33593]: pgmap v458: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T12:02:41.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:41.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:41.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:41.446 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:41 smithi064 ceph-mon[30223]: pgmap v458: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T12:02:41.447 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:41 smithi064 ceph-mon[42243]: pgmap v458: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T12:02:41.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:41.527 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:41.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:41.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:41.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:41.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:41.754 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:41.779 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:41.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:41.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:41.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:02:41.906 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:41.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:42.008 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:42.033 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:42.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:42.075 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:02:42.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:42.158 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:42.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:42.190 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2032753203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:42.190 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:42 smithi064 ceph-mon[30223]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:42.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:42.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2032753203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:42 smithi064 ceph-mon[42243]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:42.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:42.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:42.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2032753203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:42 smithi193 ceph-mon[33593]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:02:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:42.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:42.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:42.412 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:42.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:42.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:42.537 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:42.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:42.639 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:42.667 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:42.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:42.769 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:42.797 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:42.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:42.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:42.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:42.926 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:42.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:43.028 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:43.056 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:43.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:43.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:43.186 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:43.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:43.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:43 smithi064 ceph-mon[30223]: pgmap v459: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T12:02:43.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:43 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:43.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:43 smithi064 ceph-mon[30223]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:02:43.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:43 smithi064 ceph-mon[30223]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T12:02:43.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:43 smithi064 ceph-mon[42243]: pgmap v459: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T12:02:43.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:43 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:43.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:43 smithi064 ceph-mon[42243]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:02:43.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:43 smithi064 ceph-mon[42243]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T12:02:43.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:43.315 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:43.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:43 smithi193 ceph-mon[33593]: pgmap v459: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T12:02:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:43 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:43 smithi193 ceph-mon[33593]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:02:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:43 smithi193 ceph-mon[33593]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T12:02:43.418 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:43.447 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:43.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:43.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:43.574 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:43.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:43.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:43.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:43.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:43.831 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:43.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:43.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:43.960 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:43.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:44.061 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:44.089 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:44.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:44.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:44.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:44.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:44.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:44.344 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:44.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:44.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:44.469 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:44.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:44.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:44.594 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:44.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:44.670 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:44.688 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:02:44.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:44.712 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:02:44.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:44.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:44.731 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:02:44.750 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:02:44.770 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:02:44.791 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:02:44.811 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:02:44.821 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:44.831 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:02:44.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:02:44.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:02:44.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:02:44.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:02:44.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:02:44.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:02:44.833 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:02:44.833 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:02:44.849 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:44.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:44.951 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:44.978 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:44.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:45.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:02:45.078 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_2' created 2024-03-24T12:02:45.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:45.107 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:45.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:45.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:45.233 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:45.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:45.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:45.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:45 smithi193 ceph-mon[33593]: pgmap v461: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:02:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:45 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1876977696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:02:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:45 smithi193 ceph-mon[33593]: from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:02:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:45 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/445453948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:45.359 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:45.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:45.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:45.484 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:45.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:45.504 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:45 smithi064 ceph-mon[30223]: pgmap v461: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:02:45.504 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1876977696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:02:45.504 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:45 smithi064 ceph-mon[30223]: from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:02:45.505 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/445453948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:45.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:45 smithi064 ceph-mon[42243]: pgmap v461: 142 pgs: 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:02:45.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:45 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1876977696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:02:45.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:45 smithi064 ceph-mon[42243]: from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:02:45.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:45 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/445453948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:45.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:45.609 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:45.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:45.711 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:45.738 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:45.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:45.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:45.863 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:45.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:45.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-24T12:02:45.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:46.028 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:46.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:46.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:46.157 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:46.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:46.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:46.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:46.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:46 smithi064 ceph-mon[30223]: from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T12:02:46.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:46 smithi064 ceph-mon[30223]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T12:02:46.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:46 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:02:46.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:46 smithi064 ceph-mon[42243]: from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T12:02:46.288 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:46 smithi064 ceph-mon[42243]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T12:02:46.288 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:46 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:02:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:46 smithi193 ceph-mon[33593]: from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T12:02:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:46 smithi193 ceph-mon[33593]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T12:02:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:46 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:02:46.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:46.409 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:46.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:46.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:46.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:46.645 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:46.670 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:46.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:46.772 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:46.796 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:46.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:46.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:46.922 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:46.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:46.976 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:46.976 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:47.023 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:47.048 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:47.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:47.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:47.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:47.276 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:47.299 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:47.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:47.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:02:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-mon[33593]: pgmap v463: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:02:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:02:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:02:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-mon[33593]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:02:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-mon[33593]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T12:02:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:02:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:02:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:47 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:47.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[30223]: pgmap v463: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:02:47.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[30223]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[30223]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[42243]: pgmap v463: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[42243]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[42243]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T12:02:47.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:02:47.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:02:47.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:02:47.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:47.425 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:47.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:47.526 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:47.550 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:47.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:47.652 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:47.676 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:47.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:47.735 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:02:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:02:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:47.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:47.802 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:47.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:47.904 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:47.928 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:47.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:48.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:48.053 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:48.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:48.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:48.179 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:48.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:48.216 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:02:48.216 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:02:48.281 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:48.304 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:48.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:48.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:48 smithi064 ceph-mon[30223]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T12:02:48.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:48 smithi064 ceph-mon[42243]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T12:02:48.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:48.431 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:48.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:48.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:02:48.532 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:48.558 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:48.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:48 smithi193 ceph-mon[33593]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T12:02:48.658 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:48.682 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:48.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:48.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:48.809 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:48.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:48.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:48.934 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:48.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:48.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:49.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:49.061 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:49.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:49.163 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:49.187 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:49.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:49.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:49.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:49.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:49.416 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:49.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:49.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[30223]: pgmap v466: 158 pgs: 8 creating+activating, 150 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-24T12:02:49.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:02:49.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:02:49.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[30223]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T12:02:49.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:49.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[42243]: pgmap v466: 158 pgs: 8 creating+activating, 150 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-24T12:02:49.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:02:49.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:02:49.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:49 smithi064 ceph-mon[42243]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T12:02:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:49.442 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:49.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:49.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:49.568 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:49.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:49 smithi193 ceph-mon[33593]: pgmap v466: 158 pgs: 8 creating+activating, 150 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-24T12:02:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:49 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:02:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:49 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:02:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:49 smithi193 ceph-mon[33593]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T12:02:49.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:49.694 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:49.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:49.796 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:49.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:49.927 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:49.951 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:49.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:50.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:50.076 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:50.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:50.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:50.202 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:50.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:50.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:50.330 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:50.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:50.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:50.431 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:50.456 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:50.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:50.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:50.583 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:50.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:50.685 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:50.708 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:50.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:50.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:02:50.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:50.836 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:50.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:50.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:50.961 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:50.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:51.063 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:51.088 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:51.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:51.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:51.215 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:51.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:51 smithi064 ceph-mon[30223]: pgmap v468: 158 pgs: 8 creating+activating, 150 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:51 smithi064 ceph-mon[42243]: pgmap v468: 158 pgs: 8 creating+activating, 150 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:51.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:51.341 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:51.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:51.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:51 smithi193 ceph-mon[33593]: pgmap v468: 158 pgs: 8 creating+activating, 150 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:51.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:51.467 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:51.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:51.542 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:51.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:51.581 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:51.594 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:51.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:51.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:51.719 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:51.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:51.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:51.845 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:51.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:51.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:51.971 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:51.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:51.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:52.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:52.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:52.102 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-24T12:02:52.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:52.225 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:52.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:52.327 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:52.411 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:52.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:52.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:52.541 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:52.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3117463539' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:02:52.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:52 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3117463539' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:02:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:52 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3117463539' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:02:52.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:52.667 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:52.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:52.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:52.791 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:52.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:52.893 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:52.921 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:52.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:53.023 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:53.050 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:53.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:53.109 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:02:53.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:53.175 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:53.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:53.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:53.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:53 smithi064 ceph-mon[30223]: pgmap v469: 158 pgs: 3 creating+activating, 155 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:53.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3117463539' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:02:53.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:53 smithi064 ceph-mon[30223]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T12:02:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1423655860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:02:53.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:53 smithi064 ceph-mon[42243]: pgmap v469: 158 pgs: 3 creating+activating, 155 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:53.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3117463539' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:02:53.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:53 smithi064 ceph-mon[42243]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T12:02:53.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1423655860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:02:53.305 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:53.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:53.407 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:53.434 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:53.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:53.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:53.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:53.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:53 smithi193 ceph-mon[33593]: pgmap v469: 158 pgs: 3 creating+activating, 155 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3117463539' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:02:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:53 smithi193 ceph-mon[33593]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T12:02:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1423655860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:02:53.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:53.684 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:53.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:53.786 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:53.814 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:53.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:53.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:53.943 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:53.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:53.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:54.045 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:54.073 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:54.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:54.148 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:02:54.167 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:02:54.175 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:54.186 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:02:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:54.201 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:54.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:54.206 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:02:54.226 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:02:54.247 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:02:54.270 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:02:54.298 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:02:54.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:02:54.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:02:54.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:02:54.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:02:54.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:02:54.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:02:54.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:02:54.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:02:54.303 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:54.328 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:54.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:54.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:54.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:54.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1423655860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:02:54.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[30223]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T12:02:54.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T12:02:54.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T12:02:54.433 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:54.433 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1423655860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:02:54.433 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[42243]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T12:02:54.433 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T12:02:54.433 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:54 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T12:02:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:54.458 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:54.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:54.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:54.583 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:54.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:54 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:02:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1423655860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:02:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:54 smithi193 ceph-mon[33593]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T12:02:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:54 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T12:02:54.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:54 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T12:02:54.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:54.708 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:54.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:54.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:54.834 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:54.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:54.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:54.963 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:54.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:54.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:55.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:55.089 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:55.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:55.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:55.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[30223]: pgmap v472: 158 pgs: 158 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[30223]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/182540481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[42243]: pgmap v472: 158 pgs: 158 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[42243]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/182540481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T12:02:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:55 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T12:02:55.317 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:55.345 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:55.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:55.447 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:55.470 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:55.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:55.572 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:55 smithi193 ceph-mon[33593]: pgmap v472: 158 pgs: 158 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:55 smithi193 ceph-mon[33593]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:02:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:55 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/182540481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:02:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:55 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T12:02:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:55 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T12:02:55.600 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:55.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:55.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:55.729 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:55.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:55.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:55.855 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:55.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:55.950 DEBUG:teuthology.orchestra.run.smithi064:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-24T12:02:55.951 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-24T12:02:55.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:02:55.953 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T12:02:55.954 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T12:02:55.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:55.986 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:55.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:56.000 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:hints = 1 2024-03-24T12:02:56.000 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T12:02:56.000 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object prefix: benchmark_data_smithi064_148274 2024-03-24T12:02:56.001 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:02:56.001 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 0 0 0 0 0 0 - 0 2024-03-24T12:02:56.088 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:56.124 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:56.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:56.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:56.262 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:56.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:56.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:56.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:56.399 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:56.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:56.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:56.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[42243]: 4.1 scrub starts 2024-03-24T12:02:56.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[42243]: 4.1 scrub ok 2024-03-24T12:02:56.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T12:02:56.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T12:02:56.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[42243]: 4.3 scrub starts 2024-03-24T12:02:56.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[42243]: 4.3 scrub ok 2024-03-24T12:02:56.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[30223]: 4.1 scrub starts 2024-03-24T12:02:56.523 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[30223]: 4.1 scrub ok 2024-03-24T12:02:56.523 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T12:02:56.523 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T12:02:56.523 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[30223]: 4.3 scrub starts 2024-03-24T12:02:56.523 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:56 smithi064 ceph-mon[30223]: 4.3 scrub ok 2024-03-24T12:02:56.541 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:56.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:56 smithi193 ceph-mon[33593]: 4.1 scrub starts 2024-03-24T12:02:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:56 smithi193 ceph-mon[33593]: 4.1 scrub ok 2024-03-24T12:02:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:56 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T12:02:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:56 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T12:02:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:56 smithi193 ceph-mon[33593]: 4.3 scrub starts 2024-03-24T12:02:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:56 smithi193 ceph-mon[33593]: 4.3 scrub ok 2024-03-24T12:02:56.643 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:56.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:56.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:56.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:56.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:56.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:56.956 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:56.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:57.001 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 1 16 3440 3424 213.952 214 0.00367108 0.00465782 2024-03-24T12:02:57.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:02:57.096 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:57.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:02:57.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:57.234 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:57.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:57.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:57.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:57 smithi064 ceph-mon[30223]: pgmap v473: 158 pgs: 158 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-24T12:02:57.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:57 smithi064 ceph-mon[30223]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T12:02:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:57 smithi064 ceph-mon[42243]: pgmap v473: 158 pgs: 158 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-24T12:02:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:57 smithi064 ceph-mon[42243]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T12:02:57.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:57.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:57.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:57.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:02:57.513 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:57.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:57.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:02:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:02:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:57 smithi193 ceph-mon[33593]: pgmap v473: 158 pgs: 158 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-24T12:02:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:57 smithi193 ceph-mon[33593]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T12:02:57.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:57.653 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:02:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:02:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:57.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:57.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:57.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:02:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:57.801 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:57.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:57.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:57.938 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:57.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:58.001 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 2 16 6727 6711 209.68 205.438 0.00546913 0.00475839 2024-03-24T12:02:58.039 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:58.074 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:58.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:58.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:58.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:02:58.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[30223]: 6.7 scrub starts 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[30223]: 6.7 scrub ok 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T12:02:58.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[42243]: 6.7 scrub starts 2024-03-24T12:02:58.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:58 smithi064 ceph-mon[42243]: 6.7 scrub ok 2024-03-24T12:02:58.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:02:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:58.214 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:58.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:02:58.315 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:02:58.352 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:58.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:02:58.454 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:02:58.506 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:58.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:02:58.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:58 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:02:58.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:58 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:02:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:58 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T12:02:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:58 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T12:02:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:58 smithi193 ceph-mon[33593]: 6.7 scrub starts 2024-03-24T12:02:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:58 smithi193 ceph-mon[33593]: 6.7 scrub ok 2024-03-24T12:02:58.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:58.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:58.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:58.771 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:58.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:58.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:58.905 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:58.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:58.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:58.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:02:59.001 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 3 16 9614 9598 199.927 180.438 0.00530993 0.00499428 2024-03-24T12:02:59.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:02:59.051 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:59.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:02:59.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:02:59.186 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:59.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:02:59.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: pgmap v475: 158 pgs: 158 active+clean; 158 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 26 MiB/s wr, 421 op/s 2024-03-24T12:02:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: pgmap v475: 158 pgs: 158 active+clean; 158 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 26 MiB/s wr, 421 op/s 2024-03-24T12:02:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T12:02:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:02:59 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T12:02:59.323 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:59.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:59.425 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:59.461 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:59.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:59.563 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:02:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: pgmap v475: 158 pgs: 158 active+clean; 158 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 26 MiB/s wr, 421 op/s 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T12:02:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:02:59 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T12:02:59.598 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:02:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:59.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:02:59.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:59.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:59.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:59.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:02:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:02:59.886 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:02:59.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:02:59.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:00.002 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 4 16 12551 12535 195.786 183.562 0.00889107 0.00509801 2024-03-24T12:03:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:00.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:00.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:00.167 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:00 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:03:00.167 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:00 smithi064 ceph-mon[30223]: 5.7 scrub starts 2024-03-24T12:03:00.167 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:00 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:03:00.167 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:00 smithi064 ceph-mon[30223]: 5.7 scrub ok 2024-03-24T12:03:00.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:00.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:00.273 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:03:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:03:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:67040 7.a 7:5dc0d72f:::benchmark_data_smithi064_87013_object67039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:55:13.861215+0000", 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 466.28042555600001, 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18947440800000001, 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 67040 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.861215+0000", 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.861215+0000", 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.861215+0000", 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8800000000000002e-07 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.861234+0000", 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8321e-05 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.861240+0000", 2024-03-24T12:03:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9789999999999998e-06 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.861245+0000", 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.271e-06 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.865739+0000", 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044942369999999999 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.865843+0000", 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010425 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.865973+0000", 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012968800000000001 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.871757+0000", 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057835849999999999 2024-03-24T12:03:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.874215+0000", 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002458263 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:13.876407+0000", 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021921250000000001 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.050635+0000", 2024-03-24T12:03:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17422832699999999 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:55:14.050689+0000", 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3674000000000001e-05 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:133507 7.2 7:40491140:::benchmark_data_smithi064_87013_object12550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:18.821548+0000", 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 401.32009230800003, 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.191760445, 2024-03-24T12:03:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 133507 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821548+0000", 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821548+0000", 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821550+0000", 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.198e-06 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821556+0000", 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999998e-06 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821568+0000", 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1826e-05 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821584+0000", 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6300999999999999e-05 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.834084+0000", 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012499494 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.834191+0000", 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107308 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.834304+0000", 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113381 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.007991+0000", 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.173686811 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.013196+0000", 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052047880000000001 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.013208+0000", 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2132e-05 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.013259+0000", 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1023999999999998e-05 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.013308+0000", 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9141999999999998e-05 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:133508 7.a 7:598afffb:::benchmark_data_smithi064_87013_object12551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:18.821939+0000", 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 401.31970139800001, 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20546225100000001, 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 133508 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821939+0000", 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821939+0000", 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821940+0000", 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2270000000000001e-06 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821945+0000", 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7199999999999997e-06 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821955+0000", 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9909999999999997e-06 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.821962+0000", 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1709999999999996e-06 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.834404+0000", 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012442627 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.834492+0000", 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7788999999999996e-05 2024-03-24T12:03:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.834559+0000", 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6481000000000003e-05 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837074+0000", 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025149679999999998 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.008030+0000", 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17095650600000001 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.027285+0000", 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019254878999999999 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.027335+0000", 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0146999999999999e-05 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.027401+0000", 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5745000000000002e-05 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:133521 7.a 7:5b6055f4:::benchmark_data_smithi064_87013_object12564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:18.831959+0000", 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 401.30968153399999, 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195540769, 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 133521 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.831959+0000", 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.831959+0000", 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.831960+0000", 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.674e-06 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.831966+0000", 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5040000000000002e-06 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.831977+0000", 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1583000000000001e-05 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.831986+0000", 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5699999999999993e-06 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837118+0000", 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051324880000000002 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837231+0000", 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112963 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837346+0000", 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011481600000000001 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.007897+0000", 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17055082999999999 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.008102+0000", 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020519700000000001 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.027410+0000", 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019307571999999999 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.027450+0000", 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9928999999999997e-05 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.027499+0000", 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9642999999999999e-05 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:133525 7.a 7:5abafe5c:::benchmark_data_smithi064_87013_object12568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:56:18.837096+0000", 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 401.304544378, 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19184396300000001, 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 133525 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837096+0000", 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837096+0000", 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837098+0000", 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.863e-06 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837103+0000", 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.045e-06 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837113+0000", 2024-03-24T12:03:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0791999999999999e-05 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:18.837121+0000", 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4909999999999999e-06 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.009531+0000", 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.172410546 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.009659+0000", 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127293 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.009781+0000", 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121755 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.015369+0000", 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055881020000000002 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.021216+0000", 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058469100000000003 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.028804+0000", 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075879809999999997 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.028891+0000", 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7676000000000006e-05 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:56:19.028940+0000", 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8508999999999999e-05 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215971 7.11 7:8c040365:::benchmark_data_smithi064_87013_object95014:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:45.007482+0000", 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 315.13415779799999, 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99587450600000005, 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T12:03:00.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215971 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007482+0000", 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007482+0000", 2024-03-24T12:03:00.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007484+0000", 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007488+0000", 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.895e-06 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007497+0000", 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007506+0000", 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9309999999999997e-06 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009756+0000", 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002250499 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009765+0000", 2024-03-24T12:03:00.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.738e-06 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.003219+0000", 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99345422900000002 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.003357+0000", 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013740399999999999 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:216013 7.11 7:8df42d58:::benchmark_data_smithi064_87013_object95056:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:45.007629+0000", 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 315.13401072900001, 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.001424466, 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216013 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007629+0000", 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007629+0000", 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007630+0000", 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007635+0000", 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1359999999999999e-06 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007643+0000", 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.303e-06 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007652+0000", 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7870000000000007e-06 2024-03-24T12:03:00.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009789+0000", 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002137417 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009796+0000", 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1180000000000003e-06 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.003646+0000", 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99385028799999997 2024-03-24T12:03:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.003784+0000", 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013761099999999999 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.004116+0000", 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033185499999999999 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.006002+0000", 2024-03-24T12:03:00.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001885821 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.006098+0000", 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6527000000000006e-05 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.006135+0000", 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6325999999999997e-05 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.008533+0000", 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00239822 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.008925+0000", 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039180900000000002 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.009054+0000", 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012919400000000001 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221871 7.14 7:2e392a63:::benchmark_data_smithi064_87013_object100914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.055097+0000", 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 307.086543603, 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002813153, 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221871 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055097+0000", 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055097+0000", 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055099+0000", 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.813e-06 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055106+0000", 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8170000000000001e-06 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055120+0000", 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3472000000000001e-05 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055130+0000", 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0088000000000001e-05 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055217+0000", 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7029999999999999e-05 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055227+0000", 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0217999999999999e-05 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.053733+0000", 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99850598199999996 2024-03-24T12:03:00.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.053837+0000", 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010372100000000001 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.054103+0000", 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026637299999999998 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057656+0000", 2024-03-24T12:03:00.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003553301 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057669+0000", 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2533e-05 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057808+0000", 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013899800000000001 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057881+0000", 2024-03-24T12:03:00.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3225000000000004e-05 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057910+0000", 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8581999999999998e-05 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221882 7.14 7:2c6e9666:::benchmark_data_smithi064_87013_object100925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.070802+0000", 2024-03-24T12:03:00.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 307.07083856399998, 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.987318433, 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221882 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070802+0000", 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070802+0000", 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070804+0000", 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.824e-06 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070813+0000", 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6070000000000008e-06 2024-03-24T12:03:00.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070832+0000", 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9219999999999999e-05 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070843+0000", 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0601999999999999e-05 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070915+0000", 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2408000000000002e-05 2024-03-24T12:03:00.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070925+0000", 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5270000000000004e-06 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.055439+0000", 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98451412999999999 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.055547+0000", 2024-03-24T12:03:00.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108251 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.055674+0000", 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012672399999999999 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057999+0000", 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023249820000000002 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058019+0000", 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0438999999999999e-05 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058045+0000", 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5896000000000001e-05 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058090+0000", 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4468999999999997e-05 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058120+0000", 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0354000000000001e-05 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221955 7.14 7:2cc02cd6:::benchmark_data_smithi064_87013_object100998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.134401+0000", 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 307.00723961099999, 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.92390937799999995, 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221955 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134401+0000", 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134401+0000", 2024-03-24T12:03:00.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134402+0000", 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.711e-06 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134407+0000", 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9579999999999998e-06 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134424+0000", 2024-03-24T12:03:00.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6609e-05 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134432+0000", 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2619999999999998e-06 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134491+0000", 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8668999999999997e-05 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134499+0000", 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1810000000000001e-06 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.056347+0000", 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.92184798899999998 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.056432+0000", 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4890999999999997e-05 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.056642+0000", 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021018600000000001 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058140+0000", 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001497623 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058181+0000", 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1705999999999997e-05 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058244+0000", 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2739999999999999e-05 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058273+0000", 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9241999999999999e-05 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058310+0000", 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6610999999999999e-05 2024-03-24T12:03:00.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221994 7.14 7:2d64b07a:::benchmark_data_smithi064_87013_object101037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.166370+0000", 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 306.97526965600002, 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.89684323600000004, 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221994 2024-03-24T12:03:00.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166370+0000", 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166370+0000", 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166373+0000", 2024-03-24T12:03:00.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.571e-06 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166379+0000", 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6540000000000004e-06 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166394+0000", 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5044e-05 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166403+0000", 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9129999999999992e-06 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166581+0000", 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868100000000001 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166591+0000", 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3710000000000006e-06 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058358+0000", 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.89176743599999997 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058420+0000", 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2348999999999995e-05 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058552+0000", 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131366 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.061732+0000", 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031802409999999999 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063030+0000", 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012981259999999999 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063110+0000", 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9950999999999994e-05 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063167+0000", 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6508999999999997e-05 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063214+0000", 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7024000000000002e-05 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:222042 7.14 7:2868e145:::benchmark_data_smithi064_87013_object101085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.204936+0000", 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 306.93670443600001, 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86118150800000004, 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 222042 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204936+0000", 2024-03-24T12:03:00.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204936+0000", 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204937+0000", 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4610000000000001e-06 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204942+0000", 2024-03-24T12:03:00.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8960000000000002e-06 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204954+0000", 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2234e-05 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204962+0000", 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8089999999999993e-06 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.205016+0000", 2024-03-24T12:03:00.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3952000000000001e-05 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.205024+0000", 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6279999999999996e-06 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.059312+0000", 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.85428786700000003 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.059427+0000", 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115158 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.059608+0000", 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000181392 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063072+0000", 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034640309999999998 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.064297+0000", 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012244879999999999 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066020+0000", 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017236599999999999 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066074+0000", 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4057000000000001e-05 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066117+0000", 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2874999999999999e-05 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:222090 7.14 7:2e48c8aa:::benchmark_data_smithi064_87013_object101133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.259979+0000", 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 306.88166090700003, 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.80658173, 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 222090 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.259979+0000", 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.259979+0000", 2024-03-24T12:03:00.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.259981+0000", 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4980000000000001e-06 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.259985+0000", 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6929999999999998e-06 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.259996+0000", 2024-03-24T12:03:00.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.013e-05 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.260004+0000", 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2379999999999997e-06 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.260051+0000", 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7080999999999999e-05 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.260058+0000", 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0110000000000003e-06 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.060523+0000", 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.80046491500000005 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.060620+0000", 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7183999999999997e-05 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.060788+0000", 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016815600000000001 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.064385+0000", 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035966710000000001 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066125+0000", 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017401059999999999 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066470+0000", 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034475599999999999 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066521+0000", 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0968999999999999e-05 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066561+0000", 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0321999999999998e-05 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:222108 7.14 7:2826e131:::benchmark_data_smithi064_87013_object101151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.303713+0000", 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 306.83792729100003, 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.76309789900000002, 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 222108 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.356 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.303713+0000", 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.303713+0000", 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.303715+0000", 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0219999999999999e-06 2024-03-24T12:03:00.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.303721+0000", 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0190000000000003e-06 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.303735+0000", 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4579999999999999e-05 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.303743+0000", 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4599999999999997e-06 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.303795+0000", 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2055000000000002e-05 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.303803+0000", 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9500000000000001e-06 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.061764+0000", 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.75796084200000002 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.061874+0000", 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011053299999999999 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.062058+0000", 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000184117 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.064418+0000", 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023591910000000001 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066132+0000", 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017139480000000001 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066735+0000", 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060329199999999996 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066778+0000", 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2920000000000002e-05 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066811+0000", 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2969999999999998e-05 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104600 8.18 8:195e728d:::benchmark_data_smithi064_120708_object104599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:03:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.382184916, 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0728234670000001, 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104600 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759458+0000", 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6309999999999999e-06 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759492+0000", 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4329000000000003e-05 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759509+0000", 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6705e-05 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759520+0000", 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.111e-05 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759583+0000", 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3483000000000002e-05 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759592+0000", 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804373+0000", 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0447811650000001 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804557+0000", 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018477399999999999 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804989+0000", 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043201600000000002 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809404+0000", 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044143280000000003 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813653+0000", 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042489600000000004 2024-03-24T12:03:00.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.815161+0000", 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001507911 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831615+0000", 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016453925000000001 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831679+0000", 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4657000000000001e-05 2024-03-24T12:03:00.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832279+0000", 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059939900000000003 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104606 8.18 8:1f771a39:::benchmark_data_smithi064_120708_object104605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.38209946200001, 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.073005889, 2024-03-24T12:03:00.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104606 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759564+0000", 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2459999999999998e-05 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759572+0000", 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8399999999999995e-06 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759578+0000", 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5830000000000003e-06 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759626+0000", 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7576e-05 2024-03-24T12:03:00.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759632+0000", 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4500000000000001e-06 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.806996+0000", 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0473636369999999 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.807166+0000", 2024-03-24T12:03:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170104 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.807568+0000", 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000401455 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813591+0000", 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006023919 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.815217+0000", 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016253210000000001 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831571+0000", 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016353873000000001 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832375+0000", 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080415300000000005 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832441+0000", 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5860999999999998e-05 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832547+0000", 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010589799999999999 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104642 8.18 8:1b5ea0e5:::benchmark_data_smithi064_120708_object104641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:03:00.376 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.38205213399999, 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.073407242, 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104642 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759589+0000", 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759610+0000", 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0948e-05 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759616+0000", 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8589999999999999e-06 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759622+0000", 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9519999999999999e-06 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759649+0000", 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6659999999999999e-05 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759653+0000", 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9419999999999997e-06 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809440+0000", 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0497873879999999 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809603+0000", 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016285099999999999 2024-03-24T12:03:00.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.810439+0000", 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083558799999999996 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813599+0000", 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031601060000000002 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831369+0000", 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017770563 2024-03-24T12:03:00.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832331+0000", 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096128100000000003 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832805+0000", 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000473864 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832873+0000", 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8516999999999995e-05 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832995+0000", 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122139 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:168642 8.16 8:6b5fdd3f:::benchmark_data_smithi064_120708_object46616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:01:20.969513+0000", 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.172127224999997, 2024-03-24T12:03:00.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19184692, 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 168642 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:20.969513+0000", 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:20.969513+0000", 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:20.969514+0000", 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.48e-06 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:20.969519+0000", 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.933e-06 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:20.969531+0000", 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1219e-05 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:20.969538+0000", 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6380000000000004e-06 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:21.036207+0000", 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066668850000000002 2024-03-24T12:03:00.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:21.036339+0000", 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:21.036850+0000", 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00051130999999999995 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:21.042807+0000", 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059563610000000003 2024-03-24T12:03:00.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:21.042848+0000", 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1733999999999997e-05 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:21.161198+0000", 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11834937700000001 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:21.161297+0000", 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9098000000000006e-05 2024-03-24T12:03:00.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:21.161360+0000", 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3094000000000003e-05 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:172847 8.16 8:6ba49125:::benchmark_data_smithi064_120708_object50821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:01:24.679604+0000", 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.462036138000002, 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.24031903399999999, 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 172847 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.679604+0000", 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.679604+0000", 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.679605+0000", 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7800000000000001e-07 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.679608+0000", 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.298e-06 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.679615+0000", 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6239999999999996e-06 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.679620+0000", 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5260000000000001e-06 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.810922+0000", 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13130178100000001 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.811029+0000", 2024-03-24T12:03:00.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106543 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.811399+0000", 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037091400000000002 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.816674+0000", 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005274824 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.825430+0000", 2024-03-24T12:03:00.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087555270000000008 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.919791+0000", 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094361433999999994 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.919853+0000", 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2030000000000001e-05 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:24.919923+0000", 2024-03-24T12:03:00.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9754999999999999e-05 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:175614 8.16 8:68a27289:::benchmark_data_smithi064_120708_object53588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:01:27.419640+0000", 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.721999840999999, 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.208653648, 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:00.395 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 175614 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.419640+0000", 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.419640+0000", 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:03:00.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.419641+0000", 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.419643+0000", 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1959999999999998e-06 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.419648+0000", 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.921e-06 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.419653+0000", 2024-03-24T12:03:00.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.101e-06 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.580016+0000", 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.160363011 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.584516+0000", 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004500034 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.584894+0000", 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037828100000000001 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.589201+0000", 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004307178 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.593125+0000", 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039230690000000004 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:00.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.627954+0000", 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034829210999999999 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.628008+0000", 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4440000000000001e-05 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:01:27.628294+0000", 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028575699999999998 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:03:00.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:03:00.401 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:03:00.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:00.438 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:00.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:00.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:00.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:00 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:03:00.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:00 smithi064 ceph-mon[42243]: 5.7 scrub starts 2024-03-24T12:03:00.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:00 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:03:00.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:00 smithi064 ceph-mon[42243]: 5.7 scrub ok 2024-03-24T12:03:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:00.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:00 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:03:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:00 smithi193 ceph-mon[33593]: 5.7 scrub starts 2024-03-24T12:03:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:00 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:03:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:00 smithi193 ceph-mon[33593]: 5.7 scrub ok 2024-03-24T12:03:00.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:03:00.683 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:03:00.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:00.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:00.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:00.867 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:00.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:00.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:01.002 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 5 16 15145 15129 189.053 162.125 0.00520055 0.00528183 2024-03-24T12:03:01.008 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:01.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:01.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:01.148 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:01.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:01.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:01.287 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:01.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:01.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:01.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: pgmap v476: 158 pgs: 158 active+clean; 158 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 22 MiB/s wr, 349 op/s 2024-03-24T12:03:01.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:03:01.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:03:01.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T12:03:01.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T12:03:01.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: 4.11 scrub starts 2024-03-24T12:03:01.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: from='client.25324 v1:172.21.15.64:0/1815217925' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: pgmap v476: 158 pgs: 158 active+clean; 158 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 22 MiB/s wr, 349 op/s 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: 4.11 scrub starts 2024-03-24T12:03:01.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: from='client.25324 v1:172.21.15.64:0/1815217925' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:03:01.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T12:03:01.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:01 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T12:03:01.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:03:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:01.422 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:01.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: pgmap v476: 158 pgs: 158 active+clean; 158 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 22 MiB/s wr, 349 op/s 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: 4.11 scrub starts 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: from='client.25324 v1:172.21.15.64:0/1815217925' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T12:03:01.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:01 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T12:03:01.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:01.558 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:01.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:01.653 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:01.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:01.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:01.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:01.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:01.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:01.826 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:01.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:01.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:01.971 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:01.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:02.002 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 6 16 17509 17493 182.169 147.75 0.00384308 0.0054837 2024-03-24T12:03:02.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:02.111 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:02.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:02.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:02.214 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:02.237 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:02.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:02.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:02 smithi193 ceph-mon[33593]: 4.11 scrub ok 2024-03-24T12:03:02.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:02 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T12:03:02.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:02 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T12:03:02.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:02 smithi193 ceph-mon[33593]: pgmap v477: 158 pgs: 158 active+clean; 454 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 622 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-24T12:03:02.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:02 smithi193 ceph-mon[33593]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T12:03:02.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:02.366 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[30223]: 4.11 scrub ok 2024-03-24T12:03:02.366 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T12:03:02.366 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T12:03:02.366 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[30223]: pgmap v477: 158 pgs: 158 active+clean; 454 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 622 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-24T12:03:02.366 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[30223]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T12:03:02.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[42243]: 4.11 scrub ok 2024-03-24T12:03:02.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T12:03:02.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T12:03:02.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[42243]: pgmap v477: 158 pgs: 158 active+clean; 454 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 622 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-24T12:03:02.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:02 smithi064 ceph-mon[42243]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T12:03:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:02.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:02.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:02.422 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-03-24T12:02:59.334026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165812574,"num_objects":2719,"num_object_clones":0,"num_object_copies":8157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":1367,"num_read_kb":1119,"num_write":3243,"num_write_kb":161389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4804,"ondisk_log_size":4804,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3229352,"kb_used_data":190876,"kb_used_omap":12,"kb_used_meta":3038387,"kb_avail":746731864,"statfs":{"total":767960285184,"available":764653428736,"internally_reserved":0,"allocated":195457024,"data_stored":192064442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3111308848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,5,9,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2521,"num_write_kb":161344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.236616"},"pg_stats":[{"pgid":"9.5","version":"252'272","reported_seq":294,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:57.091002+0000","last_change":"2024-03-24T12:02:46.086109+0000","last_active":"2024-03-24T12:02:57.091002+0000","last_peered":"2024-03-24T12:02:57.091002+0000","last_clean":"2024-03-24T12:02:57.091002+0000","last_became_active":"2024-03-24T12:02:46.085796+0000","last_became_peered":"2024-03-24T12:02:46.085796+0000","last_unstale":"2024-03-24T12:02:57.091002+0000","last_undegraded":"2024-03-24T12:02:57.091002+0000","last_fullsized":"2024-03-24T12:02:57.091002+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:08.045160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":552,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690310+0000","last_change":"2024-03-24T12:01:58.038265+0000","last_active":"2024-03-24T12:02:53.690310+0000","last_peered":"2024-03-24T12:02:53.690310+0000","last_clean":"2024-03-24T12:02:53.690310+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:02:53.690310+0000","last_undegraded":"2024-03-24T12:02:53.690310+0000","last_fullsized":"2024-03-24T12:02:53.690310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:01:58.038146+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T11:59:08.941456+0000","last_clean_scrub_stamp":"2024-03-24T12:01:58.038146+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:25.343573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":452,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.735696+0000","last_change":"2024-03-24T11:59:59.960767+0000","last_active":"2024-03-24T12:02:53.735696+0000","last_peered":"2024-03-24T12:02:53.735696+0000","last_clean":"2024-03-24T12:02:53.735696+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T12:02:53.735696+0000","last_undegraded":"2024-03-24T12:02:53.735696+0000","last_fullsized":"2024-03-24T12:02:53.735696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T11:59:59.957817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:59.957817+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811669+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":425,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.815241+0000","last_change":"2024-03-24T12:00:08.247939+0000","last_active":"2024-03-24T12:02:56.815241+0000","last_peered":"2024-03-24T12:02:56.815241+0000","last_clean":"2024-03-24T12:02:56.815241+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T12:02:56.815241+0000","last_undegraded":"2024-03-24T12:02:56.815241+0000","last_fullsized":"2024-03-24T12:02:56.815241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:08.247807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:00:08.247807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":401,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736075+0000","last_change":"2024-03-24T12:02:01.760487+0000","last_active":"2024-03-24T12:02:53.736075+0000","last_peered":"2024-03-24T12:02:53.736075+0000","last_clean":"2024-03-24T12:02:53.736075+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T12:02:53.736075+0000","last_undegraded":"2024-03-24T12:02:53.736075+0000","last_fullsized":"2024-03-24T12:02:53.736075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:01.760400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:02:01.760400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:02.738660+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"251'97","reported_seq":117,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.360745+0000","last_change":"2024-03-24T12:02:46.438340+0000","last_active":"2024-03-24T12:02:56.360745+0000","last_peered":"2024-03-24T12:02:56.360745+0000","last_clean":"2024-03-24T12:02:56.360745+0000","last_became_active":"2024-03-24T12:02:46.437833+0000","last_became_peered":"2024-03-24T12:02:46.437833+0000","last_unstale":"2024-03-24T12:02:56.360745+0000","last_undegraded":"2024-03-24T12:02:56.360745+0000","last_fullsized":"2024-03-24T12:02:56.360745+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:08.365804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":437,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806665+0000","last_change":"2024-03-24T12:02:09.783798+0000","last_active":"2024-03-24T12:02:56.806665+0000","last_peered":"2024-03-24T12:02:56.806665+0000","last_clean":"2024-03-24T12:02:56.806665+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:02:56.806665+0000","last_undegraded":"2024-03-24T12:02:56.806665+0000","last_fullsized":"2024-03-24T12:02:56.806665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:02:09.783600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.783600+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:26.180718+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":430,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.462883+0000","last_change":"2024-03-24T12:02:56.462921+0000","last_active":"2024-03-24T12:02:56.462883+0000","last_peered":"2024-03-24T12:02:56.462883+0000","last_clean":"2024-03-24T12:02:56.462883+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:02:56.462883+0000","last_undegraded":"2024-03-24T12:02:56.462883+0000","last_fullsized":"2024-03-24T12:02:56.462883+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:56.462783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T12:02:56.462783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:09.321939+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":417,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.814726+0000","last_change":"2024-03-24T11:59:53.094141+0000","last_active":"2024-03-24T12:02:56.814726+0000","last_peered":"2024-03-24T12:02:56.814726+0000","last_clean":"2024-03-24T12:02:56.814726+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T12:02:56.814726+0000","last_undegraded":"2024-03-24T12:02:56.814726+0000","last_fullsized":"2024-03-24T12:02:56.814726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:53.093972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:53.093972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":396,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808306+0000","last_change":"2024-03-24T11:59:59.524343+0000","last_active":"2024-03-24T12:02:56.808306+0000","last_peered":"2024-03-24T12:02:56.808306+0000","last_clean":"2024-03-24T12:02:56.808306+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T12:02:56.808306+0000","last_undegraded":"2024-03-24T12:02:56.808306+0000","last_fullsized":"2024-03-24T12:02:56.808306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:59.523952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:59.523952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"251'88","reported_seq":108,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.360723+0000","last_change":"2024-03-24T12:02:46.089672+0000","last_active":"2024-03-24T12:02:56.360723+0000","last_peered":"2024-03-24T12:02:56.360723+0000","last_clean":"2024-03-24T12:02:56.360723+0000","last_became_active":"2024-03-24T12:02:46.088943+0000","last_became_peered":"2024-03-24T12:02:46.088943+0000","last_unstale":"2024-03-24T12:02:56.360723+0000","last_undegraded":"2024-03-24T12:02:56.360723+0000","last_fullsized":"2024-03-24T12:02:56.360723+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:45.632687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":5632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":471,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404408+0000","last_change":"2024-03-24T12:02:05.124570+0000","last_active":"2024-03-24T12:02:53.404408+0000","last_peered":"2024-03-24T12:02:53.404408+0000","last_clean":"2024-03-24T12:02:53.404408+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:02:53.404408+0000","last_undegraded":"2024-03-24T12:02:53.404408+0000","last_fullsized":"2024-03-24T12:02:53.404408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:02:05.124435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:05.124435+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:12.403233+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":425,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.816081+0000","last_change":"2024-03-24T12:00:02.219859+0000","last_active":"2024-03-24T12:02:56.816081+0000","last_peered":"2024-03-24T12:02:56.816081+0000","last_clean":"2024-03-24T12:02:56.816081+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T12:02:56.816081+0000","last_undegraded":"2024-03-24T12:02:56.816081+0000","last_fullsized":"2024-03-24T12:02:56.816081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:02.219529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:00:02.219529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":417,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.814556+0000","last_change":"2024-03-24T12:00:09.213447+0000","last_active":"2024-03-24T12:02:56.814556+0000","last_peered":"2024-03-24T12:02:56.814556+0000","last_clean":"2024-03-24T12:02:56.814556+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T12:02:56.814556+0000","last_undegraded":"2024-03-24T12:02:56.814556+0000","last_fullsized":"2024-03-24T12:02:56.814556+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:09.213304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:00:09.213304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.689559+0000","last_change":"2024-03-24T12:02:12.944517+0000","last_active":"2024-03-24T12:02:53.689559+0000","last_peered":"2024-03-24T12:02:53.689559+0000","last_clean":"2024-03-24T12:02:53.689559+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T12:02:53.689559+0000","last_undegraded":"2024-03-24T12:02:53.689559+0000","last_fullsized":"2024-03-24T12:02:53.689559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:12.944172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:02:12.944172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:16.306971+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"251'82","reported_seq":102,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.285133+0000","last_change":"2024-03-24T12:02:46.436843+0000","last_active":"2024-03-24T12:02:56.285133+0000","last_peered":"2024-03-24T12:02:56.285133+0000","last_clean":"2024-03-24T12:02:56.285133+0000","last_became_active":"2024-03-24T12:02:46.436511+0000","last_became_peered":"2024-03-24T12:02:46.436511+0000","last_unstale":"2024-03-24T12:02:56.285133+0000","last_undegraded":"2024-03-24T12:02:56.285133+0000","last_fullsized":"2024-03-24T12:02:56.285133+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:05.849959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373952,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":474,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.111020+0000","last_change":"2024-03-24T12:01:21.467893+0000","last_active":"2024-03-24T12:02:53.111020+0000","last_peered":"2024-03-24T12:02:53.111020+0000","last_clean":"2024-03-24T12:02:53.111020+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T12:02:53.111020+0000","last_undegraded":"2024-03-24T12:02:53.111020+0000","last_fullsized":"2024-03-24T12:02:53.111020+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'25","last_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:01:21.467704+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858490+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"67'1","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.814134+0000","last_change":"2024-03-24T12:02:11.033710+0000","last_active":"2024-03-24T12:02:56.814134+0000","last_peered":"2024-03-24T12:02:56.814134+0000","last_clean":"2024-03-24T12:02:56.814134+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T12:02:56.814134+0000","last_undegraded":"2024-03-24T12:02:56.814134+0000","last_fullsized":"2024-03-24T12:02:56.814134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:02:11.033556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:11.033556+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:12.977384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":166,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110492+0000","last_change":"2024-03-24T12:02:07.910245+0000","last_active":"2024-03-24T12:02:53.110492+0000","last_peered":"2024-03-24T12:02:53.110492+0000","last_clean":"2024-03-24T12:02:53.110492+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:02:53.110492+0000","last_undegraded":"2024-03-24T12:02:53.110492+0000","last_fullsized":"2024-03-24T12:02:53.110492+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:07.910126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:07.910126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:20.578073+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":413,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.814933+0000","last_change":"2024-03-24T12:02:09.089126+0000","last_active":"2024-03-24T12:02:56.814933+0000","last_peered":"2024-03-24T12:02:56.814933+0000","last_clean":"2024-03-24T12:02:56.814933+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T12:02:56.814933+0000","last_undegraded":"2024-03-24T12:02:56.814933+0000","last_fullsized":"2024-03-24T12:02:56.814933+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:09.088728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.088728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:09.196518+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":441,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.735935+0000","last_change":"2024-03-24T11:59:56.028640+0000","last_active":"2024-03-24T12:02:53.735935+0000","last_peered":"2024-03-24T12:02:53.735935+0000","last_clean":"2024-03-24T12:02:53.735935+0000","last_became_active":"2024-03-24T11:59:53.779445+0000","last_became_peered":"2024-03-24T11:59:53.779445+0000","last_unstale":"2024-03-24T12:02:53.735935+0000","last_undegraded":"2024-03-24T12:02:53.735935+0000","last_fullsized":"2024-03-24T12:02:53.735935+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:56.028559+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:56.028559+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811731+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"252'258","reported_seq":280,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:57.293224+0000","last_change":"2024-03-24T12:02:46.091030+0000","last_active":"2024-03-24T12:02:57.293224+0000","last_peered":"2024-03-24T12:02:57.293224+0000","last_clean":"2024-03-24T12:02:57.293224+0000","last_became_active":"2024-03-24T12:02:46.090706+0000","last_became_peered":"2024-03-24T12:02:46.090706+0000","last_unstale":"2024-03-24T12:02:57.293224+0000","last_undegraded":"2024-03-24T12:02:57.293224+0000","last_fullsized":"2024-03-24T12:02:57.293224+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:14.164422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":16512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":452,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110819+0000","last_change":"2024-03-24T12:02:04.883385+0000","last_active":"2024-03-24T12:02:53.110819+0000","last_peered":"2024-03-24T12:02:53.110819+0000","last_clean":"2024-03-24T12:02:53.110819+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:02:53.110819+0000","last_undegraded":"2024-03-24T12:02:53.110819+0000","last_fullsized":"2024-03-24T12:02:53.110819+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'15","last_scrub_stamp":"2024-03-24T12:02:04.883290+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:02:04.883290+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:31.402251+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":167,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806967+0000","last_change":"2024-03-24T12:02:08.418375+0000","last_active":"2024-03-24T12:02:56.806967+0000","last_peered":"2024-03-24T12:02:56.806967+0000","last_clean":"2024-03-24T12:02:56.806967+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:02:56.806967+0000","last_undegraded":"2024-03-24T12:02:56.806967+0000","last_fullsized":"2024-03-24T12:02:56.806967+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:08.418230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:51.949009+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.418230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:11.632607+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":554,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.815030+0000","last_change":"2024-03-24T11:59:54.146579+0000","last_active":"2024-03-24T12:02:56.815030+0000","last_peered":"2024-03-24T12:02:56.815030+0000","last_clean":"2024-03-24T12:02:56.815030+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T12:02:56.815030+0000","last_undegraded":"2024-03-24T12:02:56.815030+0000","last_fullsized":"2024-03-24T12:02:56.815030+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:54.146359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.146359+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.111130+0000","last_change":"2024-03-24T12:00:01.672184+0000","last_active":"2024-03-24T12:02:53.111130+0000","last_peered":"2024-03-24T12:02:53.111130+0000","last_clean":"2024-03-24T12:02:53.111130+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T12:02:53.111130+0000","last_undegraded":"2024-03-24T12:02:53.111130+0000","last_fullsized":"2024-03-24T12:02:53.111130+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:00:01.672059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:00:01.672059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858456+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":694,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.689982+0000","last_change":"2024-03-24T12:02:27.917194+0000","last_active":"2024-03-24T12:02:53.689982+0000","last_peered":"2024-03-24T12:02:53.689982+0000","last_clean":"2024-03-24T12:02:53.689982+0000","last_became_active":"2024-03-24T11:59:53.771452+0000","last_became_peered":"2024-03-24T11:59:53.771452+0000","last_unstale":"2024-03-24T12:02:53.689982+0000","last_undegraded":"2024-03-24T12:02:53.689982+0000","last_fullsized":"2024-03-24T12:02:53.689982+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:02:27.917096+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:02:27.917096+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:31.018144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"252'297","reported_seq":319,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:57.298122+0000","last_change":"2024-03-24T12:02:46.437086+0000","last_active":"2024-03-24T12:02:57.298122+0000","last_peered":"2024-03-24T12:02:57.298122+0000","last_clean":"2024-03-24T12:02:57.298122+0000","last_became_active":"2024-03-24T12:02:46.436446+0000","last_became_peered":"2024-03-24T12:02:46.436446+0000","last_unstale":"2024-03-24T12:02:57.298122+0000","last_undegraded":"2024-03-24T12:02:57.298122+0000","last_fullsized":"2024-03-24T12:02:57.298122+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:04.242386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":297,"num_object_clones":0,"num_object_copies":891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":297,"num_write_kb":19008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":468,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.424073+0000","last_change":"2024-03-24T12:02:06.275179+0000","last_active":"2024-03-24T12:02:53.424073+0000","last_peered":"2024-03-24T12:02:53.424073+0000","last_clean":"2024-03-24T12:02:53.424073+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:02:53.424073+0000","last_undegraded":"2024-03-24T12:02:53.424073+0000","last_fullsized":"2024-03-24T12:02:53.424073+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'22","last_scrub_stamp":"2024-03-24T12:02:06.275037+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:02:06.275037+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:28.707471+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.423216+0000","last_change":"2024-03-24T12:02:09.327241+0000","last_active":"2024-03-24T12:02:53.423216+0000","last_peered":"2024-03-24T12:02:53.423216+0000","last_clean":"2024-03-24T12:02:53.423216+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:02:53.423216+0000","last_undegraded":"2024-03-24T12:02:53.423216+0000","last_fullsized":"2024-03-24T12:02:53.423216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:09.327128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.327128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:10.522890+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":553,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.111122+0000","last_change":"2024-03-24T12:02:09.938055+0000","last_active":"2024-03-24T12:02:53.111122+0000","last_peered":"2024-03-24T12:02:53.111122+0000","last_clean":"2024-03-24T12:02:53.111122+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T12:02:53.111122+0000","last_undegraded":"2024-03-24T12:02:53.111122+0000","last_fullsized":"2024-03-24T12:02:53.111122+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:02:09.937913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.937913+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:37.870968+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":409,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.735517+0000","last_change":"2024-03-24T12:02:09.785009+0000","last_active":"2024-03-24T12:02:53.735517+0000","last_peered":"2024-03-24T12:02:53.735517+0000","last_clean":"2024-03-24T12:02:53.735517+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T12:02:53.735517+0000","last_undegraded":"2024-03-24T12:02:53.735517+0000","last_fullsized":"2024-03-24T12:02:53.735517+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:09.784916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:02:09.784916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:18.328425+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":441,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736014+0000","last_change":"2024-03-24T11:59:53.991924+0000","last_active":"2024-03-24T12:02:53.736014+0000","last_peered":"2024-03-24T12:02:53.736014+0000","last_clean":"2024-03-24T12:02:53.736014+0000","last_became_active":"2024-03-24T11:59:53.771384+0000","last_became_peered":"2024-03-24T11:59:53.771384+0000","last_unstale":"2024-03-24T12:02:53.736014+0000","last_undegraded":"2024-03-24T12:02:53.736014+0000","last_fullsized":"2024-03-24T12:02:53.736014+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:53.991713+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:53.991713+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811653+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"251'65","reported_seq":84,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.283907+0000","last_change":"2024-03-24T12:02:46.436872+0000","last_active":"2024-03-24T12:02:56.283907+0000","last_peered":"2024-03-24T12:02:56.283907+0000","last_clean":"2024-03-24T12:02:56.283907+0000","last_became_active":"2024-03-24T12:02:46.436507+0000","last_became_peered":"2024-03-24T12:02:46.436507+0000","last_unstale":"2024-03-24T12:02:56.283907+0000","last_undegraded":"2024-03-24T12:02:56.283907+0000","last_fullsized":"2024-03-24T12:02:56.283907+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:51.278209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4259840,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":4160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":445,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110841+0000","last_change":"2024-03-24T11:59:18.653135+0000","last_active":"2024-03-24T12:02:53.110841+0000","last_peered":"2024-03-24T12:02:53.110841+0000","last_clean":"2024-03-24T12:02:53.110841+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:02:53.110841+0000","last_undegraded":"2024-03-24T12:02:53.110841+0000","last_fullsized":"2024-03-24T12:02:53.110841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'16","last_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T11:59:18.653047+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858357+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.689900+0000","last_change":"2024-03-24T12:01:21.034042+0000","last_active":"2024-03-24T12:02:53.689900+0000","last_peered":"2024-03-24T12:02:53.689900+0000","last_clean":"2024-03-24T12:02:53.689900+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:02:53.689900+0000","last_undegraded":"2024-03-24T12:02:53.689900+0000","last_fullsized":"2024-03-24T12:02:53.689900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:21.033920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:01:21.033920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.762402+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.737002+0000","last_change":"2024-03-24T11:59:52.034599+0000","last_active":"2024-03-24T12:02:53.737002+0000","last_peered":"2024-03-24T12:02:53.737002+0000","last_clean":"2024-03-24T12:02:53.737002+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T12:02:53.737002+0000","last_undegraded":"2024-03-24T12:02:53.737002+0000","last_fullsized":"2024-03-24T12:02:53.737002+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:52.034485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:52.034485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807626+0000","last_change":"2024-03-24T12:02:21.355109+0000","last_active":"2024-03-24T12:02:56.807626+0000","last_peered":"2024-03-24T12:02:56.807626+0000","last_clean":"2024-03-24T12:02:56.807626+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T12:02:56.807626+0000","last_undegraded":"2024-03-24T12:02:56.807626+0000","last_fullsized":"2024-03-24T12:02:56.807626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:21.354823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:21.354823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:30.339119+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"252'291","reported_seq":312,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:57.293581+0000","last_change":"2024-03-24T12:02:46.087611+0000","last_active":"2024-03-24T12:02:57.293581+0000","last_peered":"2024-03-24T12:02:57.293581+0000","last_clean":"2024-03-24T12:02:57.293581+0000","last_became_active":"2024-03-24T12:02:46.087133+0000","last_became_peered":"2024-03-24T12:02:46.087133+0000","last_unstale":"2024-03-24T12:02:57.293581+0000","last_undegraded":"2024-03-24T12:02:57.293581+0000","last_fullsized":"2024-03-24T12:02:57.293581+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:52.448843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":291,"num_object_clones":0,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":291,"num_write_kb":18624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":547,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807268+0000","last_change":"2024-03-24T12:02:28.396623+0000","last_active":"2024-03-24T12:02:56.807268+0000","last_peered":"2024-03-24T12:02:56.807268+0000","last_clean":"2024-03-24T12:02:56.807268+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:02:56.807268+0000","last_undegraded":"2024-03-24T12:02:56.807268+0000","last_fullsized":"2024-03-24T12:02:56.807268+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:02:28.396540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:28.396540+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:29.719970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":467,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:54.859949+0000","last_change":"2024-03-24T12:02:54.860022+0000","last_active":"2024-03-24T12:02:54.859949+0000","last_peered":"2024-03-24T12:02:54.859949+0000","last_clean":"2024-03-24T12:02:54.859949+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T12:02:54.859949+0000","last_undegraded":"2024-03-24T12:02:54.859949+0000","last_fullsized":"2024-03-24T12:02:54.859949+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:02:54.859867+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:02:54.859867+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:19.402605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":420,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404628+0000","last_change":"2024-03-24T12:00:10.038752+0000","last_active":"2024-03-24T12:02:53.404628+0000","last_peered":"2024-03-24T12:02:53.404628+0000","last_clean":"2024-03-24T12:02:53.404628+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:02:53.404628+0000","last_undegraded":"2024-03-24T12:02:53.404628+0000","last_fullsized":"2024-03-24T12:02:53.404628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:10.038672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:00:10.038672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.384423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":411,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690592+0000","last_change":"2024-03-24T12:02:02.150576+0000","last_active":"2024-03-24T12:02:53.690592+0000","last_peered":"2024-03-24T12:02:53.690592+0000","last_clean":"2024-03-24T12:02:53.690592+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T12:02:53.690592+0000","last_undegraded":"2024-03-24T12:02:53.690592+0000","last_fullsized":"2024-03-24T12:02:53.690592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:02.150459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:02.150459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:04.297232+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":443,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807845+0000","last_change":"2024-03-24T11:59:54.288619+0000","last_active":"2024-03-24T12:02:56.807845+0000","last_peered":"2024-03-24T12:02:56.807845+0000","last_clean":"2024-03-24T12:02:56.807845+0000","last_became_active":"2024-03-24T11:59:53.767670+0000","last_became_peered":"2024-03-24T11:59:53.767670+0000","last_unstale":"2024-03-24T12:02:56.807845+0000","last_undegraded":"2024-03-24T12:02:56.807845+0000","last_fullsized":"2024-03-24T12:02:56.807845+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.288488+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.288488+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"251'56","reported_seq":76,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.270360+0000","last_change":"2024-03-24T12:02:46.437141+0000","last_active":"2024-03-24T12:02:56.270360+0000","last_peered":"2024-03-24T12:02:56.270360+0000","last_clean":"2024-03-24T12:02:56.270360+0000","last_became_active":"2024-03-24T12:02:46.436562+0000","last_became_peered":"2024-03-24T12:02:46.436562+0000","last_unstale":"2024-03-24T12:02:56.270360+0000","last_undegraded":"2024-03-24T12:02:56.270360+0000","last_fullsized":"2024-03-24T12:02:56.270360+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:01.561308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3670016,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":406,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110357+0000","last_change":"2024-03-24T11:59:24.679291+0000","last_active":"2024-03-24T12:02:53.110357+0000","last_peered":"2024-03-24T12:02:53.110357+0000","last_clean":"2024-03-24T12:02:53.110357+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:02:53.110357+0000","last_undegraded":"2024-03-24T12:02:53.110357+0000","last_fullsized":"2024-03-24T12:02:53.110357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.679110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":472,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806624+0000","last_change":"2024-03-24T12:00:09.391645+0000","last_active":"2024-03-24T12:02:56.806624+0000","last_peered":"2024-03-24T12:02:56.806624+0000","last_clean":"2024-03-24T12:02:56.806624+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:02:56.806624+0000","last_undegraded":"2024-03-24T12:02:56.806624+0000","last_fullsized":"2024-03-24T12:02:56.806624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'26","last_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_deep_scrub":"195'26","last_deep_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_clean_scrub_stamp":"2024-03-24T12:00:09.391234+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":173,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404571+0000","last_change":"2024-03-24T12:02:17.226291+0000","last_active":"2024-03-24T12:02:53.404571+0000","last_peered":"2024-03-24T12:02:53.404571+0000","last_clean":"2024-03-24T12:02:53.404571+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:02:53.404571+0000","last_undegraded":"2024-03-24T12:02:53.404571+0000","last_fullsized":"2024-03-24T12:02:53.404571+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:02:17.226173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:17.226173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:45.403701+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110598+0000","last_change":"2024-03-24T11:59:55.784539+0000","last_active":"2024-03-24T12:02:53.110598+0000","last_peered":"2024-03-24T12:02:53.110598+0000","last_clean":"2024-03-24T12:02:53.110598+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T12:02:53.110598+0000","last_undegraded":"2024-03-24T12:02:53.110598+0000","last_fullsized":"2024-03-24T12:02:53.110598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:55.784390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:55.784390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858389+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":441,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110541+0000","last_change":"2024-03-24T11:59:54.754584+0000","last_active":"2024-03-24T12:02:53.110541+0000","last_peered":"2024-03-24T12:02:53.110541+0000","last_clean":"2024-03-24T12:02:53.110541+0000","last_became_active":"2024-03-24T11:59:53.763541+0000","last_became_peered":"2024-03-24T11:59:53.763541+0000","last_unstale":"2024-03-24T12:02:53.110541+0000","last_undegraded":"2024-03-24T12:02:53.110541+0000","last_fullsized":"2024-03-24T12:02:53.110541+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.754447+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.754447+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858406+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"251'48","reported_seq":68,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.274924+0000","last_change":"2024-03-24T12:02:46.438470+0000","last_active":"2024-03-24T12:02:56.274924+0000","last_peered":"2024-03-24T12:02:56.274924+0000","last_clean":"2024-03-24T12:02:56.274924+0000","last_became_active":"2024-03-24T12:02:46.438121+0000","last_became_peered":"2024-03-24T12:02:46.438121+0000","last_unstale":"2024-03-24T12:02:56.274924+0000","last_undegraded":"2024-03-24T12:02:56.274924+0000","last_fullsized":"2024-03-24T12:02:56.274924+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:55.090476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":413,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404548+0000","last_change":"2024-03-24T12:02:08.167812+0000","last_active":"2024-03-24T12:02:53.404548+0000","last_peered":"2024-03-24T12:02:53.404548+0000","last_clean":"2024-03-24T12:02:53.404548+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T12:02:53.404548+0000","last_undegraded":"2024-03-24T12:02:53.404548+0000","last_fullsized":"2024-03-24T12:02:53.404548+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:08.167536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.167536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:26.877378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"241'29","reported_seq":464,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806485+0000","last_change":"2024-03-24T12:02:55.715991+0000","last_active":"2024-03-24T12:02:56.806485+0000","last_peered":"2024-03-24T12:02:56.806485+0000","last_clean":"2024-03-24T12:02:56.806485+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T12:02:56.806485+0000","last_undegraded":"2024-03-24T12:02:56.806485+0000","last_fullsized":"2024-03-24T12:02:56.806485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-24T12:02:55.715886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:55.715886+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:16.107733+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":431,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690860+0000","last_change":"2024-03-24T12:02:23.932476+0000","last_active":"2024-03-24T12:02:53.690860+0000","last_peered":"2024-03-24T12:02:53.690860+0000","last_clean":"2024-03-24T12:02:53.690860+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:02:53.690860+0000","last_undegraded":"2024-03-24T12:02:53.690860+0000","last_fullsized":"2024-03-24T12:02:53.690860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:23.932399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:02:23.932399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:44.165936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":165,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.403922+0000","last_change":"2024-03-24T12:02:24.340511+0000","last_active":"2024-03-24T12:02:53.403922+0000","last_peered":"2024-03-24T12:02:53.403922+0000","last_clean":"2024-03-24T12:02:53.403922+0000","last_became_active":"2024-03-24T11:59:19.096322+0000","last_became_peered":"2024-03-24T11:59:19.096322+0000","last_unstale":"2024-03-24T12:02:53.403922+0000","last_undegraded":"2024-03-24T12:02:53.403922+0000","last_fullsized":"2024-03-24T12:02:53.403922+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:24.340410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:24.340410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:30.676679+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":447,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404816+0000","last_change":"2024-03-24T11:59:55.936621+0000","last_active":"2024-03-24T12:02:53.404816+0000","last_peered":"2024-03-24T12:02:53.404816+0000","last_clean":"2024-03-24T12:02:53.404816+0000","last_became_active":"2024-03-24T11:59:53.771448+0000","last_became_peered":"2024-03-24T11:59:53.771448+0000","last_unstale":"2024-03-24T12:02:53.404816+0000","last_undegraded":"2024-03-24T12:02:53.404816+0000","last_fullsized":"2024-03-24T12:02:53.404816+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:55.936510+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:55.936510+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.384500+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"252'226","reported_seq":248,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:57.091152+0000","last_change":"2024-03-24T12:02:46.088062+0000","last_active":"2024-03-24T12:02:57.091152+0000","last_peered":"2024-03-24T12:02:57.091152+0000","last_clean":"2024-03-24T12:02:57.091152+0000","last_became_active":"2024-03-24T12:02:46.087237+0000","last_became_peered":"2024-03-24T12:02:46.087237+0000","last_unstale":"2024-03-24T12:02:57.091152+0000","last_undegraded":"2024-03-24T12:02:57.091152+0000","last_fullsized":"2024-03-24T12:02:57.091152+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:59.657057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":226,"num_write_kb":14464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.737049+0000","last_change":"2024-03-24T11:59:48.136534+0000","last_active":"2024-03-24T12:02:53.737049+0000","last_peered":"2024-03-24T12:02:53.737049+0000","last_clean":"2024-03-24T12:02:53.737049+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T12:02:53.737049+0000","last_undegraded":"2024-03-24T12:02:53.737049+0000","last_fullsized":"2024-03-24T12:02:53.737049+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:48.136407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:48.136407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811618+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":454,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808352+0000","last_change":"2024-03-24T12:02:15.739122+0000","last_active":"2024-03-24T12:02:56.808352+0000","last_peered":"2024-03-24T12:02:56.808352+0000","last_clean":"2024-03-24T12:02:56.808352+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T12:02:56.808352+0000","last_undegraded":"2024-03-24T12:02:56.808352+0000","last_fullsized":"2024-03-24T12:02:56.808352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:02:15.738700+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:28.384111+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":253,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.805475+0000","last_change":"2024-03-24T11:59:20.570521+0000","last_active":"2024-03-24T12:02:56.805475+0000","last_peered":"2024-03-24T12:02:56.805475+0000","last_clean":"2024-03-24T12:02:56.805475+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:02:56.805475+0000","last_undegraded":"2024-03-24T12:02:56.805475+0000","last_fullsized":"2024-03-24T12:02:56.805475+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:20.570220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:20.570220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":424,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.809935+0000","last_change":"2024-03-24T12:02:54.700745+0000","last_active":"2024-03-24T12:02:56.809935+0000","last_peered":"2024-03-24T12:02:56.809935+0000","last_clean":"2024-03-24T12:02:56.809935+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:02:56.809935+0000","last_undegraded":"2024-03-24T12:02:56.809935+0000","last_fullsized":"2024-03-24T12:02:56.809935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:54.700641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:54.700641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:00.996142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":411,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.423734+0000","last_change":"2024-03-24T12:01:20.958289+0000","last_active":"2024-03-24T12:02:53.423734+0000","last_peered":"2024-03-24T12:02:53.423734+0000","last_clean":"2024-03-24T12:02:53.423734+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T12:02:53.423734+0000","last_undegraded":"2024-03-24T12:02:53.423734+0000","last_fullsized":"2024-03-24T12:02:53.423734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:20.958158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:01:20.958158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:55.483637+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":447,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404209+0000","last_change":"2024-03-24T11:59:53.931656+0000","last_active":"2024-03-24T12:02:53.404209+0000","last_peered":"2024-03-24T12:02:53.404209+0000","last_clean":"2024-03-24T12:02:53.404209+0000","last_became_active":"2024-03-24T11:59:53.768899+0000","last_became_peered":"2024-03-24T11:59:53.768899+0000","last_unstale":"2024-03-24T12:02:53.404209+0000","last_undegraded":"2024-03-24T12:02:53.404209+0000","last_fullsized":"2024-03-24T12:02:53.404209+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:53.931538+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:53.931538+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.384445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"252'274","reported_seq":296,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:57.287452+0000","last_change":"2024-03-24T12:02:46.086845+0000","last_active":"2024-03-24T12:02:57.287452+0000","last_peered":"2024-03-24T12:02:57.287452+0000","last_clean":"2024-03-24T12:02:57.287452+0000","last_became_active":"2024-03-24T12:02:46.086523+0000","last_became_peered":"2024-03-24T12:02:46.086523+0000","last_unstale":"2024-03-24T12:02:57.287452+0000","last_undegraded":"2024-03-24T12:02:57.287452+0000","last_fullsized":"2024-03-24T12:02:57.287452+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:09.547319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17956864,"num_objects":274,"num_object_clones":0,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":274,"num_write_kb":17536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":415,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690652+0000","last_change":"2024-03-24T12:02:06.076507+0000","last_active":"2024-03-24T12:02:53.690652+0000","last_peered":"2024-03-24T12:02:53.690652+0000","last_clean":"2024-03-24T12:02:53.690652+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T12:02:53.690652+0000","last_undegraded":"2024-03-24T12:02:53.690652+0000","last_fullsized":"2024-03-24T12:02:53.690652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:06.076284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:02:06.076284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:07.238413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":594,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404396+0000","last_change":"2024-03-24T12:02:21.318626+0000","last_active":"2024-03-24T12:02:53.404396+0000","last_peered":"2024-03-24T12:02:53.404396+0000","last_clean":"2024-03-24T12:02:53.404396+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:02:53.404396+0000","last_undegraded":"2024-03-24T12:02:53.404396+0000","last_fullsized":"2024-03-24T12:02:53.404396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:02:21.318467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:21.318467+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:29.747681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":687,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.689833+0000","last_change":"2024-03-24T12:02:26.923681+0000","last_active":"2024-03-24T12:02:53.689833+0000","last_peered":"2024-03-24T12:02:53.689833+0000","last_clean":"2024-03-24T12:02:53.689833+0000","last_became_active":"2024-03-24T11:59:49.085087+0000","last_became_peered":"2024-03-24T11:59:49.085087+0000","last_unstale":"2024-03-24T12:02:53.689833+0000","last_undegraded":"2024-03-24T12:02:53.689833+0000","last_fullsized":"2024-03-24T12:02:53.689833+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:02:26.923560+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:02:26.923560+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:29.957461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736227+0000","last_change":"2024-03-24T11:59:51.078512+0000","last_active":"2024-03-24T12:02:53.736227+0000","last_peered":"2024-03-24T12:02:53.736227+0000","last_clean":"2024-03-24T12:02:53.736227+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T12:02:53.736227+0000","last_undegraded":"2024-03-24T12:02:53.736227+0000","last_fullsized":"2024-03-24T12:02:53.736227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:51.078392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:51.078392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811714+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":427,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806519+0000","last_change":"2024-03-24T12:02:24.416481+0000","last_active":"2024-03-24T12:02:56.806519+0000","last_peered":"2024-03-24T12:02:56.806519+0000","last_clean":"2024-03-24T12:02:56.806519+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T12:02:56.806519+0000","last_undegraded":"2024-03-24T12:02:56.806519+0000","last_fullsized":"2024-03-24T12:02:56.806519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:02:24.416403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:24.416403+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:29.535116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":401,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.735595+0000","last_change":"2024-03-24T11:59:24.028954+0000","last_active":"2024-03-24T12:02:53.735595+0000","last_peered":"2024-03-24T12:02:53.735595+0000","last_clean":"2024-03-24T12:02:53.735595+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T12:02:53.735595+0000","last_undegraded":"2024-03-24T12:02:53.735595+0000","last_fullsized":"2024-03-24T12:02:53.735595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:24.028868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.028868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811603+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"251'90","reported_seq":110,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.367851+0000","last_change":"2024-03-24T12:02:46.438184+0000","last_active":"2024-03-24T12:02:56.367851+0000","last_peered":"2024-03-24T12:02:56.367851+0000","last_clean":"2024-03-24T12:02:56.367851+0000","last_became_active":"2024-03-24T12:02:46.437687+0000","last_became_peered":"2024-03-24T12:02:46.437687+0000","last_unstale":"2024-03-24T12:02:56.367851+0000","last_undegraded":"2024-03-24T12:02:56.367851+0000","last_fullsized":"2024-03-24T12:02:56.367851+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:52.221048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":5760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":425,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.815412+0000","last_change":"2024-03-24T12:02:14.030414+0000","last_active":"2024-03-24T12:02:56.815412+0000","last_peered":"2024-03-24T12:02:56.815412+0000","last_clean":"2024-03-24T12:02:56.815412+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T12:02:56.815412+0000","last_undegraded":"2024-03-24T12:02:56.815412+0000","last_fullsized":"2024-03-24T12:02:56.815412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:14.030283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T12:02:14.030283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:41.766239+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":451,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.424058+0000","last_change":"2024-03-24T12:02:14.278544+0000","last_active":"2024-03-24T12:02:53.424058+0000","last_peered":"2024-03-24T12:02:53.424058+0000","last_clean":"2024-03-24T12:02:53.424058+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T12:02:53.424058+0000","last_undegraded":"2024-03-24T12:02:53.424058+0000","last_fullsized":"2024-03-24T12:02:53.424058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-03-24T12:02:14.278296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:14.278296+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:31.287935+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":434,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.813966+0000","last_change":"2024-03-24T12:02:22.184831+0000","last_active":"2024-03-24T12:02:56.813966+0000","last_peered":"2024-03-24T12:02:56.813966+0000","last_clean":"2024-03-24T12:02:56.813966+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T12:02:56.813966+0000","last_undegraded":"2024-03-24T12:02:56.813966+0000","last_fullsized":"2024-03-24T12:02:56.813966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:22.184708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:22.184708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:39.923735+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":426,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.813896+0000","last_change":"2024-03-24T11:59:22.979714+0000","last_active":"2024-03-24T12:02:56.813896+0000","last_peered":"2024-03-24T12:02:56.813896+0000","last_clean":"2024-03-24T12:02:56.813896+0000","last_became_active":"2024-03-24T11:59:19.095838+0000","last_became_peered":"2024-03-24T11:59:19.095838+0000","last_unstale":"2024-03-24T12:02:56.813896+0000","last_undegraded":"2024-03-24T12:02:56.813896+0000","last_fullsized":"2024-03-24T12:02:56.813896+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T11:59:22.978824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.978824+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":224,"num_read_kb":219,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":411,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690812+0000","last_change":"2024-03-24T12:02:20.995891+0000","last_active":"2024-03-24T12:02:53.690812+0000","last_peered":"2024-03-24T12:02:53.690812+0000","last_clean":"2024-03-24T12:02:53.690812+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:02:53.690812+0000","last_undegraded":"2024-03-24T12:02:53.690812+0000","last_fullsized":"2024-03-24T12:02:53.690812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:20.995809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:20.995809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:40.357955+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":694,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690323+0000","last_change":"2024-03-24T12:02:22.961543+0000","last_active":"2024-03-24T12:02:53.690323+0000","last_peered":"2024-03-24T12:02:53.690323+0000","last_clean":"2024-03-24T12:02:53.690323+0000","last_became_active":"2024-03-24T11:59:53.778968+0000","last_became_peered":"2024-03-24T11:59:53.778968+0000","last_unstale":"2024-03-24T12:02:53.690323+0000","last_undegraded":"2024-03-24T12:02:53.690323+0000","last_fullsized":"2024-03-24T12:02:53.690323+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:02:22.961404+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:02:22.961404+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:26.184269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"252'242","reported_seq":264,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:57.095418+0000","last_change":"2024-03-24T12:02:46.090758+0000","last_active":"2024-03-24T12:02:57.095418+0000","last_peered":"2024-03-24T12:02:57.095418+0000","last_clean":"2024-03-24T12:02:57.095418+0000","last_became_active":"2024-03-24T12:02:46.090175+0000","last_became_peered":"2024-03-24T12:02:46.090175+0000","last_unstale":"2024-03-24T12:02:57.095418+0000","last_undegraded":"2024-03-24T12:02:57.095418+0000","last_fullsized":"2024-03-24T12:02:57.095418+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:06.411191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":242,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":409,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.809251+0000","last_change":"2024-03-24T12:02:20.778350+0000","last_active":"2024-03-24T12:02:56.809251+0000","last_peered":"2024-03-24T12:02:56.809251+0000","last_clean":"2024-03-24T12:02:56.809251+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:02:56.809251+0000","last_undegraded":"2024-03-24T12:02:56.809251+0000","last_fullsized":"2024-03-24T12:02:56.809251+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:20.778229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:20.778229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:48.436337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"241'41","reported_seq":556,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736251+0000","last_change":"2024-03-24T12:02:04.818118+0000","last_active":"2024-03-24T12:02:53.736251+0000","last_peered":"2024-03-24T12:02:53.736251+0000","last_clean":"2024-03-24T12:02:53.736251+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T12:02:53.736251+0000","last_undegraded":"2024-03-24T12:02:53.736251+0000","last_fullsized":"2024-03-24T12:02:53.736251+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-03-24T12:02:04.818018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:04.818018+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:19.869690+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736218+0000","last_change":"2024-03-24T12:02:14.810673+0000","last_active":"2024-03-24T12:02:53.736218+0000","last_peered":"2024-03-24T12:02:53.736218+0000","last_clean":"2024-03-24T12:02:53.736218+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T12:02:53.736218+0000","last_undegraded":"2024-03-24T12:02:53.736218+0000","last_fullsized":"2024-03-24T12:02:53.736218+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:14.810556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:02:14.810556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:44.551255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807347+0000","last_change":"2024-03-24T12:02:06.504143+0000","last_active":"2024-03-24T12:02:56.807347+0000","last_peered":"2024-03-24T12:02:56.807347+0000","last_clean":"2024-03-24T12:02:56.807347+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:02:56.807347+0000","last_undegraded":"2024-03-24T12:02:56.807347+0000","last_fullsized":"2024-03-24T12:02:56.807347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:06.504045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:06.504045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:26.888906+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":441,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.735859+0000","last_change":"2024-03-24T11:59:54.989730+0000","last_active":"2024-03-24T12:02:53.735859+0000","last_peered":"2024-03-24T12:02:53.735859+0000","last_clean":"2024-03-24T12:02:53.735859+0000","last_became_active":"2024-03-24T11:59:53.773904+0000","last_became_peered":"2024-03-24T11:59:53.773904+0000","last_unstale":"2024-03-24T12:02:53.735859+0000","last_undegraded":"2024-03-24T12:02:53.735859+0000","last_fullsized":"2024-03-24T12:02:53.735859+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.989619+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.989619+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811635+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"251'78","reported_seq":98,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.364257+0000","last_change":"2024-03-24T12:02:46.437070+0000","last_active":"2024-03-24T12:02:56.364257+0000","last_peered":"2024-03-24T12:02:56.364257+0000","last_clean":"2024-03-24T12:02:56.364257+0000","last_became_active":"2024-03-24T12:02:46.436594+0000","last_became_peered":"2024-03-24T12:02:46.436594+0000","last_unstale":"2024-03-24T12:02:56.364257+0000","last_undegraded":"2024-03-24T12:02:56.364257+0000","last_fullsized":"2024-03-24T12:02:56.364257+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:55.220652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":78,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":4992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":546,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806764+0000","last_change":"2024-03-24T11:59:51.517974+0000","last_active":"2024-03-24T12:02:56.806764+0000","last_peered":"2024-03-24T12:02:56.806764+0000","last_clean":"2024-03-24T12:02:56.806764+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T12:02:56.806764+0000","last_undegraded":"2024-03-24T12:02:56.806764+0000","last_fullsized":"2024-03-24T12:02:56.806764+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:51.517833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:51.517833+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":454,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.423562+0000","last_change":"2024-03-24T12:02:08.326578+0000","last_active":"2024-03-24T12:02:53.423562+0000","last_peered":"2024-03-24T12:02:53.423562+0000","last_clean":"2024-03-24T12:02:53.423562+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T12:02:53.423562+0000","last_undegraded":"2024-03-24T12:02:53.423562+0000","last_fullsized":"2024-03-24T12:02:53.423562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:02:08.325831+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.325831+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:25.659060+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":167,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807951+0000","last_change":"2024-03-24T12:02:53.692420+0000","last_active":"2024-03-24T12:02:56.807951+0000","last_peered":"2024-03-24T12:02:56.807951+0000","last_clean":"2024-03-24T12:02:56.807951+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:02:56.807951+0000","last_undegraded":"2024-03-24T12:02:56.807951+0000","last_fullsized":"2024-03-24T12:02:56.807951+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:53.692321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:02:53.692321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:01.438757+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":397,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.805191+0000","last_change":"2024-03-24T11:59:18.622093+0000","last_active":"2024-03-24T12:02:56.805191+0000","last_peered":"2024-03-24T12:02:56.805191+0000","last_clean":"2024-03-24T12:02:56.805191+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T12:02:56.805191+0000","last_undegraded":"2024-03-24T12:02:56.805191+0000","last_fullsized":"2024-03-24T12:02:56.805191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T11:59:18.621992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":447,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404461+0000","last_change":"2024-03-24T11:59:54.973664+0000","last_active":"2024-03-24T12:02:53.404461+0000","last_peered":"2024-03-24T12:02:53.404461+0000","last_clean":"2024-03-24T12:02:53.404461+0000","last_became_active":"2024-03-24T11:59:53.769258+0000","last_became_peered":"2024-03-24T11:59:53.769258+0000","last_unstale":"2024-03-24T12:02:53.404461+0000","last_undegraded":"2024-03-24T12:02:53.404461+0000","last_fullsized":"2024-03-24T12:02:53.404461+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T11:59:54.973532+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T11:59:54.973532+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.384465+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"251'57","reported_seq":77,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.264576+0000","last_change":"2024-03-24T12:02:46.438746+0000","last_active":"2024-03-24T12:02:56.264576+0000","last_peered":"2024-03-24T12:02:56.264576+0000","last_clean":"2024-03-24T12:02:56.264576+0000","last_became_active":"2024-03-24T12:02:46.438409+0000","last_became_peered":"2024-03-24T12:02:46.438409+0000","last_unstale":"2024-03-24T12:02:56.264576+0000","last_undegraded":"2024-03-24T12:02:56.264576+0000","last_fullsized":"2024-03-24T12:02:56.264576+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:13.560954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":417,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.814164+0000","last_change":"2024-03-24T12:02:07.096552+0000","last_active":"2024-03-24T12:02:56.814164+0000","last_peered":"2024-03-24T12:02:56.814164+0000","last_clean":"2024-03-24T12:02:56.814164+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:02:56.814164+0000","last_undegraded":"2024-03-24T12:02:56.814164+0000","last_fullsized":"2024-03-24T12:02:56.814164+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:07.096421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:07.096421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:36.060668+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":467,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110383+0000","last_change":"2024-03-24T11:59:16.735826+0000","last_active":"2024-03-24T12:02:53.110383+0000","last_peered":"2024-03-24T12:02:53.110383+0000","last_clean":"2024-03-24T12:02:53.110383+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:02:53.110383+0000","last_undegraded":"2024-03-24T12:02:53.110383+0000","last_fullsized":"2024-03-24T12:02:53.110383+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T11:59:16.735674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:16.735674+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858323+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":433,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110899+0000","last_change":"2024-03-24T12:01:58.883035+0000","last_active":"2024-03-24T12:02:53.110899+0000","last_peered":"2024-03-24T12:02:53.110899+0000","last_clean":"2024-03-24T12:02:53.110899+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:02:53.110899+0000","last_undegraded":"2024-03-24T12:02:53.110899+0000","last_fullsized":"2024-03-24T12:02:53.110899+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:01:58.882896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:01:58.882896+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:17.108850+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807411+0000","last_change":"2024-03-24T12:02:23.435590+0000","last_active":"2024-03-24T12:02:56.807411+0000","last_peered":"2024-03-24T12:02:56.807411+0000","last_clean":"2024-03-24T12:02:56.807411+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:02:56.807411+0000","last_undegraded":"2024-03-24T12:02:56.807411+0000","last_fullsized":"2024-03-24T12:02:56.807411+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:02:23.435452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:49.327056+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":441,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808408+0000","last_change":"2024-03-24T12:02:22.755170+0000","last_active":"2024-03-24T12:02:56.808408+0000","last_peered":"2024-03-24T12:02:56.808408+0000","last_clean":"2024-03-24T12:02:56.808408+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:02:56.808408+0000","last_undegraded":"2024-03-24T12:02:56.808408+0000","last_fullsized":"2024-03-24T12:02:56.808408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:02:22.755046+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:02:22.755046+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:36.845669+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":425,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.814074+0000","last_change":"2024-03-24T12:02:03.999738+0000","last_active":"2024-03-24T12:02:56.814074+0000","last_peered":"2024-03-24T12:02:56.814074+0000","last_clean":"2024-03-24T12:02:56.814074+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T12:02:56.814074+0000","last_undegraded":"2024-03-24T12:02:56.814074+0000","last_fullsized":"2024-03-24T12:02:56.814074+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:03.999630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:03.999630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:29.107834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":401,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807187+0000","last_change":"2024-03-24T11:59:14.960274+0000","last_active":"2024-03-24T12:02:56.807187+0000","last_peered":"2024-03-24T12:02:56.807187+0000","last_clean":"2024-03-24T12:02:56.807187+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:02:56.807187+0000","last_undegraded":"2024-03-24T12:02:56.807187+0000","last_fullsized":"2024-03-24T12:02:56.807187+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:14.960108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:14.960108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.814348+0000","last_change":"2024-03-24T11:59:23.956872+0000","last_active":"2024-03-24T12:02:56.814348+0000","last_peered":"2024-03-24T12:02:56.814348+0000","last_clean":"2024-03-24T12:02:56.814348+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T12:02:56.814348+0000","last_undegraded":"2024-03-24T12:02:56.814348+0000","last_fullsized":"2024-03-24T12:02:56.814348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:23.956595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:23.956595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":431,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807088+0000","last_change":"2024-03-24T11:59:23.086796+0000","last_active":"2024-03-24T12:02:56.807088+0000","last_peered":"2024-03-24T12:02:56.807088+0000","last_clean":"2024-03-24T12:02:56.807088+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:02:56.807088+0000","last_undegraded":"2024-03-24T12:02:56.807088+0000","last_fullsized":"2024-03-24T12:02:56.807088+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T11:59:23.086667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:23.086667+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":420,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404098+0000","last_change":"2024-03-24T11:59:48.877101+0000","last_active":"2024-03-24T12:02:53.404098+0000","last_peered":"2024-03-24T12:02:53.404098+0000","last_clean":"2024-03-24T12:02:53.404098+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:02:53.404098+0000","last_undegraded":"2024-03-24T12:02:53.404098+0000","last_fullsized":"2024-03-24T12:02:53.404098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:48.876979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T11:59:48.876979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.384481+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736722+0000","last_change":"2024-03-24T12:01:56.816948+0000","last_active":"2024-03-24T12:02:53.736722+0000","last_peered":"2024-03-24T12:02:53.736722+0000","last_clean":"2024-03-24T12:02:53.736722+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T12:02:53.736722+0000","last_undegraded":"2024-03-24T12:02:53.736722+0000","last_fullsized":"2024-03-24T12:02:53.736722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:56.816694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:01:56.816694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:24.786941+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690567+0000","last_change":"2024-03-24T12:02:10.050207+0000","last_active":"2024-03-24T12:02:53.690567+0000","last_peered":"2024-03-24T12:02:53.690567+0000","last_clean":"2024-03-24T12:02:53.690567+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:02:53.690567+0000","last_undegraded":"2024-03-24T12:02:53.690567+0000","last_fullsized":"2024-03-24T12:02:53.690567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:10.050104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:10.050104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:12.810986+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":448,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807676+0000","last_change":"2024-03-24T11:59:57.333063+0000","last_active":"2024-03-24T12:02:56.807676+0000","last_peered":"2024-03-24T12:02:56.807676+0000","last_clean":"2024-03-24T12:02:56.807676+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:02:56.807676+0000","last_undegraded":"2024-03-24T12:02:56.807676+0000","last_fullsized":"2024-03-24T12:02:56.807676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-24T11:59:57.331734+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T11:59:57.331734+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":421,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736161+0000","last_change":"2024-03-24T12:02:26.804317+0000","last_active":"2024-03-24T12:02:53.736161+0000","last_peered":"2024-03-24T12:02:53.736161+0000","last_clean":"2024-03-24T12:02:53.736161+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T12:02:53.736161+0000","last_undegraded":"2024-03-24T12:02:53.736161+0000","last_fullsized":"2024-03-24T12:02:53.736161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:26.804218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:02:26.804218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:48.666552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":550,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736646+0000","last_change":"2024-03-24T12:01:16.301972+0000","last_active":"2024-03-24T12:02:53.736646+0000","last_peered":"2024-03-24T12:02:53.736646+0000","last_clean":"2024-03-24T12:02:53.736646+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T12:02:53.736646+0000","last_undegraded":"2024-03-24T12:02:53.736646+0000","last_fullsized":"2024-03-24T12:02:53.736646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:01:16.301884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:01:16.301884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":404,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808290+0000","last_change":"2024-03-24T12:02:26.691822+0000","last_active":"2024-03-24T12:02:56.808290+0000","last_peered":"2024-03-24T12:02:56.808290+0000","last_clean":"2024-03-24T12:02:56.808290+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:02:56.808290+0000","last_undegraded":"2024-03-24T12:02:56.808290+0000","last_fullsized":"2024-03-24T12:02:56.808290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:26.691726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:26.691726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:44.511439+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":442,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:54.865119+0000","last_change":"2024-03-24T12:02:54.865156+0000","last_active":"2024-03-24T12:02:54.865119+0000","last_peered":"2024-03-24T12:02:54.865119+0000","last_clean":"2024-03-24T12:02:54.865119+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T12:02:54.865119+0000","last_undegraded":"2024-03-24T12:02:54.865119+0000","last_fullsized":"2024-03-24T12:02:54.865119+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:02:54.865050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:54.865050+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:09.584696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":408,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806582+0000","last_change":"2024-03-24T11:59:22.532876+0000","last_active":"2024-03-24T12:02:56.806582+0000","last_peered":"2024-03-24T12:02:56.806582+0000","last_clean":"2024-03-24T12:02:56.806582+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T12:02:56.806582+0000","last_undegraded":"2024-03-24T12:02:56.806582+0000","last_fullsized":"2024-03-24T12:02:56.806582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.532751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.532751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":546,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807925+0000","last_change":"2024-03-24T11:59:24.105607+0000","last_active":"2024-03-24T12:02:56.807925+0000","last_peered":"2024-03-24T12:02:56.807925+0000","last_clean":"2024-03-24T12:02:56.807925+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T12:02:56.807925+0000","last_undegraded":"2024-03-24T12:02:56.807925+0000","last_fullsized":"2024-03-24T12:02:56.807925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T11:59:24.105308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:24.105308+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":409,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.111041+0000","last_change":"2024-03-24T12:02:08.957308+0000","last_active":"2024-03-24T12:02:53.111041+0000","last_peered":"2024-03-24T12:02:53.111041+0000","last_clean":"2024-03-24T12:02:53.111041+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T12:02:53.111041+0000","last_undegraded":"2024-03-24T12:02:53.111041+0000","last_fullsized":"2024-03-24T12:02:53.111041+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:08.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:37.411715+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":209,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404202+0000","last_change":"2024-03-24T12:02:14.181039+0000","last_active":"2024-03-24T12:02:53.404202+0000","last_peered":"2024-03-24T12:02:53.404202+0000","last_clean":"2024-03-24T12:02:53.404202+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:02:53.404202+0000","last_undegraded":"2024-03-24T12:02:53.404202+0000","last_fullsized":"2024-03-24T12:02:53.404202+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:02:14.180895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:14.180895+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:19.749588+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":428,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404397+0000","last_change":"2024-03-24T12:02:26.299507+0000","last_active":"2024-03-24T12:02:53.404397+0000","last_peered":"2024-03-24T12:02:53.404397+0000","last_clean":"2024-03-24T12:02:53.404397+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T12:02:53.404397+0000","last_undegraded":"2024-03-24T12:02:53.404397+0000","last_fullsized":"2024-03-24T12:02:53.404397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:02:26.299382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:51.680918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":409,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807405+0000","last_change":"2024-03-24T12:01:19.475234+0000","last_active":"2024-03-24T12:02:56.807405+0000","last_peered":"2024-03-24T12:02:56.807405+0000","last_clean":"2024-03-24T12:02:56.807405+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T12:02:56.807405+0000","last_undegraded":"2024-03-24T12:02:56.807405+0000","last_fullsized":"2024-03-24T12:02:56.807405+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:19.474426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.474426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":404,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.805113+0000","last_change":"2024-03-24T12:02:04.803951+0000","last_active":"2024-03-24T12:02:56.805113+0000","last_peered":"2024-03-24T12:02:56.805113+0000","last_clean":"2024-03-24T12:02:56.805113+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T12:02:56.805113+0000","last_undegraded":"2024-03-24T12:02:56.805113+0000","last_fullsized":"2024-03-24T12:02:56.805113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:04.803578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T12:02:04.803578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:28.836025+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":440,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110629+0000","last_change":"2024-03-24T11:59:49.721576+0000","last_active":"2024-03-24T12:02:53.110629+0000","last_peered":"2024-03-24T12:02:53.110629+0000","last_clean":"2024-03-24T12:02:53.110629+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:02:53.110629+0000","last_undegraded":"2024-03-24T12:02:53.110629+0000","last_fullsized":"2024-03-24T12:02:53.110629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:49.721488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:49.721488+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":430,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.423805+0000","last_change":"2024-03-24T12:02:24.278486+0000","last_active":"2024-03-24T12:02:53.423805+0000","last_peered":"2024-03-24T12:02:53.423805+0000","last_clean":"2024-03-24T12:02:53.423805+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T12:02:53.423805+0000","last_undegraded":"2024-03-24T12:02:53.423805+0000","last_fullsized":"2024-03-24T12:02:53.423805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:24.278397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:24.278397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:41.117003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":166,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736461+0000","last_change":"2024-03-24T11:59:58.999301+0000","last_active":"2024-03-24T12:02:53.736461+0000","last_peered":"2024-03-24T12:02:53.736461+0000","last_clean":"2024-03-24T12:02:53.736461+0000","last_became_active":"2024-03-24T11:59:19.099779+0000","last_became_peered":"2024-03-24T11:59:19.099779+0000","last_unstale":"2024-03-24T12:02:53.736461+0000","last_undegraded":"2024-03-24T12:02:53.736461+0000","last_fullsized":"2024-03-24T12:02:53.736461+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:58.998975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T11:59:58.998975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811684+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":397,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.812541+0000","last_change":"2024-03-24T11:59:08.970434+0000","last_active":"2024-03-24T12:02:56.812541+0000","last_peered":"2024-03-24T12:02:56.812541+0000","last_clean":"2024-03-24T12:02:56.812541+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T12:02:56.812541+0000","last_undegraded":"2024-03-24T12:02:56.812541+0000","last_fullsized":"2024-03-24T12:02:56.812541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:08.970296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:08.970296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":427,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807148+0000","last_change":"2024-03-24T11:59:25.157079+0000","last_active":"2024-03-24T12:02:56.807148+0000","last_peered":"2024-03-24T12:02:56.807148+0000","last_clean":"2024-03-24T12:02:56.807148+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:02:56.807148+0000","last_undegraded":"2024-03-24T12:02:56.807148+0000","last_fullsized":"2024-03-24T12:02:56.807148+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'6","last_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T11:59:25.156553+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":413,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736822+0000","last_change":"2024-03-24T12:01:20.259860+0000","last_active":"2024-03-24T12:02:53.736822+0000","last_peered":"2024-03-24T12:02:53.736822+0000","last_clean":"2024-03-24T12:02:53.736822+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T12:02:53.736822+0000","last_undegraded":"2024-03-24T12:02:53.736822+0000","last_fullsized":"2024-03-24T12:02:53.736822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:20.259734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:01:20.259734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811746+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":412,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404061+0000","last_change":"2024-03-24T12:00:07.037958+0000","last_active":"2024-03-24T12:02:53.404061+0000","last_peered":"2024-03-24T12:02:53.404061+0000","last_clean":"2024-03-24T12:02:53.404061+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:02:53.404061+0000","last_undegraded":"2024-03-24T12:02:53.404061+0000","last_fullsized":"2024-03-24T12:02:53.404061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:07.037878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:00:07.037878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.384517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":402,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.689563+0000","last_change":"2024-03-24T12:00:01.072539+0000","last_active":"2024-03-24T12:02:53.689563+0000","last_peered":"2024-03-24T12:02:53.689563+0000","last_clean":"2024-03-24T12:02:53.689563+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:02:53.689563+0000","last_undegraded":"2024-03-24T12:02:53.689563+0000","last_fullsized":"2024-03-24T12:02:53.689563+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:01.072391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:00:01.072391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.762362+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":436,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808157+0000","last_change":"2024-03-24T11:59:07.046859+0000","last_active":"2024-03-24T12:02:56.808157+0000","last_peered":"2024-03-24T12:02:56.808157+0000","last_clean":"2024-03-24T12:02:56.808157+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:02:56.808157+0000","last_undegraded":"2024-03-24T12:02:56.808157+0000","last_fullsized":"2024-03-24T12:02:56.808157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T11:59:07.046695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T11:59:07.046695+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":420,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404602+0000","last_change":"2024-03-24T12:02:28.309650+0000","last_active":"2024-03-24T12:02:53.404602+0000","last_peered":"2024-03-24T12:02:53.404602+0000","last_clean":"2024-03-24T12:02:53.404602+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T12:02:53.404602+0000","last_undegraded":"2024-03-24T12:02:53.404602+0000","last_fullsized":"2024-03-24T12:02:53.404602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:28.309522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:28.309522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:41.912401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":406,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690061+0000","last_change":"2024-03-24T12:01:19.064543+0000","last_active":"2024-03-24T12:02:53.690061+0000","last_peered":"2024-03-24T12:02:53.690061+0000","last_clean":"2024-03-24T12:02:53.690061+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:02:53.690061+0000","last_undegraded":"2024-03-24T12:02:53.690061+0000","last_fullsized":"2024-03-24T12:02:53.690061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:19.064391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.064391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.762382+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":397,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807512+0000","last_change":"2024-03-24T11:59:06.058871+0000","last_active":"2024-03-24T12:02:56.807512+0000","last_peered":"2024-03-24T12:02:56.807512+0000","last_clean":"2024-03-24T12:02:56.807512+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:02:56.807512+0000","last_undegraded":"2024-03-24T12:02:56.807512+0000","last_fullsized":"2024-03-24T12:02:56.807512+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:06.058633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:06.058633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":448,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808646+0000","last_change":"2024-03-24T12:02:16.773052+0000","last_active":"2024-03-24T12:02:56.808646+0000","last_peered":"2024-03-24T12:02:56.808646+0000","last_clean":"2024-03-24T12:02:56.808646+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:02:56.808646+0000","last_undegraded":"2024-03-24T12:02:56.808646+0000","last_fullsized":"2024-03-24T12:02:56.808646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-03-24T12:02:16.771488+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:02:16.771488+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:36.015402+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":421,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.735820+0000","last_change":"2024-03-24T12:02:27.792896+0000","last_active":"2024-03-24T12:02:53.735820+0000","last_peered":"2024-03-24T12:02:53.735820+0000","last_clean":"2024-03-24T12:02:53.735820+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T12:02:53.735820+0000","last_undegraded":"2024-03-24T12:02:53.735820+0000","last_fullsized":"2024-03-24T12:02:53.735820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:27.792783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:27.792783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:35.238163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":553,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404803+0000","last_change":"2024-03-24T12:02:23.353413+0000","last_active":"2024-03-24T12:02:53.404803+0000","last_peered":"2024-03-24T12:02:53.404803+0000","last_clean":"2024-03-24T12:02:53.404803+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:02:53.404803+0000","last_undegraded":"2024-03-24T12:02:53.404803+0000","last_fullsized":"2024-03-24T12:02:53.404803+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:02:23.353279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:23.353279+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:33.835418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":420,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690793+0000","last_change":"2024-03-24T12:02:24.907144+0000","last_active":"2024-03-24T12:02:53.690793+0000","last_peered":"2024-03-24T12:02:53.690793+0000","last_clean":"2024-03-24T12:02:53.690793+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T12:02:53.690793+0000","last_undegraded":"2024-03-24T12:02:53.690793+0000","last_fullsized":"2024-03-24T12:02:53.690793+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:02:24.907051+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:02:24.907051+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:40.414825+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":456,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404744+0000","last_change":"2024-03-24T12:02:12.161719+0000","last_active":"2024-03-24T12:02:53.404744+0000","last_peered":"2024-03-24T12:02:53.404744+0000","last_clean":"2024-03-24T12:02:53.404744+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T12:02:53.404744+0000","last_undegraded":"2024-03-24T12:02:53.404744+0000","last_fullsized":"2024-03-24T12:02:53.404744+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:02:12.161589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:12.161589+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:19.376501+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":410,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807783+0000","last_change":"2024-03-24T11:59:09.953682+0000","last_active":"2024-03-24T12:02:56.807783+0000","last_peered":"2024-03-24T12:02:56.807783+0000","last_clean":"2024-03-24T12:02:56.807783+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:02:56.807783+0000","last_undegraded":"2024-03-24T12:02:56.807783+0000","last_fullsized":"2024-03-24T12:02:56.807783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T11:59:09.953549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T11:59:09.953549+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":409,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808759+0000","last_change":"2024-03-24T12:00:02.387977+0000","last_active":"2024-03-24T12:02:56.808759+0000","last_peered":"2024-03-24T12:02:56.808759+0000","last_clean":"2024-03-24T12:02:56.808759+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T12:02:56.808759+0000","last_undegraded":"2024-03-24T12:02:56.808759+0000","last_fullsized":"2024-03-24T12:02:56.808759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:02.387837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:00:02.387837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":165,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736700+0000","last_change":"2024-03-24T12:02:25.784806+0000","last_active":"2024-03-24T12:02:53.736700+0000","last_peered":"2024-03-24T12:02:53.736700+0000","last_clean":"2024-03-24T12:02:53.736700+0000","last_became_active":"2024-03-24T11:59:19.098557+0000","last_became_peered":"2024-03-24T11:59:19.098557+0000","last_unstale":"2024-03-24T12:02:53.736700+0000","last_undegraded":"2024-03-24T12:02:53.736700+0000","last_fullsized":"2024-03-24T12:02:53.736700+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:25.784638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:25.784638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:41.852293+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":471,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.816268+0000","last_change":"2024-03-24T12:02:01.949233+0000","last_active":"2024-03-24T12:02:56.816268+0000","last_peered":"2024-03-24T12:02:56.816268+0000","last_clean":"2024-03-24T12:02:56.816268+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T12:02:56.816268+0000","last_undegraded":"2024-03-24T12:02:56.816268+0000","last_fullsized":"2024-03-24T12:02:56.816268+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'22","last_scrub_stamp":"2024-03-24T12:02:01.948968+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:02:01.948968+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:11.553113+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":420,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.404401+0000","last_change":"2024-03-24T11:59:20.890264+0000","last_active":"2024-03-24T12:02:53.404401+0000","last_peered":"2024-03-24T12:02:53.404401+0000","last_clean":"2024-03-24T12:02:53.404401+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:02:53.404401+0000","last_undegraded":"2024-03-24T12:02:53.404401+0000","last_fullsized":"2024-03-24T12:02:53.404401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_clean_scrub_stamp":"2024-03-24T11:59:20.890106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.384401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":174,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.111161+0000","last_change":"2024-03-24T12:02:26.788689+0000","last_active":"2024-03-24T12:02:53.111161+0000","last_peered":"2024-03-24T12:02:53.111161+0000","last_clean":"2024-03-24T12:02:53.111161+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:02:53.111161+0000","last_undegraded":"2024-03-24T12:02:53.111161+0000","last_fullsized":"2024-03-24T12:02:53.111161+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:26.788584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:26.788584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:29.594410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110952+0000","last_change":"2024-03-24T12:02:18.004526+0000","last_active":"2024-03-24T12:02:53.110952+0000","last_peered":"2024-03-24T12:02:53.110952+0000","last_clean":"2024-03-24T12:02:53.110952+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T12:02:53.110952+0000","last_undegraded":"2024-03-24T12:02:53.110952+0000","last_fullsized":"2024-03-24T12:02:53.110952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:18.004438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:22.363231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":468,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806779+0000","last_change":"2024-03-24T12:02:26.446693+0000","last_active":"2024-03-24T12:02:56.806779+0000","last_peered":"2024-03-24T12:02:56.806779+0000","last_clean":"2024-03-24T12:02:56.806779+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:02:56.806779+0000","last_undegraded":"2024-03-24T12:02:56.806779+0000","last_fullsized":"2024-03-24T12:02:56.806779+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:02:26.446597+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:50.904133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":423,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807303+0000","last_change":"2024-03-24T12:00:05.346648+0000","last_active":"2024-03-24T12:02:56.807303+0000","last_peered":"2024-03-24T12:02:56.807303+0000","last_clean":"2024-03-24T12:02:56.807303+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:02:56.807303+0000","last_undegraded":"2024-03-24T12:02:56.807303+0000","last_fullsized":"2024-03-24T12:02:56.807303+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:00:05.346515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:00:05.346515+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110704+0000","last_change":"2024-03-24T11:59:22.719982+0000","last_active":"2024-03-24T12:02:53.110704+0000","last_peered":"2024-03-24T12:02:53.110704+0000","last_clean":"2024-03-24T12:02:53.110704+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T12:02:53.110704+0000","last_undegraded":"2024-03-24T12:02:53.110704+0000","last_fullsized":"2024-03-24T12:02:53.110704+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:22.719851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T11:59:22.719851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":406,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.806984+0000","last_change":"2024-03-24T11:59:21.206602+0000","last_active":"2024-03-24T12:02:56.806984+0000","last_peered":"2024-03-24T12:02:56.806984+0000","last_clean":"2024-03-24T12:02:56.806984+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T12:02:56.806984+0000","last_undegraded":"2024-03-24T12:02:56.806984+0000","last_fullsized":"2024-03-24T12:02:56.806984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:21.206482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T11:59:21.206482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":442,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110352+0000","last_change":"2024-03-24T12:00:03.691490+0000","last_active":"2024-03-24T12:02:53.110352+0000","last_peered":"2024-03-24T12:02:53.110352+0000","last_clean":"2024-03-24T12:02:53.110352+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:02:53.110352+0000","last_undegraded":"2024-03-24T12:02:53.110352+0000","last_fullsized":"2024-03-24T12:02:53.110352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:00:03.691210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:00:03.691210+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858440+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":425,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690481+0000","last_change":"2024-03-24T12:02:07.041906+0000","last_active":"2024-03-24T12:02:53.690481+0000","last_peered":"2024-03-24T12:02:53.690481+0000","last_clean":"2024-03-24T12:02:53.690481+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T12:02:53.690481+0000","last_undegraded":"2024-03-24T12:02:53.690481+0000","last_fullsized":"2024-03-24T12:02:53.690481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:07.041777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:02:07.041777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:12.080752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.736224+0000","last_change":"2024-03-24T12:01:14.211440+0000","last_active":"2024-03-24T12:02:53.736224+0000","last_peered":"2024-03-24T12:02:53.736224+0000","last_clean":"2024-03-24T12:02:53.736224+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T12:02:53.736224+0000","last_undegraded":"2024-03-24T12:02:53.736224+0000","last_fullsized":"2024-03-24T12:02:53.736224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:14.211332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:01:14.211332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811779+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":396,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.805525+0000","last_change":"2024-03-24T12:02:05.800911+0000","last_active":"2024-03-24T12:02:56.805525+0000","last_peered":"2024-03-24T12:02:56.805525+0000","last_clean":"2024-03-24T12:02:56.805525+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:02:56.805525+0000","last_undegraded":"2024-03-24T12:02:56.805525+0000","last_fullsized":"2024-03-24T12:02:56.805525+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:05.800774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:05.800774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:17.921551+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":432,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110548+0000","last_change":"2024-03-24T12:01:19.495169+0000","last_active":"2024-03-24T12:02:53.110548+0000","last_peered":"2024-03-24T12:02:53.110548+0000","last_clean":"2024-03-24T12:02:53.110548+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:02:53.110548+0000","last_undegraded":"2024-03-24T12:02:53.110548+0000","last_fullsized":"2024-03-24T12:02:53.110548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-24T12:01:19.495033+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:01:19.495033+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858473+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690648+0000","last_change":"2024-03-24T12:00:03.053959+0000","last_active":"2024-03-24T12:02:53.690648+0000","last_peered":"2024-03-24T12:02:53.690648+0000","last_clean":"2024-03-24T12:02:53.690648+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:02:53.690648+0000","last_undegraded":"2024-03-24T12:02:53.690648+0000","last_fullsized":"2024-03-24T12:02:53.690648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:00:03.053839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:57.762341+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.690151+0000","last_change":"2024-03-24T12:02:08.062917+0000","last_active":"2024-03-24T12:02:53.690151+0000","last_peered":"2024-03-24T12:02:53.690151+0000","last_clean":"2024-03-24T12:02:53.690151+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:02:53.690151+0000","last_undegraded":"2024-03-24T12:02:53.690151+0000","last_fullsized":"2024-03-24T12:02:53.690151+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:08.062779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T12:02:08.062779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:08.263500+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":409,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.735452+0000","last_change":"2024-03-24T12:01:21.229722+0000","last_active":"2024-03-24T12:02:53.735452+0000","last_peered":"2024-03-24T12:02:53.735452+0000","last_clean":"2024-03-24T12:02:53.735452+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T12:02:53.735452+0000","last_undegraded":"2024-03-24T12:02:53.735452+0000","last_fullsized":"2024-03-24T12:02:53.735452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:21.229585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:01:21.229585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.811824+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":446,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110060+0000","last_change":"2024-03-24T12:02:28.731440+0000","last_active":"2024-03-24T12:02:53.110060+0000","last_peered":"2024-03-24T12:02:53.110060+0000","last_clean":"2024-03-24T12:02:53.110060+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:02:53.110060+0000","last_undegraded":"2024-03-24T12:02:53.110060+0000","last_fullsized":"2024-03-24T12:02:53.110060+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:02:28.731326+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:02:28.731326+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:55.441296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":418,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807253+0000","last_change":"2024-03-24T12:00:06.381223+0000","last_active":"2024-03-24T12:02:56.807253+0000","last_peered":"2024-03-24T12:02:56.807253+0000","last_clean":"2024-03-24T12:02:56.807253+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T12:02:56.807253+0000","last_undegraded":"2024-03-24T12:02:56.807253+0000","last_fullsized":"2024-03-24T12:02:56.807253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:00:06.381099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":166,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.735724+0000","last_change":"2024-03-24T12:02:17.838660+0000","last_active":"2024-03-24T12:02:53.735724+0000","last_peered":"2024-03-24T12:02:53.735724+0000","last_clean":"2024-03-24T12:02:53.735724+0000","last_became_active":"2024-03-24T11:59:19.117244+0000","last_became_peered":"2024-03-24T11:59:19.117244+0000","last_unstale":"2024-03-24T12:02:53.735724+0000","last_undegraded":"2024-03-24T12:02:53.735724+0000","last_fullsized":"2024-03-24T12:02:53.735724+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:17.838539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:17.838539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:19.996793+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":409,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110264+0000","last_change":"2024-03-24T12:01:59.894957+0000","last_active":"2024-03-24T12:02:53.110264+0000","last_peered":"2024-03-24T12:02:53.110264+0000","last_clean":"2024-03-24T12:02:53.110264+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T12:02:53.110264+0000","last_undegraded":"2024-03-24T12:02:53.110264+0000","last_fullsized":"2024-03-24T12:02:53.110264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:01:59.894840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:01:59.894840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:21.806890+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":463,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808255+0000","last_change":"2024-03-24T12:02:02.488565+0000","last_active":"2024-03-24T12:02:56.808255+0000","last_peered":"2024-03-24T12:02:56.808255+0000","last_clean":"2024-03-24T12:02:56.808255+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T12:02:56.808255+0000","last_undegraded":"2024-03-24T12:02:56.808255+0000","last_fullsized":"2024-03-24T12:02:56.808255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'22","last_scrub_stamp":"2024-03-24T12:02:02.488418+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:02:02.488418+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:32.060704+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":409,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.808681+0000","last_change":"2024-03-24T11:59:50.555909+0000","last_active":"2024-03-24T12:02:56.808681+0000","last_peered":"2024-03-24T12:02:56.808681+0000","last_clean":"2024-03-24T12:02:56.808681+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:02:56.808681+0000","last_undegraded":"2024-03-24T12:02:56.808681+0000","last_fullsized":"2024-03-24T12:02:56.808681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:50.555718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T11:59:50.555718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":413,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-24T12:02:53.110325+0000","last_change":"2024-03-24T11:59:17.681747+0000","last_active":"2024-03-24T12:02:53.110325+0000","last_peered":"2024-03-24T12:02:53.110325+0000","last_clean":"2024-03-24T12:02:53.110325+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T12:02:53.110325+0000","last_undegraded":"2024-03-24T12:02:53.110325+0000","last_fullsized":"2024-03-24T12:02:53.110325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:59:17.681628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:29.715710+0000","last_clean_scrub_stamp":"2024-03-24T11:59:17.681628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:53.858307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-24T12:02:56.807070+0000","last_change":"2024-03-24T12:02:25.491876+0000","last_active":"2024-03-24T12:02:56.807070+0000","last_peered":"2024-03-24T12:02:56.807070+0000","last_clean":"2024-03-24T12:02:56.807070+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:02:56.807070+0000","last_undegraded":"2024-03-24T12:02:56.807070+0000","last_fullsized":"2024-03-24T12:02:56.807070+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:25.491759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:02:25.491759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:49.273445+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":161344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":431947776,"data_stored":431947776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2521,"ondisk_log_size":2521,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":219,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":772,"ondisk_log_size":772,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005188,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357232,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93387920,"statfs":{"total":95995035648,"available":95629230080,"internally_reserved":0,"allocated":1216512,"data_stored":800275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":364575178},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332235,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347180,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93397972,"statfs":{"total":95995035648,"available":95639523328,"internally_reserved":0,"allocated":884736,"data_stored":472413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415220,"kb_used_data":59180,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93329932,"statfs":{"total":95995035648,"available":95569850368,"internally_reserved":0,"allocated":60600320,"data_stored":60175288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":462444,"kb_used_data":43940,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93282708,"statfs":{"total":95995035648,"available":95521492992,"internally_reserved":0,"allocated":44994560,"data_stored":44577190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411487,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":414708,"kb_used_data":14252,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93330444,"statfs":{"total":95995035648,"available":95570374656,"internally_reserved":0,"allocated":14594048,"data_stored":14169980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501512,"kb_used_data":69120,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93243640,"statfs":{"total":95995035648,"available":95481487360,"internally_reserved":0,"allocated":70778880,"data_stored":70334762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":442759623},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347192,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93397960,"statfs":{"total":95995035648,"available":95639511040,"internally_reserved":0,"allocated":901120,"data_stored":472027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659279,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":383864,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93361288,"statfs":{"total":95995035648,"available":95601958912,"internally_reserved":0,"allocated":1486848,"data_stored":1062507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63700992,"data_stored":63700992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69992448,"data_stored":69992448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41549824,"data_stored":41549824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":143982592,"data_stored":143982592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11075584,"data_stored":11075584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":51576832,"data_stored":51576832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13107200,"data_stored":13107200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36962304,"data_stored":36962304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:03:02.429 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T12:03:02.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-24T12:03:02.437 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:02.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:02.514 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:02.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:02.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:02.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:02.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:02.804 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:02.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:02.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:02.906 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:02.940 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:02.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:03.002 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 7 16 19870 19854 177.224 147.562 0.00647782 0.0056362 2024-03-24T12:03:03.042 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:03.079 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:03.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:03.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:03.218 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:03.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:03.320 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:03.359 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:03.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:03.461 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:03.495 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:03.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:03 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:03:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:03 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:03:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:03 smithi064 ceph-mon[30223]: 6.0 scrub starts 2024-03-24T12:03:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:03 smithi064 ceph-mon[30223]: 6.0 scrub ok 2024-03-24T12:03:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:03 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:03:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:03 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:03:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:03 smithi064 ceph-mon[42243]: 6.0 scrub starts 2024-03-24T12:03:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:03 smithi064 ceph-mon[42243]: 6.0 scrub ok 2024-03-24T12:03:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:03 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:03:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:03 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:03:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:03 smithi193 ceph-mon[33593]: 6.0 scrub starts 2024-03-24T12:03:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:03 smithi193 ceph-mon[33593]: 6.0 scrub ok 2024-03-24T12:03:03.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:03.642 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:03.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:03.744 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:03.779 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:03.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:03.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:03.928 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:03.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:04.002 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 8 15 22411 22396 174.929 158.875 0.0032573 0.00571121 2024-03-24T12:03:04.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:04.068 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:04.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:04.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:04.207 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:04.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:04.308 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:04.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:04 smithi193 ceph-mon[33593]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T12:03:04.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:04 smithi193 ceph-mon[33593]: pgmap v480: 158 pgs: 158 active+clean; 1.0 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-24T12:03:04.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:04 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T12:03:04.331 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:04 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T12:03:04.331 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:04 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T12:03:04.331 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:04 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T12:03:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:04.350 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:04.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:04.446 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[30223]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[30223]: pgmap v480: 158 pgs: 158 active+clean; 1.0 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[42243]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[42243]: pgmap v480: 158 pgs: 158 active+clean; 1.0 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T12:03:04.447 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:04 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T12:03:04.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:04.494 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:04.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:04.547 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:03:04.548 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:03:04.548 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-03-24T12:03:04.548 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:03:04.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:04.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:04.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:04.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:04.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:04.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:04.911 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:04.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:05.002 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 9 16 24649 24633 171.026 139.812 0.006963 0.005841 2024-03-24T12:03:05.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:05.053 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:05.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:05.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:05.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:05.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:05.328 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:05.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:05.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:05.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:05.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[30223]: 1.1 scrub starts 2024-03-24T12:03:05.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[30223]: 1.1 scrub ok 2024-03-24T12:03:05.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[42243]: 1.1 scrub starts 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[42243]: 1.1 scrub ok 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:03:05.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:05 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:03:05.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:05 smithi193 ceph-mon[33593]: 1.1 scrub starts 2024-03-24T12:03:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:05 smithi193 ceph-mon[33593]: 1.1 scrub ok 2024-03-24T12:03:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:05 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T12:03:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:05 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T12:03:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:05 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:03:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:05 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:03:05.587 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:05.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:05.689 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:05.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:05.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:05.828 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:05.862 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:05.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:05.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:05.990 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:05.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:06.005 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 10 16 26356 26340 164.592 106.688 0.00172258 0.00604312 2024-03-24T12:03:06.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:06.138 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:06.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:06.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:06.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:06.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:06 smithi064 ceph-mon[30223]: 6.11 scrub starts 2024-03-24T12:03:06.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:06 smithi064 ceph-mon[30223]: 6.11 scrub ok 2024-03-24T12:03:06.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:06 smithi064 ceph-mon[30223]: pgmap v481: 158 pgs: 158 active+clean; 1.0 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-24T12:03:06.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:06 smithi064 ceph-mon[42243]: 6.11 scrub starts 2024-03-24T12:03:06.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:06 smithi064 ceph-mon[42243]: 6.11 scrub ok 2024-03-24T12:03:06.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:06 smithi064 ceph-mon[42243]: pgmap v481: 158 pgs: 158 active+clean; 1.0 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-24T12:03:06.281 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:06.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:06.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:06.418 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:03:06.423 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:06.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:06.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:06.454 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:06 smithi193 ceph-mon[33593]: 6.11 scrub starts 2024-03-24T12:03:06.454 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:06 smithi193 ceph-mon[33593]: 6.11 scrub ok 2024-03-24T12:03:06.454 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:06 smithi193 ceph-mon[33593]: pgmap v481: 158 pgs: 158 active+clean; 1.0 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-24T12:03:06.525 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:06.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:06.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:06.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:06.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:06.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:06.836 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:06.843 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:06.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:06.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:06.986 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:06.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:07.002 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 11 16 28447 28431 161.509 130.688 0.00448913 0.0061867 2024-03-24T12:03:07.088 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:07.126 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:07.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:07.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:07.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:07.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:07.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:07 smithi193 ceph-mon[33593]: 1.7 scrub starts 2024-03-24T12:03:07.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:07 smithi193 ceph-mon[33593]: 1.7 scrub ok 2024-03-24T12:03:07.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:07 smithi193 ceph-mon[33593]: 6.13 scrub starts 2024-03-24T12:03:07.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:07 smithi193 ceph-mon[33593]: 6.13 scrub ok 2024-03-24T12:03:07.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:07 smithi193 ceph-mon[33593]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T12:03:07.365 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:07.406 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:07.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:07.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:07.532 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:07.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:07.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:03:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[30223]: 1.7 scrub starts 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[30223]: 1.7 scrub ok 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[30223]: 6.13 scrub starts 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[30223]: 6.13 scrub ok 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[30223]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[42243]: 1.7 scrub starts 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[42243]: 1.7 scrub ok 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[42243]: 6.13 scrub starts 2024-03-24T12:03:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[42243]: 6.13 scrub ok 2024-03-24T12:03:07.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:07 smithi064 ceph-mon[42243]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T12:03:07.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:03:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:03:07.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:07.661 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:07.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:07.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:07.762 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:07.800 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:07.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:07.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:07.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:08.002 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 12 16 30018 30002 156.232 98.1875 0.00340441 0.00632871 2024-03-24T12:03:08.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:08.051 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:08.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:08.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:08.187 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:08.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:08.210 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:03:08.253 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:03:08.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: 1.a scrub starts 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: 1.a scrub ok 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: pgmap v483: 158 pgs: 158 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 214 MiB/s wr, 3.42k op/s 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/12658492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: 1.a scrub starts 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: 1.a scrub ok 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:08.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: pgmap v483: 158 pgs: 158 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 214 MiB/s wr, 3.42k op/s 2024-03-24T12:03:08.265 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T12:03:08.265 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T12:03:08.265 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:08 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/12658492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:08.289 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:08.294 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:03:08.320 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:08.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:08.332 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:03:08.357 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:03:08.402 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:03:08.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:08.426 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:03:08.461 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:08.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:08.464 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:03:08.466 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:03:08.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:03:08.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:03:08.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:03:08.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:03:08.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:03:08.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:03:08.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:03:08.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: 1.a scrub starts 2024-03-24T12:03:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: 1.a scrub ok 2024-03-24T12:03:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T12:03:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T12:03:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: pgmap v483: 158 pgs: 158 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 214 MiB/s wr, 3.42k op/s 2024-03-24T12:03:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T12:03:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T12:03:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:08 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/12658492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:08.598 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:08.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:08.699 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:08.737 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:08.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:08.838 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:08.874 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:08.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:08.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:08.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:09.002 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:09.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:09.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 13 16 31704 31688 152.32 105.375 0.00160695 0.00654224 2024-03-24T12:03:09.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:09.138 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:09.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:09.240 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:09.277 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:09.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:09.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:09.404 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:09.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:09.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:09.542 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:09.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T12:03:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[30223]: 4.8 scrub starts 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[30223]: 4.8 scrub ok 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[42243]: 4.8 scrub starts 2024-03-24T12:03:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:09 smithi064 ceph-mon[42243]: 4.8 scrub ok 2024-03-24T12:03:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:09 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T12:03:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:09 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T12:03:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:09 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T12:03:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:09 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T12:03:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:09 smithi193 ceph-mon[33593]: 4.8 scrub starts 2024-03-24T12:03:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:09 smithi193 ceph-mon[33593]: 4.8 scrub ok 2024-03-24T12:03:09.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:09.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:09.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:09.824 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:09.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:09.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:09.953 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:09.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:09.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:10.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 14 16 33507 33491 149.488 112.688 0.00339135 0.00668346 2024-03-24T12:03:10.055 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:10.091 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:10.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:10.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:10.229 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:10.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:10.331 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:10.354 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:10.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:10.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:10.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:10.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 1.6 scrub starts 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 1.6 scrub ok 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T12:03:10.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: pgmap v484: 158 pgs: 158 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 2.19 scrub starts 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[30223]: 2.19 scrub ok 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 1.6 scrub starts 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 1.6 scrub ok 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: pgmap v484: 158 pgs: 158 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 2.19 scrub starts 2024-03-24T12:03:10.517 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:10 smithi064 ceph-mon[42243]: 2.19 scrub ok 2024-03-24T12:03:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 1.6 scrub starts 2024-03-24T12:03:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 1.6 scrub ok 2024-03-24T12:03:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T12:03:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T12:03:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T12:03:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T12:03:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T12:03:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T12:03:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: pgmap v484: 158 pgs: 158 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-24T12:03:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 2.19 scrub starts 2024-03-24T12:03:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:10 smithi193 ceph-mon[33593]: 2.19 scrub ok 2024-03-24T12:03:10.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:10.634 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:10.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:10.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:10.772 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:10.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:10.874 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:10.910 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:10.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:11.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 15 16 35486 35470 147.768 123.688 0.00761991 0.00676261 2024-03-24T12:03:11.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:11.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:11.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:11.202 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:11.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:11.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:11.343 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:11.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:11.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:11.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:11 smithi193 ceph-mon[33593]: 4.e scrub starts 2024-03-24T12:03:11.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:11 smithi193 ceph-mon[33593]: 4.e scrub ok 2024-03-24T12:03:11.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:11 smithi193 ceph-mon[33593]: 5.12 scrub starts 2024-03-24T12:03:11.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:11 smithi193 ceph-mon[33593]: 5.12 scrub ok 2024-03-24T12:03:11.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:11 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T12:03:11.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:11 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T12:03:11.487 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:11.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:11.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[30223]: 4.e scrub starts 2024-03-24T12:03:11.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[30223]: 4.e scrub ok 2024-03-24T12:03:11.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[30223]: 5.12 scrub starts 2024-03-24T12:03:11.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[30223]: 5.12 scrub ok 2024-03-24T12:03:11.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T12:03:11.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T12:03:11.503 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[42243]: 4.e scrub starts 2024-03-24T12:03:11.504 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[42243]: 4.e scrub ok 2024-03-24T12:03:11.504 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[42243]: 5.12 scrub starts 2024-03-24T12:03:11.504 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[42243]: 5.12 scrub ok 2024-03-24T12:03:11.504 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T12:03:11.504 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:11 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T12:03:11.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:11.622 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:11.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:11.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:11.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:11.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:11.913 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:11.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:11.932 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:03:11.932 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:03:12.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 16 16 37339 37323 145.77 115.812 0.00445742 0.00685403 2024-03-24T12:03:12.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:12.050 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:12.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:12.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:12.185 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:12.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:12.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:12.323 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:12.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:12.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:12.464 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:12.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:12.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[30223]: pgmap v485: 158 pgs: 158 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 626 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[30223]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[42243]: pgmap v485: 158 pgs: 158 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 626 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-24T12:03:12.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:12.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:12 smithi064 ceph-mon[42243]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T12:03:12.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:12 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T12:03:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:12 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T12:03:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:12 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:03:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:12 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:03:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:12 smithi193 ceph-mon[33593]: pgmap v485: 158 pgs: 158 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 626 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-24T12:03:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:12 smithi193 ceph-mon[33593]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T12:03:12.606 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:12.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:12.643 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:03:12.707 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:12.742 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:12.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:12.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:12.887 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:12.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:12.989 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:13.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 17 16 39164 39148 143.904 114.062 0.00796049 0.00694365 2024-03-24T12:03:13.034 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:13.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:13.082 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:13.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:13.175 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:13.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:13.253 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:03:13.262 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 1 pg_num to 21 2024-03-24T12:03:13.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:13.321 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:13.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:13.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:13.466 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:13.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:13.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: 5.17 scrub starts 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: 5.17 scrub ok 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1827239056' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: from='client.35321 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2866910456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: 5.17 scrub starts 2024-03-24T12:03:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: 5.17 scrub ok 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1827239056' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: from='client.35321 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:13 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2866910456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:13.567 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: 5.17 scrub starts 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: 5.17 scrub ok 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1827239056' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: from='client.35321 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T12:03:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:13 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2866910456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:13.603 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:13.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:13.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:13.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:13.746 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:13.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:13.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:13.884 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:13.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:13.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:14.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 18 16 40930 40914 142.041 110.375 0.00610937 0.00703432 2024-03-24T12:03:14.026 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:14.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:14.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:14.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:14.167 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:14.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:14.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 5.1 scrub starts 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 5.1 scrub ok 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T12:03:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: from='client.35321 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2866910456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: pgmap v488: 158 pgs: 158 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 5.1e scrub starts 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[30223]: 5.1e scrub ok 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T12:03:14.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 5.1 scrub starts 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 5.1 scrub ok 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: from='client.35321 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2866910456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: pgmap v488: 158 pgs: 158 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 5.1e scrub starts 2024-03-24T12:03:14.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:14 smithi064 ceph-mon[42243]: 5.1e scrub ok 2024-03-24T12:03:14.312 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:14.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:14.414 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:14.450 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:14.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:14.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:14.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 5.1 scrub starts 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 5.1 scrub ok 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: from='client.35321 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-24T12:03:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2866910456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:14.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T12:03:14.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: pgmap v488: 158 pgs: 158 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-24T12:03:14.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T12:03:14.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 5.1e scrub starts 2024-03-24T12:03:14.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:14 smithi193 ceph-mon[33593]: 5.1e scrub ok 2024-03-24T12:03:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:14.593 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:14.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:14.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:14.736 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:14.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:14.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:14.891 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:14.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:14.993 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:15.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 19 16 42764 42748 140.598 114.625 0.00325437 0.00710627 2024-03-24T12:03:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:15.036 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:15.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:15.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:15.172 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:15.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:15.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:15.312 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:15.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:15.357 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:03:15.357 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:03:15.357 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-24T12:03:15.357 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[30223]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T12:03:15.357 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T12:03:15.357 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T12:03:15.357 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:03:15.357 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:03:15.358 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-24T12:03:15.358 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[42243]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T12:03:15.358 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T12:03:15.358 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:15 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T12:03:15.370 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:03:15.370 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:03:15.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:15.453 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:15.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:15.555 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:15 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:03:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:15 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:03:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:15 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-24T12:03:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:15 smithi193 ceph-mon[33593]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T12:03:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:15 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T12:03:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:15 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T12:03:15.586 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:15.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:15.688 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:15.721 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:15.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:15.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:15.864 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:15.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:15.966 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:16.002 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:16.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:16.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:03:16.005133+0000 min lat: 0.00138521 max lat: 0.190405 avg lat: 0.00720655 2024-03-24T12:03:16.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:03:16.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 20 16 44396 44380 138.667 102 0.00858502 0.00720655 2024-03-24T12:03:16.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:03:16.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:03:16.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:16.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:16.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:16.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:16.310 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:16.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:16.412 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:16.450 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:16.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:16.452 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:16.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:16.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: 2.9 deep-scrub starts 2024-03-24T12:03:16.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: 2.9 deep-scrub ok 2024-03-24T12:03:16.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T12:03:16.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T12:03:16.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: 1.8 scrub starts 2024-03-24T12:03:16.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: 1.8 scrub ok 2024-03-24T12:03:16.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T12:03:16.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T12:03:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T12:03:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: pgmap v491: 168 pgs: 10 unknown, 158 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-24T12:03:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:16 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T12:03:16.601 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:16.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:16.615 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:16.691 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: 2.9 deep-scrub starts 2024-03-24T12:03:16.691 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: 2.9 deep-scrub ok 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: 1.8 scrub starts 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: 1.8 scrub ok 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: pgmap v491: 168 pgs: 10 unknown, 158 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-24T12:03:16.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: 2.9 deep-scrub starts 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: 2.9 deep-scrub ok 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: 1.8 scrub starts 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: 1.8 scrub ok 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T12:03:16.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: pgmap v491: 168 pgs: 10 unknown, 158 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-24T12:03:16.694 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:16 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T12:03:16.704 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:16.745 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:16.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:16.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:16.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:16.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:16.982 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:17.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 21 16 45833 45817 136.341 89.8125 0.00640911 0.0073295 2024-03-24T12:03:17.017 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:17.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:17.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:17.159 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:17.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:17.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:17.305 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:17.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:17.407 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:17.444 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:17.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:17.512 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:03:17.545 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.6 scrub starts 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.6 scrub ok 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.4 scrub starts 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.4 scrub ok 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.d scrub starts 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.d scrub ok 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.1 scrub starts 2024-03-24T12:03:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: 1.1 scrub ok 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[30223]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:03:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.6 scrub starts 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.6 scrub ok 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.4 scrub starts 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.4 scrub ok 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:03:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:03:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.d scrub starts 2024-03-24T12:03:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.d scrub ok 2024-03-24T12:03:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.1 scrub starts 2024-03-24T12:03:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: 1.1 scrub ok 2024-03-24T12:03:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-24T12:03:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T12:03:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:17 smithi064 ceph-mon[42243]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T12:03:17.558 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:03:17.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:03:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.6 scrub starts 2024-03-24T12:03:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.6 scrub ok 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.4 scrub starts 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.4 scrub ok 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.d scrub starts 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.d scrub ok 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.1 scrub starts 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: 1.1 scrub ok 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T12:03:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:17 smithi193 ceph-mon[33593]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T12:03:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:17.587 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:17.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:17.602 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:03:17.645 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:03:17.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:17.689 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:03:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:17.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:17.729 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:03:17.766 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:03:17.815 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:03:17.823 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:03:17.823 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:03:17.823 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:03:17.823 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:03:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:03:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:03:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:03:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:03:17.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:17.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:17.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:18.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 22 16 47400 47384 134.595 97.9375 0.00449687 0.00742466 2024-03-24T12:03:18.017 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:18.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:18.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:18.157 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:18.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:18.259 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:18.302 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:18.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 1.0 deep-scrub starts 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 1.0 deep-scrub ok 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 1.5 scrub starts 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 1.5 scrub ok 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T12:03:18.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: pgmap v494: 168 pgs: 7 remapped+peering, 1 unknown, 160 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 18 KiB/s rd, 135 MiB/s wr, 2.19k op/s; 98 KiB/s, 0 objects/s recovering 2024-03-24T12:03:18.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:18.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2426339755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:18.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T12:03:18.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:18 smithi193 ceph-mon[33593]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T12:03:18.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:18.441 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:18.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:18.456 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:03:18.456 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:03:18.456 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.456 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35306.0:48157 9.6 9:631e9638:::benchmark_data_smithi064_148274_object48156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-24T12:03:18.456 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:18.449316+0000", 2024-03-24T12:03:18.456 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0057935690000000001, 2024-03-24T12:03:18.456 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0058358969999999996, 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35306", 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 48157 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449316+0000", 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.457 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449316+0000", 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449318+0000", 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.922e-06 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449371+0000", 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2889999999999997e-05 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.458 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449392+0000", 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1188000000000001e-05 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449400+0000", 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2770000000000005e-06 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449529+0000", 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012837800000000001 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449736+0000", 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000207042 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449966+0000", 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023048799999999999 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.460 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35306.0:48158 9.7 9:ea6711a7:::benchmark_data_smithi064_148274_object48157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:18.449425+0000", 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0056842480000000003, 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0058307009999999998, 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35306", 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 48158 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449425+0000", 2024-03-24T12:03:18.461 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449425+0000", 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449426+0000", 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5199999999999996e-07 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449455+0000", 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8606999999999999e-05 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449461+0000", 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7020000000000004e-06 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449467+0000", 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.57e-06 2024-03-24T12:03:18.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449656+0000", 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018876200000000001 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449793+0000", 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000137186 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.449937+0000", 2024-03-24T12:03:18.464 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014415399999999999 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452101+0000", 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021638249999999999 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452560+0000", 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00045947199999999999 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35306.0:48161 9.6 9:68738c48:::benchmark_data_smithi064_148274_object48160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:18.452108+0000", 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0030009680000000001, 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032481250000000001, 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35306", 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 48161 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:18.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452108+0000", 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452108+0000", 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452110+0000", 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5090000000000001e-06 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452173+0000", 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3272e-05 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452183+0000", 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9599999999999995e-06 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452191+0000", 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4800000000000004e-06 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.455313+0000", 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031221299999999999 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35306.0:48160 9.8 e262/246 9:1481614b:::benchmark_data_smithi064_148274_object48159:head v 262'2986, mlcod=262'2984)", 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:18.452406+0000", 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0027034329999999999, 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030190070000000002, 2024-03-24T12:03:18.469 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452406+0000", 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452406+0000", 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452406+0000", 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7300000000000001e-07 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452438+0000", 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1109999999999999e-05 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452444+0000", 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4980000000000003e-06 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:18.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452449+0000", 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3009999999999997e-06 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452492+0000", 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2945000000000003e-05 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.452510+0000", 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7448999999999998e-05 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop_reply(client.35306.0:48157 9.6 e262/246)", 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:18.454199+0000", 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.00091084800000000002, 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012882379999999999, 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "queued for pg", 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454199+0000", 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:18.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454199+0000", 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454200+0000", 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.127e-06 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454205+0000", 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8420000000000004e-06 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:18.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454214+0000", 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2369999999999998e-06 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454220+0000", 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4749999999999998e-06 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35306.0:48162 9.5 e262/246 9:aab246c6:::benchmark_data_smithi064_148274_object48161:head v 262'3060, mlcod=262'3058)", 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:18.454246+0000", 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.00086375700000000004, 2024-03-24T12:03:18.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001296721, 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454246+0000", 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454246+0000", 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454246+0000", 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3399999999999999e-07 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454275+0000", 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9102999999999999e-05 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454282+0000", 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7220000000000002e-06 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454287+0000", 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2809999999999999e-06 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454348+0000", 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0129e-05 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.454366+0000", 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8499e-05 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.478 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop_reply(client.35306.0:48158 9.7 e262/246)", 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:18.455321+0000", 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 4294967295.9997883, 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027755899999999998, 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "queued for pg", 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.455321+0000", 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.455321+0000", 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.455322+0000", 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0499999999999999e-06 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.455327+0000", 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0529999999999997e-06 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:18.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.455336+0000", 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.02e-06 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:18.455343+0000", 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0469999999999996e-06 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 7 2024-03-24T12:03:18.481 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:03:18.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:03:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:03:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 1.0 deep-scrub starts 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 1.0 deep-scrub ok 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 1.5 scrub starts 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 1.5 scrub ok 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: pgmap v494: 168 pgs: 7 remapped+peering, 1 unknown, 160 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 18 KiB/s rd, 135 MiB/s wr, 2.19k op/s; 98 KiB/s, 0 objects/s recovering 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2426339755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T12:03:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[30223]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 1.0 deep-scrub starts 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 1.0 deep-scrub ok 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 1.5 scrub starts 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 1.5 scrub ok 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T12:03:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T12:03:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: pgmap v494: 168 pgs: 7 remapped+peering, 1 unknown, 160 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 18 KiB/s rd, 135 MiB/s wr, 2.19k op/s; 98 KiB/s, 0 objects/s recovering 2024-03-24T12:03:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2426339755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T12:03:18.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:18 smithi064 ceph-mon[42243]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T12:03:18.576 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:18.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:18.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:18.714 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:18.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:18.816 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:18.854 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:18.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:18.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:18.993 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:18.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:19.003 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 23 16 49103 49087 133.37 106.438 0.00472077 0.00749235 2024-03-24T12:03:19.095 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:19.129 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:19.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:19.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:19.276 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:19.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:19.336 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T12:03:19.377 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:19.418 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:19.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:19.520 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:19.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 1.7 deep-scrub starts 2024-03-24T12:03:19.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 1.7 deep-scrub ok 2024-03-24T12:03:19.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T12:03:19.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T12:03:19.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 2.16 scrub starts 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 2.16 scrub ok 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 2.b deep-scrub starts 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: 2.b deep-scrub ok 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[30223]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 1.7 deep-scrub starts 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 1.7 deep-scrub ok 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T12:03:19.530 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T12:03:19.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T12:03:19.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T12:03:19.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 2.16 scrub starts 2024-03-24T12:03:19.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 2.16 scrub ok 2024-03-24T12:03:19.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 2.b deep-scrub starts 2024-03-24T12:03:19.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: 2.b deep-scrub ok 2024-03-24T12:03:19.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:03:19.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:19 smithi064 ceph-mon[42243]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T12:03:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:19.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:19.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 1.7 deep-scrub starts 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 1.7 deep-scrub ok 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 2.16 scrub starts 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 2.16 scrub ok 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 2.b deep-scrub starts 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: 2.b deep-scrub ok 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:03:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:19 smithi193 ceph-mon[33593]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T12:03:19.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:19.716 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:19.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:19.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:19.753 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:19.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:19.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:19.886 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:03:19.886 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:03:19.886 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-03-24T12:03:19.886 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T12:03:19.886 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T12:03:19.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-03-24T12:03:19.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:20.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:20.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:20.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 24 16 50591 50575 131.688 93 0.0112082 0.00758767 2024-03-24T12:03:20.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:20.143 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:20.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:20.244 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:20.283 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:20.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:20.339 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:20.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:20.419 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:20.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:20.520 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:20.555 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:20.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:20 smithi193 ceph-mon[33593]: pgmap v497: 168 pgs: 7 remapped+peering, 1 unknown, 160 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 18 KiB/s rd, 135 MiB/s wr, 2.19k op/s; 98 KiB/s, 0 objects/s recovering 2024-03-24T12:03:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:20 smithi193 ceph-mon[33593]: 6.18 deep-scrub starts 2024-03-24T12:03:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:20 smithi193 ceph-mon[33593]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T12:03:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:20 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1086612099' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:03:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:20 smithi193 ceph-mon[33593]: from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:03:20.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:20.697 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:20.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[30223]: pgmap v497: 168 pgs: 7 remapped+peering, 1 unknown, 160 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 18 KiB/s rd, 135 MiB/s wr, 2.19k op/s; 98 KiB/s, 0 objects/s recovering 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[30223]: 6.18 deep-scrub starts 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[30223]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1086612099' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[30223]: from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[42243]: pgmap v497: 168 pgs: 7 remapped+peering, 1 unknown, 160 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 18 KiB/s rd, 135 MiB/s wr, 2.19k op/s; 98 KiB/s, 0 objects/s recovering 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[42243]: 6.18 deep-scrub starts 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[42243]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T12:03:20.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1086612099' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:03:20.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:20 smithi064 ceph-mon[42243]: from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:03:20.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:20.831 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:03:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:20.839 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:20.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:20.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:20.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:21.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 25 16 52146 52130 130.307 97.1875 0.0102746 0.00766945 2024-03-24T12:03:21.078 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:21.122 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:21.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:21.223 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:21.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:03:21.265 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:21.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:21.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:21.414 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:21.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:21.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:21.551 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:21.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.d scrub starts 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 6.18 deep-scrub ok 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.d scrub ok 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 6.1f scrub ok 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.e scrub starts 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.e scrub ok 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.10 scrub starts 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.10 scrub ok 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.14 scrub starts 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.14 scrub ok 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.f scrub starts 2024-03-24T12:03:21.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.f scrub ok 2024-03-24T12:03:21.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.b scrub starts 2024-03-24T12:03:21.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: 1.b scrub ok 2024-03-24T12:03:21.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:21.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:03:21.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:21 smithi193 ceph-mon[33593]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T12:03:21.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:21.680 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.d scrub starts 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 6.18 deep-scrub ok 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.d scrub ok 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 6.1f scrub ok 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.e scrub starts 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.e scrub ok 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.10 scrub starts 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.10 scrub ok 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.14 scrub starts 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.14 scrub ok 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.f scrub starts 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.f scrub ok 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.b scrub starts 2024-03-24T12:03:21.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: 1.b scrub ok 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[30223]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.d scrub starts 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 6.18 deep-scrub ok 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.d scrub ok 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 6.1f scrub ok 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.e scrub starts 2024-03-24T12:03:21.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.e scrub ok 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.10 scrub starts 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.10 scrub ok 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.14 scrub starts 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.14 scrub ok 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.f scrub starts 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.f scrub ok 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.b scrub starts 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: 1.b scrub ok 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:03:21.683 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:21 smithi064 ceph-mon[42243]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T12:03:21.694 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:21.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:21.778 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:03:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:03:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T12:03:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T12:03:21.778 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:03:21.795 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:21.835 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:21.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:21.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:21.989 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:21.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:22.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 26 16 53623 53607 128.845 92.3125 0.00443795 0.00775635 2024-03-24T12:03:22.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:22.131 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:22.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:22.232 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:22.274 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:22.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:22.375 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:22.415 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:22.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:22.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:22.574 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:22.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:22 smithi193 ceph-mon[33593]: 1.13 scrub starts 2024-03-24T12:03:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:22 smithi193 ceph-mon[33593]: 1.13 scrub ok 2024-03-24T12:03:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:22 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T12:03:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:22 smithi193 ceph-mon[33593]: pgmap v500: 168 pgs: 1 active+clean+scrubbing, 5 remapped+peering, 1 unknown, 161 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 39 MiB/s wr, 619 op/s 2024-03-24T12:03:22.638 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T12:03:22.662 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:22 smithi064 ceph-mon[30223]: 1.13 scrub starts 2024-03-24T12:03:22.662 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:22 smithi064 ceph-mon[30223]: 1.13 scrub ok 2024-03-24T12:03:22.662 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:22 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T12:03:22.663 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:22 smithi064 ceph-mon[30223]: pgmap v500: 168 pgs: 1 active+clean+scrubbing, 5 remapped+peering, 1 unknown, 161 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 39 MiB/s wr, 619 op/s 2024-03-24T12:03:22.663 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:22 smithi064 ceph-mon[42243]: 1.13 scrub starts 2024-03-24T12:03:22.663 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:22 smithi064 ceph-mon[42243]: 1.13 scrub ok 2024-03-24T12:03:22.663 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:22 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T12:03:22.663 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:22 smithi064 ceph-mon[42243]: pgmap v500: 168 pgs: 1 active+clean+scrubbing, 5 remapped+peering, 1 unknown, 161 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 39 MiB/s wr, 619 op/s 2024-03-24T12:03:22.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:22.716 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:22.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:22.818 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:22.868 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:22.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:22.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:23.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:23.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 27 16 55079 55063 127.443 91 0.00977478 0.00784036 2024-03-24T12:03:23.028 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:23.071 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:23.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:23.153 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:23.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:23.255 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:23.291 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:23.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:23.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:23.434 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:23.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:23.536 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:23.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:23 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:03:23.550 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:23 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:03:23.550 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:23 smithi193 ceph-mon[33593]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T12:03:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:23.579 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:23.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:23.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:23.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:23 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:03:23.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:23 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:03:23.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:23 smithi064 ceph-mon[30223]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T12:03:23.695 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:23 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:03:23.695 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:23 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:03:23.695 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:23 smithi064 ceph-mon[42243]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T12:03:23.719 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:23.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:23.821 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:23.859 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:23.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:23.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:24.003 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:24.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:24.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 28 16 56425 56409 125.896 84.125 0.00704376 0.00793763 2024-03-24T12:03:24.098 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:03:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "" 2024-03-24T12:03:24.098 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:03:24.105 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:24.141 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:24.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:24.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:24.282 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:24.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:24.383 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:24.425 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:24.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:24.527 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:24.571 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:24.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:24.647 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:24 smithi193 ceph-mon[33593]: pgmap v502: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 19 KiB/s, 0 objects/s recovering 2024-03-24T12:03:24.647 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:24 smithi193 ceph-mon[33593]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-24T12:03:24.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:24.716 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:24.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:24.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:24 smithi064 ceph-mon[30223]: pgmap v502: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 19 KiB/s, 0 objects/s recovering 2024-03-24T12:03:24.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:24 smithi064 ceph-mon[30223]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-24T12:03:24.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:24 smithi064 ceph-mon[42243]: pgmap v502: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 19 KiB/s, 0 objects/s recovering 2024-03-24T12:03:24.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:24 smithi064 ceph-mon[42243]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-24T12:03:24.818 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:24.857 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:24.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:24.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:24.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:03:25.000 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:25.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:25.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 29 16 57957 57941 124.856 95.75 0.00800425 0.00800389 2024-03-24T12:03:25.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:25.137 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:25.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:25.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:25.286 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:25.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:25.353 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:144664 8.11 8:8e5fcc86:::benchmark_data_smithi064_120708_object22638:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.092521+0000", 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 152.25885674899999, 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0449925920000001, 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.354 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 144664 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.092521+0000", 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.092521+0000", 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.092523+0000", 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1210000000000001e-06 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.092527+0000", 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9790000000000004e-06 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.092537+0000", 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0208e-05 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.092546+0000", 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6320000000000003e-06 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.092682+0000", 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000135136 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.092690+0000", 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.799e-06 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.117671+0000", 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.024980311 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for missing object", 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.117797+0000", 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012593399999999999 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137383+0000", 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.019586723 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137513+0000", 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012974899999999999 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:144729 8.11 8:89657c24:::benchmark_data_smithi064_120708_object22703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.142797+0000", 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 152.208580431, 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.98706914800000001, 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.359 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 144729 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.142797+0000", 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.142797+0000", 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.142798+0000", 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.548e-06 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.142803+0000", 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7230000000000004e-06 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.142811+0000", 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9310000000000008e-06 2024-03-24T12:03:25.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.142818+0000", 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1910000000000003e-06 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.142869+0000", 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1106000000000002e-05 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.142878+0000", 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9819999999999997e-06 2024-03-24T12:03:25.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.123784+0000", 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.98090579499999997 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.123877+0000", 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3010000000000006e-05 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.124239+0000", 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000361734 2024-03-24T12:03:25.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.128644+0000", 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0044049620000000001 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.129653+0000", 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001009678 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.129669+0000", 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5038999999999999e-05 2024-03-24T12:03:25.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.129730+0000", 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1319e-05 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.129866+0000", 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013613000000000001 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:144743 8.11 8:8c6a227f:::benchmark_data_smithi064_120708_object22717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.154420+0000", 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 152.19695722200001, 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.97597256399999999, 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 144743 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.154420+0000", 2024-03-24T12:03:25.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.367 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.154420+0000", 2024-03-24T12:03:25.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.154421+0000", 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.035e-06 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.154433+0000", 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2309e-05 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.154444+0000", 2024-03-24T12:03:25.368 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0934999999999999e-05 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.154452+0000", 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4399999999999999e-06 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.154502+0000", 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.062e-05 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.154512+0000", 2024-03-24T12:03:25.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3319999999999998e-06 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.126939+0000", 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.97242679600000004 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.127067+0000", 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012804099999999999 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.127354+0000", 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028738599999999999 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.129878+0000", 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025245340000000002 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.129939+0000", 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0071999999999997e-05 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.371 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.130134+0000", 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019514099999999999 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.130272+0000", 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013878 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.130393+0000", 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000120143 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.372 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:144802 8.11 8:8a5aef31:::benchmark_data_smithi064_120708_object22776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.215254+0000", 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 152.13612315500001, 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92609139100000004, 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 144802 2024-03-24T12:03:25.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.215254+0000", 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.215254+0000", 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.215256+0000", 2024-03-24T12:03:25.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2800000000000002e-06 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.215263+0000", 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2339999999999999e-06 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.215277+0000", 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4623999999999999e-05 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.215286+0000", 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0699999999999996e-06 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.215356+0000", 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9877999999999997e-05 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.215366+0000", 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8320000000000001e-06 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.130607+0000", 2024-03-24T12:03:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91524097299999996 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.130727+0000", 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000120238 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.131008+0000", 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028030699999999998 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137450+0000", 2024-03-24T12:03:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0064421180000000001 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137491+0000", 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1564999999999999e-05 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141179+0000", 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003687446 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141236+0000", 2024-03-24T12:03:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7219999999999998e-05 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141345+0000", 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010960599999999999 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:144880 8.11 8:8dfdcec3:::benchmark_data_smithi064_120708_object22854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.286711+0000", 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 152.064666644, 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85483104099999996, 2024-03-24T12:03:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 144880 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.286711+0000", 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.286711+0000", 2024-03-24T12:03:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.286712+0000", 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3829999999999999e-06 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.286719+0000", 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.849e-06 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.286732+0000", 2024-03-24T12:03:25.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2877000000000001e-05 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.286740+0000", 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5310000000000001e-06 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.286805+0000", 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4478000000000002e-05 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.286812+0000", 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8569999999999997e-06 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.132695+0000", 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.84588344199999999 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.132812+0000", 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000116932 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.133062+0000", 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024995699999999998 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137454+0000", 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0043921170000000001 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137572+0000", 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000118236 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141391+0000", 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0038186019999999999 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141444+0000", 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.312e-05 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141542+0000", 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7659999999999997e-05 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.385 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145051 8.11 8:8d5a623c:::benchmark_data_smithi064_120708_object23025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.443579+0000", 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.90779835500001, 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.69814182499999999, 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145051 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.386 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.443579+0000", 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.443579+0000", 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.443580+0000", 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3340000000000001e-06 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.443587+0000", 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1470000000000004e-06 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.443597+0000", 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9890000000000004e-06 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.443605+0000", 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4000000000000003e-06 2024-03-24T12:03:25.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.443716+0000", 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000111109 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.443724+0000", 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.627e-06 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.134979+0000", 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.69125595100000004 2024-03-24T12:03:25.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.135090+0000", 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000110327 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.135320+0000", 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000230318 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137510+0000", 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002190127 2024-03-24T12:03:25.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141137+0000", 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036268910000000001 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141579+0000", 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00044216399999999999 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141626+0000", 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6968000000000002e-05 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141721+0000", 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4473000000000002e-05 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145071 8.11 8:8b0b88de:::benchmark_data_smithi064_120708_object23045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.464964+0000", 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.88641371599999, 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.68708107299999999, 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.392 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.393 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:25.394 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.394 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145071 2024-03-24T12:03:25.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.464964+0000", 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.464964+0000", 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.464965+0000", 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9180000000000001e-06 2024-03-24T12:03:25.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.464971+0000", 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6550000000000001e-06 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.464982+0000", 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1046999999999999e-05 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.464990+0000", 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3129999999999999e-06 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.465050+0000", 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9836000000000001e-05 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.465060+0000", 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7380000000000006e-06 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137623+0000", 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.67256261900000003 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.397 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137743+0000", 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000120588 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.137995+0000", 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025143700000000003 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141767+0000", 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003772344 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.151857+0000", 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010090419 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.151911+0000", 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3989000000000001e-05 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.151995+0000", 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3150000000000002e-05 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152045+0000", 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0019999999999999e-05 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145075 8.11 8:89e924b2:::benchmark_data_smithi064_120708_object23049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.466496+0000", 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.88488164699999, 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.68566944900000004, 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.400 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145075 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.466496+0000", 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.466496+0000", 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.466497+0000", 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.728e-06 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.466506+0000", 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9479999999999997e-06 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.466521+0000", 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5004999999999999e-05 2024-03-24T12:03:25.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.466530+0000", 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2689999999999999e-06 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.468605+0000", 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020757649999999998 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.468615+0000", 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9340000000000001e-06 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.139200+0000", 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.67058456899999996 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.139326+0000", 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012594599999999999 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.139556+0000", 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022998100000000001 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.404 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.146741+0000", 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.007185632 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.151901+0000", 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.005159584 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152054+0000", 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015275799999999999 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152115+0000", 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1140000000000001e-05 2024-03-24T12:03:25.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152165+0000", 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0189999999999999e-05 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145077 8.11 8:8d7f9e2d:::benchmark_data_smithi064_120708_object23051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.470423+0000", 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.88095386800001, 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.68207349799999994, 2024-03-24T12:03:25.406 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145077 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.470423+0000", 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.470423+0000", 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.470425+0000", 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8700000000000001e-06 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.470431+0000", 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.502e-06 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.470442+0000", 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1069e-05 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.470450+0000", 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7249999999999997e-06 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.470533+0000", 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3591999999999995e-05 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.470544+0000", 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.048e-05 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141812+0000", 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.67126797999999999 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.141922+0000", 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010993300000000001 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.142208+0000", 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028627399999999999 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152173+0000", 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0099655569999999999 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152369+0000", 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000195236 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152411+0000", 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2370999999999998e-05 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152461+0000", 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9827999999999998e-05 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152497+0000", 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6081000000000002e-05 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145107 8.11 8:8b33a7bc:::benchmark_data_smithi064_120708_object23081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-24T12:03:25.412 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:53.503573+0000", 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.84780396799999, 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.64909589599999995, 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145107 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.503573+0000", 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.503573+0000", 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.503575+0000", 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4610000000000001e-06 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.503579+0000", 2024-03-24T12:03:25.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8820000000000003e-06 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.503587+0000", 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2339999999999994e-06 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.503592+0000", 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6030000000000004e-06 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.503636+0000", 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3510999999999997e-05 2024-03-24T12:03:25.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:53.503642+0000", 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8119999999999996e-06 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.144243+0000", 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.64060163299999995 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.144358+0000", 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0001149 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.144647+0000", 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000288826 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152178+0000", 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0075309340000000004 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152537+0000", 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00035940199999999999 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152582+0000", 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4864999999999998e-05 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152635+0000", 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2939999999999998e-05 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.152669+0000", 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3893000000000002e-05 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145373 8.19 8:9c3cd734:::benchmark_data_smithi064_120708_object23347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.174387+0000", 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.17699015100001, 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.94946325399999998, 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145373 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.419 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.174387+0000", 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.174387+0000", 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.174388+0000", 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.24e-06 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.174393+0000", 2024-03-24T12:03:25.420 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7659999999999998e-06 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.174403+0000", 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5109999999999992e-06 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.174409+0000", 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6209999999999998e-06 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.174502+0000", 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3219000000000003e-05 2024-03-24T12:03:25.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.174518+0000", 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5913000000000002e-05 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.111119+0000", 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.93660073200000005 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.111246+0000", 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000126506 2024-03-24T12:03:25.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.111622+0000", 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037607299999999999 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.118847+0000", 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0072253839999999996 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.118884+0000", 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6717000000000001e-05 2024-03-24T12:03:25.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.123748+0000", 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048641489999999999 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.123803+0000", 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4565999999999999e-05 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.123850+0000", 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7856999999999999e-05 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.424 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145382 8.19 8:9eed9df7:::benchmark_data_smithi064_120708_object23356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.179909+0000", 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.17146870900001, 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95296704300000001, 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145382 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.425 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.179909+0000", 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.179909+0000", 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.179909+0000", 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0699999999999996e-07 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.179912+0000", 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7669999999999999e-06 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.179921+0000", 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0920000000000004e-06 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.179927+0000", 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.874e-06 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.427 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.179973+0000", 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6119999999999999e-05 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.179981+0000", 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3450000000000006e-06 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.113256+0000", 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.93327433400000004 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.113363+0000", 2024-03-24T12:03:25.428 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107136 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.113584+0000", 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022072500000000001 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.118917+0000", 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0053337059999999997 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.118954+0000", 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6188e-05 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.429 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132795+0000", 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013841561 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132833+0000", 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7623999999999998e-05 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132876+0000", 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2864000000000002e-05 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.430 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145393 8.19 8:9ce4919b:::benchmark_data_smithi064_120708_object23367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.193087+0000", 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.15829004, 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.93986989099999996, 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145393 2024-03-24T12:03:25.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.193087+0000", 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.193087+0000", 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.433 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.193089+0000", 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4130000000000001e-06 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.193092+0000", 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.641e-06 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.193100+0000", 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0620000000000001e-06 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.434 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.193106+0000", 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.84e-06 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.193158+0000", 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1947000000000003e-05 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.193174+0000", 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5532e-05 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.435 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.116673+0000", 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.923499662 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.116801+0000", 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012754500000000001 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.117026+0000", 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000225298 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.436 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.123882+0000", 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068560929999999997 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132778+0000", 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0088960259999999996 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132885+0000", 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000106853 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.437 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132929+0000", 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3776000000000002e-05 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132957+0000", 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8203000000000001e-05 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145398 8.19 8:9cd5a892:::benchmark_data_smithi064_120708_object23372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.200245+0000", 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.15113181999999, 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.93282656600000002, 2024-03-24T12:03:25.438 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145398 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.200245+0000", 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.200245+0000", 2024-03-24T12:03:25.439 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.200247+0000", 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6700000000000001e-06 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.200252+0000", 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9450000000000001e-06 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.200262+0000", 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0370999999999999e-05 2024-03-24T12:03:25.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.200268+0000", 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0639999999999999e-06 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.200318+0000", 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9641000000000001e-05 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.200327+0000", 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3009999999999995e-06 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.118980+0000", 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91865278100000003 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.119075+0000", 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4501999999999997e-05 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.119329+0000", 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025436499999999999 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.124012+0000", 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0046830129999999998 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132965+0000", 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0089527359999999993 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133004+0000", 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9487999999999999e-05 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.443 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133034+0000", 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9872000000000001e-05 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133072+0000", 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7817000000000001e-05 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145452 8.19 8:9a3797c9:::benchmark_data_smithi064_120708_object23426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.250767+0000", 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.10061044899999, 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88658575699999997, 2024-03-24T12:03:25.444 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145452 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.250767+0000", 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.250767+0000", 2024-03-24T12:03:25.445 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.250769+0000", 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9850000000000001e-06 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.250775+0000", 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6680000000000004e-06 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.250798+0000", 2024-03-24T12:03:25.446 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2782e-05 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.250809+0000", 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0341e-05 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.250870+0000", 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1179000000000005e-05 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.250885+0000", 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4963999999999999e-05 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.122070+0000", 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.87118556000000003 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.122188+0000", 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000117847 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.122478+0000", 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00029027300000000002 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.448 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.127374+0000", 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048955769999999999 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.132971+0000", 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055971129999999999 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137260+0000", 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042888070000000004 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137310+0000", 2024-03-24T12:03:25.449 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0476999999999998e-05 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137353+0000", 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2184000000000001e-05 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145471 8.19 8:9ee54e77:::benchmark_data_smithi064_120708_object23445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.270839+0000", 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 151.080537822, 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.86665825699999999, 2024-03-24T12:03:25.450 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145471 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.270839+0000", 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.451 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.270839+0000", 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.270841+0000", 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.145e-06 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.270845+0000", 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3900000000000003e-06 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.452 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.270856+0000", 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.096e-05 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.270863+0000", 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1550000000000001e-06 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.270913+0000", 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0015999999999997e-05 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.453 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.270921+0000", 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0229999999999992e-06 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.124054+0000", 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85313299399999998 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.124123+0000", 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8833999999999999e-05 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.454 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.124289+0000", 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016633800000000001 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.127427+0000", 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031372470000000001 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133118+0000", 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0056912040000000001 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137426+0000", 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0043083310000000003 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.455 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137466+0000", 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9907999999999998e-05 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137498+0000", 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1711999999999997e-05 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145558 8.19 8:9a6854a4:::benchmark_data_smithi064_120708_object23532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.456 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.422665+0000", 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 150.92871203300001, 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.71493597799999997, 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145558 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.422665+0000", 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.457 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.422665+0000", 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.422667+0000", 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7829999999999999e-06 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.422672+0000", 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7360000000000001e-06 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.458 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.422683+0000", 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0893999999999999e-05 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.422690+0000", 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2169999999999997e-06 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.422744+0000", 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4381e-05 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.422753+0000", 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2390000000000008e-06 2024-03-24T12:03:25.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.125550+0000", 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.70279618899999996 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.125653+0000", 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010375 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.125885+0000", 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000231506 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133108+0000", 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0072229460000000001 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133124+0000", 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5968999999999999e-05 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137533+0000", 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0044086689999999996 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137571+0000", 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8507999999999998e-05 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137601+0000", 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0190999999999999e-05 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145579 8.19 8:98a16a0c:::benchmark_data_smithi064_120708_object23553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.462 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.450707+0000", 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 150.90067069, 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.74561089899999999, 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145579 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.450707+0000", 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.463 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.450707+0000", 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.450709+0000", 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1689999999999999e-06 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.450718+0000", 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3540000000000006e-06 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.464 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.450732+0000", 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3725e-05 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.450738+0000", 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.959e-06 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.450800+0000", 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2543000000000001e-05 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.465 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.450813+0000", 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2391999999999999e-05 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.127466+0000", 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.67665321899999997 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.127579+0000", 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011339 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.127805+0000", 2024-03-24T12:03:25.466 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022552100000000001 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133130+0000", 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0053253679999999996 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133176+0000", 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6137000000000002e-05 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.196204+0000", 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.063027454999999996 2024-03-24T12:03:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.196265+0000", 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0760000000000001e-05 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.196317+0000", 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2907e-05 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:145587 8.19 8:9b205386:::benchmark_data_smithi064_120708_object23561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:54.466259+0000", 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 150.885118092, 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.73025868900000002, 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 145587 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.466259+0000", 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.466259+0000", 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.466262+0000", 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3369999999999998e-06 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.466267+0000", 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7019999999999998e-06 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.466281+0000", 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4099e-05 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.466290+0000", 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3650000000000004e-06 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.466358+0000", 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8169000000000004e-05 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:54.466368+0000", 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0505e-05 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.129496+0000", 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.66312807699999998 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.129622+0000", 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012590600000000001 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.131104+0000", 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014813649999999999 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.133136+0000", 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002032435 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.137390+0000", 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042539359999999998 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.196420+0000", 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.059030394999999999 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.196476+0000", 2024-03-24T12:03:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5121000000000003e-05 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:55.196518+0000", 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2277000000000002e-05 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35306.0:29996 9.7 9:e3871b11:::benchmark_data_smithi064_148274_object29995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:07.871140+0000", 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 17.480237536000001, 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.18415999799999999, 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35306", 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 29996 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.871140+0000", 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.871140+0000", 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.871141+0000", 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4959999999999999e-06 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.871197+0000", 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6045999999999998e-05 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.871207+0000", 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8120000000000002e-06 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:25.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.871215+0000", 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.661e-06 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.871813+0000", 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00059836100000000005 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.871944+0000", 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000130576 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.872074+0000", 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013014 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.873388+0000", 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013140000000000001 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:07.875522+0000", 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021344850000000002 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:25.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:08.055197+0000", 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17967482800000001 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:08.055257+0000", 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9868999999999999e-05 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:08.055300+0000", 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2724e-05 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:25.480 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:03:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:25.518 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:25.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:25.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:25.656 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:25.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:25.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:25.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:25.894 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:25.929 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:25.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:25.933 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:26.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 30 16 59407 59391 123.715 90.625 0.00832357 0.00807757 2024-03-24T12:03:26.030 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:26.070 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:26.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:26.171 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:26.206 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:26.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:26.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:26.341 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:26.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:26.443 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:26.482 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:26.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:26.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:26.586 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:03:26.609 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:26.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:26.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:26.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:26.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:26 smithi064 ceph-mon[30223]: pgmap v503: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 14 KiB/s, 0 objects/s recovering 2024-03-24T12:03:26.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:26 smithi064 ceph-mon[42243]: pgmap v503: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 14 KiB/s, 0 objects/s recovering 2024-03-24T12:03:26.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:26 smithi193 ceph-mon[33593]: pgmap v503: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 14 KiB/s, 0 objects/s recovering 2024-03-24T12:03:26.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:26.877 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:26.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:26.948 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:26.979 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:27.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 31 16 60580 60564 122.089 73.3125 0.00844145 0.00818555 2024-03-24T12:03:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:27.028 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:27.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:27.129 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:27.165 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:27.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:27.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:27.302 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:27.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:27.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:27.426 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:27.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:27.528 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:27.564 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:27.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:27.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:27.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:27.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:27.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:03:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:27.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:03:27.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:03:27.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-mon[30223]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T12:03:27.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:27.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:03:27.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:03:27.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-mon[42243]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T12:03:27.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:27.805 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:27.834 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:27.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:27.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:03:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:03:27.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:27 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:03:27.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:27 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:03:27.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:27 smithi193 ceph-mon[33593]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T12:03:27.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:27.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:27.971 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:27.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:28.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 32 16 61831 61815 120.717 78.1875 0.00594336 0.00827914 2024-03-24T12:03:28.073 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:28.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:28.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:28.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:28.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:28.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:28.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:28.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:28.374 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:28.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:28.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:28.502 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:28.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:28.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:28.642 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:28.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:28.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:28.768 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:28.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:28.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:28.885 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:03:28.909 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:28.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:28.930 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:03:28.969 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:03:29.005 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 33 16 63006 62990 119.284 73.4375 0.00921184 0.00837818 2024-03-24T12:03:29.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:29.011 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:03:29.045 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:29.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:29.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:29.046 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:03:29.046 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:28 smithi064 ceph-mon[30223]: pgmap v505: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s; 13 KiB/s, 0 objects/s recovering 2024-03-24T12:03:29.047 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:29.047 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:28 smithi064 ceph-mon[42243]: pgmap v505: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s; 13 KiB/s, 0 objects/s recovering 2024-03-24T12:03:29.047 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:28 smithi193 ceph-mon[33593]: pgmap v505: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s; 13 KiB/s, 0 objects/s recovering 2024-03-24T12:03:29.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:29.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:03:29.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:29.159 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:03:29.191 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:29.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:29.204 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:03:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:03:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:03:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:03:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:03:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:03:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:03:29.213 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:03:29.213 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:03:29.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:29.334 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:29.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:29.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:29.459 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:29.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:29.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:29.585 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:29.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:29.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:29.710 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:29.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:29.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:29.835 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:29.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:29.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:29.951 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:29 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1427282150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:29.951 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:29 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1427282150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:29.974 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:29.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:30.004 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 34 16 63884 63868 117.39 54.875 0.00758132 0.00851353 2024-03-24T12:03:30.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:30.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:29 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1427282150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:30.113 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:30.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:30.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:30.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:30.339 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:30.378 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:30.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:30.432 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:03:30.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-24T12:03:30.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:30.511 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:30.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:30.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:30.658 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:30.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:30.759 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:30.793 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:30.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:30.893 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:30.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:30.930 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:30.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:31.005 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 35 16 65119 65103 116.241 77.1875 0.00583293 0.00859772 2024-03-24T12:03:31.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:31.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:30 smithi064 ceph-mon[30223]: pgmap v506: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s rd, 97 MiB/s wr, 1.54k op/s; 30 KiB/s, 0 objects/s recovering 2024-03-24T12:03:31.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:30 smithi064 ceph-mon[42243]: pgmap v506: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s rd, 97 MiB/s wr, 1.54k op/s; 30 KiB/s, 0 objects/s recovering 2024-03-24T12:03:31.057 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:31.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:31.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:30 smithi193 ceph-mon[33593]: pgmap v506: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s rd, 97 MiB/s wr, 1.54k op/s; 30 KiB/s, 0 objects/s recovering 2024-03-24T12:03:31.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:31.191 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:31.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:31.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:31.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:31.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:31.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:31.347 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:31.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:31.456 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:31.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:31.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:31.583 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:31.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:31.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:31.725 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:31.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:31.827 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:31.852 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:31.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:31.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:31.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:31.988 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:31.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:32.005 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 36 16 66292 66276 115.048 73.3125 0.00358924 0.00868001 2024-03-24T12:03:32.089 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:32.123 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:32.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:32.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:32.267 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:32.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:32.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:32.409 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:32.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:32.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:32.549 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:32.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:32.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:32.673 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:32.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:32.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:32.775 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:32.823 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:32.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:32.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:32.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:32.963 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:32.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:33.005 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 37 16 67419 67403 113.842 70.4375 0.00810565 0.00877896 2024-03-24T12:03:33.049 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:32 smithi193 ceph-mon[33593]: pgmap v507: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 569 B/s rd, 54 MiB/s wr, 868 op/s 2024-03-24T12:03:33.049 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:32 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2101692427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:33.049 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:32 smithi193 ceph-mon[33593]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:33.065 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:33.104 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:33.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:33.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:33.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:33.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:33.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:32 smithi064 ceph-mon[30223]: pgmap v507: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 569 B/s rd, 54 MiB/s wr, 868 op/s 2024-03-24T12:03:33.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:32 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2101692427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:33.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:32 smithi064 ceph-mon[30223]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:33.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:32 smithi064 ceph-mon[42243]: pgmap v507: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 569 B/s rd, 54 MiB/s wr, 868 op/s 2024-03-24T12:03:33.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:32 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2101692427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:33.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:32 smithi064 ceph-mon[42243]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:33.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:03:33.351 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:33.381 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:33.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:33.400 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:03:33.400 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:03:33.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:33.521 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:33.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:33.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:33.657 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:33.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:33.759 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:33.792 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:33.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:33.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:33.919 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:33.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:34.005 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 38 16 68681 68665 112.922 78.875 0.0111964 0.00885067 2024-03-24T12:03:34.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:34.057 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:34.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:34.158 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:34.193 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:34.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:34.295 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:34.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:34.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[30223]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:34.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[30223]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T12:03:34.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:03:34.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:03:34.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:34.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[42243]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:34.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[42243]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T12:03:34.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:03:34.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:33 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:03:34.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:33 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:34.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:33 smithi193 ceph-mon[33593]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:34.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:33 smithi193 ceph-mon[33593]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T12:03:34.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:33 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:03:34.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:33 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:03:34.336 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:34.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:34.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:03:34.438 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:34.490 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:34.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:34.562 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:03:34.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:34.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:34.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:34.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:34.795 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:34.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:34.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:34.835 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:34.897 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:34.932 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:34.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:35.005 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 39 16 69975 69959 112.1 80.875 0.00829899 0.00891507 2024-03-24T12:03:35.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:35.072 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:35.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:35.174 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:35.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:35.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:35.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:34 smithi064 ceph-mon[30223]: pgmap v509: 168 pgs: 168 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-24T12:03:35.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:34 smithi064 ceph-mon[30223]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T12:03:35.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:34 smithi064 ceph-mon[42243]: pgmap v509: 168 pgs: 168 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-24T12:03:35.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:34 smithi064 ceph-mon[42243]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T12:03:35.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:35.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:34 smithi193 ceph-mon[33593]: pgmap v509: 168 pgs: 168 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-24T12:03:35.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:34 smithi193 ceph-mon[33593]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T12:03:35.356 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:35.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:35.458 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:35.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:35.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:35.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:35.641 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:35.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:35.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:35.779 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:35.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:35.880 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:35.917 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:35.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:35.928 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:36.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:03:36.008630+0000 min lat: 0.00138521 max lat: 0.190405 avg lat: 0.00897434 2024-03-24T12:03:36.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:03:36.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 40 16 71295 71279 111.356 82.5 0.0128503 0.00897434 2024-03-24T12:03:36.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:36.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:36.054 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:36.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:36.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:36.198 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:36.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:36.301 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:36.335 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:36.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:36.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:36.484 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:36.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:36.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:36.619 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:36.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:36.720 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:36.761 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:36.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:36.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:36.902 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:36.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:36.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:37.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:37.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 41 16 72676 72660 110.745 86.3125 0.0110223 0.00902474 2024-03-24T12:03:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:37.059 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:37.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:37.142 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:36 smithi064 ceph-mon[30223]: pgmap v511: 168 pgs: 168 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 919 op/s 2024-03-24T12:03:37.143 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:36 smithi064 ceph-mon[42243]: pgmap v511: 168 pgs: 168 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 919 op/s 2024-03-24T12:03:37.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:37.193 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:37.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:37.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:37.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:36 smithi193 ceph-mon[33593]: pgmap v511: 168 pgs: 168 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 919 op/s 2024-03-24T12:03:37.338 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:37.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:37.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:03:37.434 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:03:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:03:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:37.440 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:37.472 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:37.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:37.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:37.606 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:37.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:37.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:03:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:03:37.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:37.766 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:37.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:37.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:37.908 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-24T12:03:37.912 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:37.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:38.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 42 16 73841 73825 109.841 72.8125 0.0215472 0.00909897 2024-03-24T12:03:38.014 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:38.055 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:38.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:38.125 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:03:38.125 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:03:38.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:38.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:37 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2725222922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:03:38.196 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:38.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:38.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:38.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:37 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2725222922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:03:38.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:37 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2725222922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:03:38.336 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:38.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:38.437 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:38.592 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:38.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:38.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:38.732 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:38.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:38.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:38.855 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:03:38.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-24T12:03:38.882 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:38.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:38.984 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:39.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 43 16 75202 75186 109.265 85.0625 0.0143513 0.00914719 2024-03-24T12:03:39.024 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:39.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:38 smithi193 ceph-mon[33593]: pgmap v512: 168 pgs: 168 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T12:03:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2725222922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:03:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:38 smithi193 ceph-mon[33593]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T12:03:39.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:39.129 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:39.161 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:39.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:39.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:38 smithi064 ceph-mon[42243]: pgmap v512: 168 pgs: 168 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T12:03:39.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:39.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2725222922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:03:39.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:38 smithi064 ceph-mon[42243]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T12:03:39.198 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:03:39.200 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:38 smithi064 ceph-mon[30223]: pgmap v512: 168 pgs: 168 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T12:03:39.200 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:39.200 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2725222922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:03:39.200 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:38 smithi064 ceph-mon[30223]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T12:03:39.221 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:39.245 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:03:39.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:39.283 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:03:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:39.314 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:39.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:39.330 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:03:39.368 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:03:39.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:39.417 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:03:39.461 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:03:39.470 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:39.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:39.508 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:03:39.517 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:03:39.517 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:03:39.517 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:03:39.517 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:03:39.517 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:03:39.517 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:03:39.517 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:03:39.517 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:03:39.572 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:39.610 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:39.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:39.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:39.749 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:39.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:39.851 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:39.890 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:39.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:39.992 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:40.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 44 16 76451 76435 108.556 78.0625 0.0137149 0.00920721 2024-03-24T12:03:40.026 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:40.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:40.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:39 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1667196682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:40.128 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:40.169 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:40.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:40.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:39 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1667196682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:39 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1667196682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:40.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:40.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:40.448 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:40.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:40.550 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:40.600 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:40.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:40.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:40.750 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:40.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:40.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:40.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:40.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:40.998 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:41.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 45 16 77798 77782 108.014 84.1875 0.0195072 0.00925311 2024-03-24T12:03:41.035 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:41.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:41.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:40 smithi064 ceph-mon[30223]: pgmap v514: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 789 B/s rd, 61 MiB/s wr, 974 op/s 2024-03-24T12:03:41.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:40 smithi064 ceph-mon[42243]: pgmap v514: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 789 B/s rd, 61 MiB/s wr, 974 op/s 2024-03-24T12:03:41.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:40 smithi193 ceph-mon[33593]: pgmap v514: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 789 B/s rd, 61 MiB/s wr, 974 op/s 2024-03-24T12:03:41.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:41.179 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:41.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:41.281 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:41.288 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:03:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T12:03:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T12:03:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:03:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28186 7.7 7:e54252ff:::benchmark_data_smithi064_87013_object28185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T12:03:41.288 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.185466+0000", 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 534.09999644699997, 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16022612, 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28186 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185466+0000", 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185466+0000", 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185468+0000", 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.886e-06 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185507+0000", 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9397999999999998e-05 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185517+0000", 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.005e-05 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185524+0000", 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.328e-06 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185583+0000", 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8666999999999999e-05 2024-03-24T12:03:41.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185742+0000", 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015933700000000001 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185936+0000", 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019364800000000001 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188116+0000", 2024-03-24T12:03:41.292 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021797510000000002 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.339991+0000", 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.151875807 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345426+0000", 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054349259999999996 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345654+0000", 2024-03-24T12:03:41.293 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022796399999999999 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345692+0000", 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7357999999999997e-05 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28187 7.7 7:ebde3806:::benchmark_data_smithi064_87013_object28186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.185544+0000", 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 534.099918452, 2024-03-24T12:03:41.294 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16043975899999999, 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28187 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185544+0000", 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185544+0000", 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185544+0000", 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5899999999999995e-07 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185630+0000", 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5143999999999997e-05 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185637+0000", 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9820000000000002e-06 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.185641+0000", 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.882e-06 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187084+0000", 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014425 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.297 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187210+0000", 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012592300000000001 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.187345+0000", 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000134647 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188284+0000", 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000939368 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.340027+0000", 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15174342199999999 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345915+0000", 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0058873720000000001 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345958+0000", 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3513000000000002e-05 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.299 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.345983+0000", 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5147000000000002e-05 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28192 7.c 7:3adf894d:::benchmark_data_smithi064_87013_object28191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.188462+0000", 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 534.09699989000001, 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15559229499999999, 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.300 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28192 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188462+0000", 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188462+0000", 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188464+0000", 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6050000000000001e-06 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188526+0000", 2024-03-24T12:03:41.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2663000000000006e-05 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188536+0000", 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5349999999999993e-06 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188542+0000", 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9179999999999999e-06 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188599+0000", 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6616000000000001e-05 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188689+0000", 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0540999999999999e-05 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.188776+0000", 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6911999999999997e-05 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.189737+0000", 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00096098199999999998 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.190043+0000", 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00030576399999999998 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.343860+0000", 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15381721500000001 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.344001+0000", 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014119999999999999 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.344054+0000", 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3344000000000003e-05 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28593 7.c 7:30603fbe:::benchmark_data_smithi064_87013_object28592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.547265+0000", 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 533.73819742199998, 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15797111, 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28593 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547265+0000", 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547265+0000", 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547266+0000", 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.463e-06 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547316+0000", 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9428999999999997e-05 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547326+0000", 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0365e-05 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547333+0000", 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0029999999999997e-06 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547374+0000", 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1217999999999997e-05 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547462+0000", 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7540999999999994e-05 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547546+0000", 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4585000000000004e-05 2024-03-24T12:03:41.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548738+0000", 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0011919260000000001 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548948+0000", 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000209495 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705156+0000", 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15620814699999999 2024-03-24T12:03:41.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705194+0000", 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8552e-05 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705236+0000", 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1386e-05 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.312 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28597 7.7 7:e8c53c18:::benchmark_data_smithi064_87013_object28596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.547932+0000", 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 533.73752999299995, 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15730358899999999, 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28597 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547932+0000", 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547932+0000", 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547933+0000", 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.251e-06 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547980+0000", 2024-03-24T12:03:41.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.71e-05 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547991+0000", 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.059e-05 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.547999+0000", 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7309999999999993e-06 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548053+0000", 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4187999999999997e-05 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548136+0000", 2024-03-24T12:03:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3467000000000006e-05 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.548231+0000", 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4534e-05 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.549937+0000", 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017055709999999999 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552361+0000", 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024243770000000001 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705177+0000", 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.152815912 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705208+0000", 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1489999999999998e-05 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.705236+0000", 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7378000000000001e-05 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28602 7.c 7:38c9ee92:::benchmark_data_smithi064_87013_object28601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.550423+0000", 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 533.73503936500003, 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.161645607, 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28602 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550423+0000", 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550423+0000", 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550424+0000", 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4309999999999999e-06 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550479+0000", 2024-03-24T12:03:41.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5044999999999999e-05 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550488+0000", 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7180000000000002e-06 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550495+0000", 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4749999999999996e-06 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550540+0000", 2024-03-24T12:03:41.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4774000000000002e-05 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550626+0000", 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5971999999999996e-05 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.550717+0000", 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0962000000000001e-05 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.551765+0000", 2024-03-24T12:03:41.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0010477710000000001 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552014+0000", 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024906499999999997 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.711966+0000", 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15995161299999999 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712024+0000", 2024-03-24T12:03:41.324 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8112999999999997e-05 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712068+0000", 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4668e-05 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:28607 7.c 7:3d17d51d:::benchmark_data_smithi064_87013_object28606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:54:47.552587+0000", 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 533.73287499000003, 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15961876999999999, 2024-03-24T12:03:41.325 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 28607 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552587+0000", 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552587+0000", 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552589+0000", 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.618e-06 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552646+0000", 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7494999999999998e-05 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552658+0000", 2024-03-24T12:03:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1987999999999999e-05 2024-03-24T12:03:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552666+0000", 2024-03-24T12:03:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3370000000000002e-06 2024-03-24T12:03:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552710+0000", 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4261999999999997e-05 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552807+0000", 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7011999999999999e-05 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.552891+0000", 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4592000000000006e-05 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.554174+0000", 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001282748 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.554186+0000", 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1783999999999999e-05 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712113+0000", 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15792745499999999 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712164+0000", 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0955999999999998e-05 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:54:47.712206+0000", 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1523000000000002e-05 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:65830 7.c 7:35fc51e6:::benchmark_data_smithi064_87013_object65829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:12.717425+0000", 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 508.56803684599998, 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14593989900000001, 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 65830 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717425+0000", 2024-03-24T12:03:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717425+0000", 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717427+0000", 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.796e-06 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717482+0000", 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5118999999999999e-05 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717499+0000", 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6694999999999999e-05 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717507+0000", 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.49e-06 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.335 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717553+0000", 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5330999999999997e-05 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717647+0000", 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4835999999999998e-05 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.717743+0000", 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5464000000000006e-05 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.719456+0000", 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017131799999999999 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.719864+0000", 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00040831899999999998 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.863247+0000", 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14338300300000001 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.863312+0000", 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4553000000000003e-05 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:12.863365+0000", 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3112999999999998e-05 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67059 7.7 7:e00040fd:::benchmark_data_smithi064_87013_object67058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.338 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:13.875611+0000", 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.40985094899997, 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.18307747699999999, 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67059 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875611+0000", 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875611+0000", 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875612+0000", 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0809999999999999e-06 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875685+0000", 2024-03-24T12:03:41.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2611e-05 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875694+0000", 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3519999999999996e-06 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875700+0000", 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0619999999999997e-06 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875751+0000", 2024-03-24T12:03:41.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0868999999999997e-05 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.875873+0000", 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000122045 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876004+0000", 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000131027 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878138+0000", 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021343310000000002 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.881941+0000", 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0038020810000000001 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.058578+0000", 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.176637233 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.058641+0000", 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2897000000000004e-05 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.058689+0000", 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7888e-05 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67060 7.c 7:37ae5022:::benchmark_data_smithi064_87013_object67059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:13.876645+0000", 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.40881749099998, 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17569818300000001, 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67060 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876645+0000", 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876645+0000", 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876646+0000", 2024-03-24T12:03:41.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1310000000000001e-06 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876726+0000", 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0137999999999997e-05 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876738+0000", 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2257e-05 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876746+0000", 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7700000000000001e-06 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876803+0000", 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7612999999999999e-05 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876893+0000", 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0004000000000001e-05 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.348 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.876978+0000", 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4778999999999996e-05 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878831+0000", 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018530280000000001 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.879123+0000", 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00029211500000000002 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.052232+0000", 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.173108807 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.052302+0000", 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0028000000000001e-05 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.052343+0000", 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0513000000000003e-05 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.350 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67063 7.7 7:e32fc8d7:::benchmark_data_smithi064_87013_object67062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:13.878827+0000", 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.40663539299999, 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.182956279, 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67063 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878827+0000", 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878827+0000", 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878828+0000", 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.793e-06 2024-03-24T12:03:41.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878863+0000", 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4944999999999998e-05 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878874+0000", 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0152e-05 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878882+0000", 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9619999999999993e-06 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.878920+0000", 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8940999999999999e-05 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.879015+0000", 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4102000000000001e-05 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.879142+0000", 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012751400000000001 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:13.880753+0000", 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001610956 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.024206+0000", 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.143453308 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.061630+0000", 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037423374000000002 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.061742+0000", 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011199700000000001 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.061783+0000", 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1235000000000001e-05 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67336 7.d 7:b31b3432:::benchmark_data_smithi064_87013_object67335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.356 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.241068+0000", 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.04439389999999, 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.23462398400000001, 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67336 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241068+0000", 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241068+0000", 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241070+0000", 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.702e-06 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241167+0000", 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6700000000000006e-05 2024-03-24T12:03:41.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241181+0000", 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4052e-05 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241200+0000", 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9684e-05 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241253+0000", 2024-03-24T12:03:41.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2281000000000003e-05 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241375+0000", 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012239799999999999 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.241535+0000", 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016008099999999999 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244010+0000", 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024753790000000002 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244856+0000", 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00084533399999999999 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475606+0000", 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.230749913 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475658+0000", 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2725000000000001e-05 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475692+0000", 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3735000000000001e-05 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67345 7.7 7:ed1efbc8:::benchmark_data_smithi064_87013_object67344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.244668+0000", 2024-03-24T12:03:41.362 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.04079404700002, 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13090933799999999, 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67345 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244668+0000", 2024-03-24T12:03:41.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244668+0000", 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244669+0000", 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3799999999999996e-07 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244711+0000", 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1737999999999999e-05 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244718+0000", 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9670000000000004e-06 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244726+0000", 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3240000000000003e-06 2024-03-24T12:03:41.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.246631+0000", 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0019046449999999999 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.246739+0000", 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000108255 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:03:41.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.246827+0000", 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7949999999999996e-05 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248266+0000", 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001439552 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.249776+0000", 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0015096510000000001 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.367 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.375474+0000", 2024-03-24T12:03:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.125697846 2024-03-24T12:03:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.375532+0000", 2024-03-24T12:03:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8628999999999997e-05 2024-03-24T12:03:41.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.375577+0000", 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4943e-05 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:41.372 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:41.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67346 7.d 7:b983acce:::benchmark_data_smithi064_87013_object67345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.244737+0000", 2024-03-24T12:03:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.04072469300002, 2024-03-24T12:03:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.23104503000000001, 2024-03-24T12:03:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67346 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244737+0000", 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244737+0000", 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244738+0000", 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9100000000000003e-07 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244768+0000", 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.987e-05 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244775+0000", 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0790000000000003e-06 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244783+0000", 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9240000000000007e-06 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244881+0000", 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8303999999999999e-05 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.244971+0000", 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0192999999999995e-05 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.245064+0000", 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2862999999999995e-05 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.246920+0000", 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018555080000000001 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248033+0000", 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001113492 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475728+0000", 2024-03-24T12:03:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.22769488399999999 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475759+0000", 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0743e-05 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.475782+0000", 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3479e-05 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67349 7.d 7:bb7ecb5e:::benchmark_data_smithi064_87013_object67348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.248333+0000", 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.03712872599999, 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.22997953400000001, 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67349 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248333+0000", 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248333+0000", 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248335+0000", 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.691e-06 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248394+0000", 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8768999999999999e-05 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248404+0000", 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0180000000000001e-05 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248412+0000", 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8839999999999994e-06 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248458+0000", 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6019000000000001e-05 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248536+0000", 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8510999999999994e-05 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.248624+0000", 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7881999999999996e-05 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.250421+0000", 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001797079 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.251047+0000", 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00062609000000000004 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.478197+0000", 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.227149829 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.478258+0000", 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0225e-05 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.478313+0000", 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5374999999999998e-05 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:67354 7.7 7:edc7c1d6:::benchmark_data_smithi064_87013_object67353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:55:14.252428+0000", 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.03303446500001, 2024-03-24T12:03:41.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.21488665200000001, 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 67354 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252428+0000", 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252428+0000", 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252428+0000", 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7199999999999997e-07 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252486+0000", 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7927000000000003e-05 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252495+0000", 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.952e-06 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252500+0000", 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7779999999999998e-06 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252541+0000", 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1078000000000002e-05 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252625+0000", 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4281000000000002e-05 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.252715+0000", 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9786999999999999e-05 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.254391+0000", 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00167537 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.254646+0000", 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025550399999999999 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.467237+0000", 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.21259097299999999 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.390 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.467280+0000", 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3071999999999997e-05 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:55:14.467314+0000", 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4057999999999998e-05 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221637 7.12 7:4f878fc8:::benchmark_data_smithi064_87013_object100680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:52.063663+0000", 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 349.22179863899999, 2024-03-24T12:03:41.391 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.99110485800000003, 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221637 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063663+0000", 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.392 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063663+0000", 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063665+0000", 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8589999999999999e-06 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063670+0000", 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6079999999999998e-06 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063680+0000", 2024-03-24T12:03:41.393 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0261999999999999e-05 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063686+0000", 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0959999999999997e-06 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063756+0000", 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9313999999999994e-05 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063767+0000", 2024-03-24T12:03:41.394 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1250000000000001e-05 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.051956+0000", 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.98818874599999995 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.052070+0000", 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000114833 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.052151+0000", 2024-03-24T12:03:41.395 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0321999999999994e-05 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054645+0000", 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024945760000000001 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054678+0000", 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2839000000000002e-05 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054713+0000", 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4647000000000002e-05 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054742+0000", 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9116999999999999e-05 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054768+0000", 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6389000000000001e-05 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221714 7.12 7:4da8459c:::benchmark_data_smithi064_87013_object100757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:52.125166+0000", 2024-03-24T12:03:41.397 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 349.16029628799998, 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92984263700000003, 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221714 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.125166+0000", 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.125166+0000", 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.125168+0000", 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1459999999999999e-06 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.125173+0000", 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9660000000000004e-06 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.125185+0000", 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1902000000000001e-05 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.125192+0000", 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5170000000000002e-06 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.125250+0000", 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8071e-05 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:41.400 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.125260+0000", 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3309999999999993e-06 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.053208+0000", 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92794865400000004 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.053283+0000", 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4134999999999994e-05 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.053347+0000", 2024-03-24T12:03:41.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4777999999999995e-05 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054845+0000", 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001497382 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054876+0000", 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1155000000000002e-05 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.402 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054939+0000", 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3350999999999997e-05 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054975+0000", 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6037000000000001e-05 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.055008+0000", 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3212e-05 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.403 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221721 7.12 7:4e096267:::benchmark_data_smithi064_87013_object100764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:52.132681+0000", 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 349.15278111499998, 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92248741899999998, 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221721 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.132681+0000", 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.132681+0000", 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.132683+0000", 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8309999999999999e-06 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.132690+0000", 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5560000000000002e-06 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.132701+0000", 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0655e-05 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.132708+0000", 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5849999999999996e-06 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.132756+0000", 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8590000000000001e-05 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.132763+0000", 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3119999999999996e-06 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.053897+0000", 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92113309399999999 2024-03-24T12:03:41.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.053967+0000", 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0508999999999999e-05 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054032+0000", 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4795999999999994e-05 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054906+0000", 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00087418800000000003 2024-03-24T12:03:41.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.054913+0000", 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7000000000000002e-06 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.055118+0000", 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000205689 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.055145+0000", 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6431000000000002e-05 2024-03-24T12:03:41.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.055168+0000", 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3482999999999999e-05 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:35810 8.9 8:9bea66dd:::benchmark_data_smithi064_120708_object35809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:15.849825+0000", 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 265.435637434, 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13141013400000001, 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:41.410 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 35810 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.849825+0000", 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.849825+0000", 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:41.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.849827+0000", 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1900000000000002e-06 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.849925+0000", 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8245999999999994e-05 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.849937+0000", 2024-03-24T12:03:41.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1717e-05 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.849945+0000", 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5669999999999995e-06 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.850053+0000", 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107354 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.850231+0000", 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00017798 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.850336+0000", 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000105419 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851546+0000", 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012102689999999999 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851711+0000", 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016492899999999999 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.981137+0000", 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.129425663 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.981196+0000", 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8533999999999999e-05 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.981235+0000", 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9266e-05 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.415 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.416 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:41.416 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:41.416 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:03:41.416 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:41.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:41.514 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:41.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:41.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:41.655 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:41.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:41.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:41.798 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:41.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:41.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:41.943 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:41.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:41.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:41.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:41 smithi064 ceph-mon[30223]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T12:03:41.956 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:41.956 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:41 smithi064 ceph-mon[42243]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T12:03:42.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 46 15 79071 79056 107.397 79.625 0.0135414 0.00930681 2024-03-24T12:03:42.044 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:42.085 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:42.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:42.187 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:42.221 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:42.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:42.323 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:41 smithi193 ceph-mon[33593]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T12:03:42.360 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:42.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:42.462 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:42.495 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:42.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:42.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:42.607 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:03:42.639 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:42.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:42.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:42.776 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:42.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:42.878 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:42.934 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:42.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:42.941 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:03:43.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 47 16 80341 80325 106.799 79.3125 0.00809397 0.00935866 2024-03-24T12:03:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:43.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:43.073 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:43.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:43.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:43.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:42 smithi193 ceph-mon[33593]: pgmap v515: 168 pgs: 168 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-24T12:03:43.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3878005502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:03:43.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:42 smithi193 ceph-mon[33593]: from='client.35351 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:03:43.222 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:43.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:42 smithi064 ceph-mon[30223]: pgmap v515: 168 pgs: 168 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-24T12:03:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3878005502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:03:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:42 smithi064 ceph-mon[30223]: from='client.35351 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:03:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:42 smithi064 ceph-mon[42243]: pgmap v515: 168 pgs: 168 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-24T12:03:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3878005502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:03:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:42 smithi064 ceph-mon[42243]: from='client.35351 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:03:43.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:43.357 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:43.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:43.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:03:43.458 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:43.493 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:43.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:43.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:43.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:43.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:43.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:43.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:43.769 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:43.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:43.871 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:43.906 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:43.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:44.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:44.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 48 16 81629 81613 106.251 80.5 0.0119785 0.0094074 2024-03-24T12:03:44.046 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:44.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:44.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:44.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:44.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[30223]: from='client.35351 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:03:44.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[30223]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T12:03:44.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[30223]: 1.c scrub starts 2024-03-24T12:03:44.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[30223]: 1.c scrub ok 2024-03-24T12:03:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[42243]: from='client.35351 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:03:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[42243]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T12:03:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[42243]: 1.c scrub starts 2024-03-24T12:03:44.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:43 smithi064 ceph-mon[42243]: 1.c scrub ok 2024-03-24T12:03:44.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:44.178 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:44.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:44.280 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:44.328 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:44.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:43 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:03:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:43 smithi193 ceph-mon[33593]: from='client.35351 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:03:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:43 smithi193 ceph-mon[33593]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T12:03:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:43 smithi193 ceph-mon[33593]: 1.c scrub starts 2024-03-24T12:03:44.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:43 smithi193 ceph-mon[33593]: 1.c scrub ok 2024-03-24T12:03:44.430 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:44.441 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:44.472 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:44.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:44.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:44.616 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:44.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:44.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:44.766 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:44.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:44.868 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:44.902 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:44.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:44.958 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-24T12:03:45.003 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:45.007 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 49 16 82856 82840 105.648 76.6875 0.00916015 0.00945989 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[30223]: pgmap v518: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/77681060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[42243]: pgmap v518: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T12:03:45.035 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T12:03:45.036 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/77681060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:03:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:45.050 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:45.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:45.108 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:03:45.108 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:03:45.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.108 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35306.0:82973 9.9 e273/246 9:90de1ee9:::benchmark_data_smithi064_148274_object82972:head v 273'5068, mlcod=273'5065)", 2024-03-24T12:03:45.108 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:45.104986+0000", 2024-03-24T12:03:45.108 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0023378370000000002, 2024-03-24T12:03:45.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023668920000000002, 2024-03-24T12:03:45.108 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104986+0000", 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104986+0000", 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104986+0000", 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9500000000000002e-07 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105013+0000", 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7229999999999998e-05 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105019+0000", 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8780000000000003e-06 2024-03-24T12:03:45.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105024+0000", 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6709999999999998e-06 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106243+0000", 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00121857 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:45.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106260+0000", 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6898999999999999e-05 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35306.0:82967 9.c e273/246 9:304c257f:::benchmark_data_smithi064_148274_object82966:head v 273'5183, mlcod=273'5179)", 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:45.105070+0000", 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0022537030000000001, 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023546180000000002, 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:45.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105070+0000", 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105070+0000", 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105071+0000", 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9999999999999996e-07 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105101+0000", 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0540000000000002e-05 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105109+0000", 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4760000000000001e-06 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105115+0000", 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7089999999999996e-06 2024-03-24T12:03:45.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105694+0000", 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00057893099999999998 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105712+0000", 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7844000000000001e-05 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35306.0:82977 9.5 e273/246 9:a14f62d6:::benchmark_data_smithi064_148274_object82976:head v 273'5214, mlcod=273'5212)", 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:45.106206+0000", 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0011175639999999999, 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012841160000000001, 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106206+0000", 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106206+0000", 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106207+0000", 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0440000000000001e-06 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106262+0000", 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5340000000000002e-05 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106270+0000", 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7689999999999996e-06 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106277+0000", 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7920000000000004e-06 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106337+0000", 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9923000000000002e-05 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.106354+0000", 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7062999999999999e-05 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35306.0:82966 9.9 e273/246 9:945bf636:::benchmark_data_smithi064_148274_object82965:head v 273'5067, mlcod=273'5065)", 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:45.101139+0000", 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.006184798, 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0064198600000000003, 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:03:45.119 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.101139+0000", 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.101139+0000", 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.101140+0000", 2024-03-24T12:03:45.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2500000000000001e-06 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.101168+0000", 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7758000000000001e-05 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.101180+0000", 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2203e-05 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.101187+0000", 2024-03-24T12:03:45.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0700000000000001e-06 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.101252+0000", 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4701000000000003e-05 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.101272+0000", 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0636e-05 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35306.0:82965 9.c e273/246 9:37d1b1af:::benchmark_data_smithi064_148274_object82964:head v 273'5182, mlcod=273'5179)", 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:45.102345+0000", 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0049783229999999998, 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052760400000000001, 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.102345+0000", 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.102345+0000", 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.102346+0000", 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.17e-06 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.102381+0000", 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4555999999999999e-05 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:45.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.102394+0000", 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3558e-05 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.102401+0000", 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.596e-06 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.103216+0000", 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081493500000000003 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:45.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.103250+0000", 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4087000000000001e-05 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.35306.0:82971 9.1 e273/246 9:8694645b:::benchmark_data_smithi064_148274_object82970:head v 273'5115, mlcod=273'5112)", 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:45.104907+0000", 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.002416811, 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0027719519999999998, 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:03:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104907+0000", 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104907+0000", 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104908+0000", 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.409e-06 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:03:45.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104950+0000", 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1650999999999998e-05 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104958+0000", 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4780000000000008e-06 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.104965+0000", 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7519999999999999e-06 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:03:45.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105100+0000", 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013550199999999999 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:45.105121+0000", 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0630999999999999e-05 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 6 2024-03-24T12:03:45.129 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:03:45.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:45.193 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:45.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:45.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:45.331 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:45.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:44 smithi193 ceph-mon[33593]: pgmap v518: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-24T12:03:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:44 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T12:03:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:44 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T12:03:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:44 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T12:03:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:44 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T12:03:45.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/77681060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:03:45.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:45.472 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:45.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:45.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:45.611 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:45.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:45.713 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:45.753 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:45.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:45.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:45.840 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:03:45.855 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:45.857 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:03:45.946 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:45.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:46.008 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 50 16 84174 84158 105.182 82.375 0.00513262 0.00950286 2024-03-24T12:03:46.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:46.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:45 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T12:03:46.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:45 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T12:03:46.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:45 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/77681060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:03:46.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:45 smithi193 ceph-mon[33593]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T12:03:46.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:45 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T12:03:46.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:45 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T12:03:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:46.112 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:46.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:46.213 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:46.229 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:46.264 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:46.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T12:03:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T12:03:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/77681060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[30223]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/77681060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[42243]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T12:03:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:45 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T12:03:46.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:46.413 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:46.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:46.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:46.553 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:46.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:46.653 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:46.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-24T12:03:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:46.691 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:46.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:46.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:46.845 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:46.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:46.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:46.981 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:46.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:47.008 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 51 16 85232 85216 104.416 66.125 0.0299548 0.00957047 2024-03-24T12:03:47.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:46 smithi064 ceph-mon[30223]: pgmap v520: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:03:47.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:46 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T12:03:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:46 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T12:03:47.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:46 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:03:47.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:46 smithi064 ceph-mon[42243]: pgmap v520: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:03:47.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:46 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T12:03:47.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:46 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T12:03:47.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:46 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:03:47.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:47.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:47.243 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:47.299 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:47.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:46 smithi193 ceph-mon[33593]: pgmap v520: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:03:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:46 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T12:03:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:46 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T12:03:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:46 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:03:47.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:47.403 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:03:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:47.403 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:03:47.450 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:03:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:47.453 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:47.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:47.484 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:47.490 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:03:47.507 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:47.534 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:03:47.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:47.575 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:03:47.614 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:03:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:47.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:47.658 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:03:47.696 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:03:47.700 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:03:47.700 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:03:47.700 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:03:47.700 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:03:47.700 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:03:47.701 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:03:47.701 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:03:47.701 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:03:47.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:47.770 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:47.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:47.772 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:03:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:03:47.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:47.917 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:47.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:48.008 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 52 16 86107 86091 103.46 54.6875 0.0151139 0.00965988 2024-03-24T12:03:48.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:48.025 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:47 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:03:48.025 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:47 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:03:48.025 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:47 smithi193 ceph-mon[33593]: 4.14 scrub starts 2024-03-24T12:03:48.025 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:47 smithi193 ceph-mon[33593]: 4.14 scrub ok 2024-03-24T12:03:48.025 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:47 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T12:03:48.025 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:47 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2384922246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:48.025 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:47 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T12:03:48.057 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:48.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:48.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:48.198 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:48.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[30223]: 4.14 scrub starts 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[30223]: 4.14 scrub ok 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2384922246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:03:48.278 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:03:48.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[42243]: 4.14 scrub starts 2024-03-24T12:03:48.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[42243]: 4.14 scrub ok 2024-03-24T12:03:48.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T12:03:48.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2384922246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:03:48.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:47 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T12:03:48.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:48.334 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:48.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:48.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:48.479 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:48.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:48.557 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:03:48.557 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:03:48.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:48.617 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:48.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:48.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:48.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:48.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:48.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:48.898 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:48.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:48.993 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: pgmap v521: 168 pgs: 168 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T12:03:49.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:49.008 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 53 16 87335 87319 102.956 76.75 0.00561321 0.00970844 2024-03-24T12:03:49.039 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:49.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:49.141 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:49.177 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:49.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:49.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: pgmap v521: 168 pgs: 168 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T12:03:49.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:49.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 1.11 scrub starts 2024-03-24T12:03:49.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 1.11 scrub ok 2024-03-24T12:03:49.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 1.12 scrub starts 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 1.12 scrub ok 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:03:49.262 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:03:49.262 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.262 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T12:03:49.262 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T12:03:49.262 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T12:03:49.262 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:48 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T12:03:49.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:49.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: pgmap v521: 168 pgs: 168 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 1.11 scrub starts 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 1.11 scrub ok 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 1.12 scrub starts 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 1.12 scrub ok 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T12:03:49.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 1.11 scrub starts 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 1.11 scrub ok 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 1.12 scrub starts 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 1.12 scrub ok 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.305 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T12:03:49.306 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:48 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T12:03:49.308 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:03:49.318 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:49.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:49.419 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:49.465 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:49.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:49.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:49.605 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:49.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:49.707 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:49.720 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:03:49.752 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:49.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:49.854 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:49.891 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:49.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:49.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:50.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 54 16 88578 88562 102.487 77.6875 0.0188877 0.00975191 2024-03-24T12:03:50.033 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:50.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:50.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:50.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:50.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:50.197 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:50 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi193 to 3248M 2024-03-24T12:03:50.197 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:50 smithi193 ceph-mon[33593]: Adjusting osd_memory_target on smithi064 to 3248M 2024-03-24T12:03:50.197 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:50 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:03:50.197 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:50 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:03:50.197 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:50 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T12:03:50.197 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:50 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T12:03:50.197 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:50 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:03:50.197 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:50 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:03:50.279 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi193 to 3248M 2024-03-24T12:03:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[30223]: Adjusting osd_memory_target on smithi064 to 3248M 2024-03-24T12:03:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:03:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:03:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T12:03:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T12:03:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:03:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:03:50.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi193 to 3248M 2024-03-24T12:03:50.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[42243]: Adjusting osd_memory_target on smithi064 to 3248M 2024-03-24T12:03:50.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:03:50.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:03:50.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T12:03:50.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T12:03:50.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:03:50.283 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:50 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:03:50.316 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:50.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:50.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:50.456 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:50.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:50.558 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:50.601 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:50.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:50.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:50.737 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:50.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:50.839 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:50.878 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:50.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:50.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:51.008 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 55 16 89887 89871 102.111 81.8125 0.0126976 0.00978782 2024-03-24T12:03:51.010 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:51.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:51.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:51.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:51.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:51.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: pgmap v522: 168 pgs: 168 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T12:03:51.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 4.1b scrub starts 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 4.1b scrub ok 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: pgmap v522: 168 pgs: 168 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T12:03:51.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 4.1b scrub starts 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 4.1b scrub ok 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:03:51.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:51 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:03:51.251 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:51.287 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:51.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:51.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:51.327 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: pgmap v522: 168 pgs: 168 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 4.1b scrub starts 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 4.1b scrub ok 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:03:51.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:51 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:03:51.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:51.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:51.430 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:51.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:51.532 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:51.573 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:51.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:51.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:03:51.675 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:51.712 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:51.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:51.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:51.858 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:51.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:51.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:52.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 56 16 91168 91152 101.717 80.0625 0.0270792 0.00982614 2024-03-24T12:03:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:52.010 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:52.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:52.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:52.131 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:03:52.131 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:03:52.132 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:52 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T12:03:52.132 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:52 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T12:03:52.154 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:52.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:52.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:52 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T12:03:52.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:52 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T12:03:52.217 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:52 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T12:03:52.217 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:52 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T12:03:52.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:52.296 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:52.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:52.397 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:52.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:52.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:52.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:52.575 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:52.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:52.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:52.714 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:52.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:52.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:52.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:52.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:03:52.860 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:52.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:52.961 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:52.995 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:52.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:53.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 57 16 92329 92313 101.205 72.5625 0.017844 0.00987549 2024-03-24T12:03:53.085 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:03:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=cleanup t=2024-03-24T12:03:52.886943554Z level=info msg="Completed cleanup jobs" duration=2.711092ms 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: pgmap v523: 168 pgs: 168 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 607 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 5.1b scrub starts 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 5.1b scrub ok 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T12:03:53.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:53 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T12:03:53.096 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:53.129 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:53.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:53.156 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:03:53.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:53.295 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:53.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:53.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: pgmap v523: 168 pgs: 168 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 607 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-24T12:03:53.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T12:03:53.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T12:03:53.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T12:03:53.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T12:03:53.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 5.1b scrub starts 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 5.1b scrub ok 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: pgmap v523: 168 pgs: 168 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 607 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 5.1b scrub starts 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 5.1b scrub ok 2024-03-24T12:03:53.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T12:03:53.298 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T12:03:53.298 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T12:03:53.298 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:53 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T12:03:53.396 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:53.431 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:53.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:53.533 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:53.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:53.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:53.668 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:53.707 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:53.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:53.808 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:53.848 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:53.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:53.950 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:53.987 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:53.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:54.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 58 16 93430 93414 100.647 68.8125 0.00599591 0.0099311 2024-03-24T12:03:54.061 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-24T12:03:54.065 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:03:54.088 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:54.122 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:54.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:54.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:54.268 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:54.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:54.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: 4.0 scrub starts 2024-03-24T12:03:54.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: 4.0 scrub ok 2024-03-24T12:03:54.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: 6.10 scrub starts 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: 4.d scrub starts 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: 4.d scrub ok 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: 6.10 scrub ok 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4049638451' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/32784032' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[30223]: from='client.35372 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:03:54.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: 4.0 scrub starts 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: 4.0 scrub ok 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: 6.10 scrub starts 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: 4.d scrub starts 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: 4.d scrub ok 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: 6.10 scrub ok 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4049638451' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/32784032' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:03:54.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:54 smithi064 ceph-mon[42243]: from='client.35372 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:03:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: 4.0 scrub starts 2024-03-24T12:03:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: 4.0 scrub ok 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: 6.10 scrub starts 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: 4.d scrub starts 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: 4.d scrub ok 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: 6.10 scrub ok 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4049638451' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/32784032' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:03:54.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:54 smithi193 ceph-mon[33593]: from='client.35372 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:03:54.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:54.403 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:54.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:54.504 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:54.544 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:54.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:54.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:54.684 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:54.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:54.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:54.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:54.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:54.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:54.959 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:54.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:55.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 59 16 94710 94694 100.296 80 0.00758809 0.00996516 2024-03-24T12:03:55.060 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:55.099 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:55.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:55.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:55.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: pgmap v524: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4049638451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:03:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[30223]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: pgmap v524: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4049638451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:03:55.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:55 smithi064 ceph-mon[42243]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T12:03:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: pgmap v524: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-24T12:03:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:03:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:03:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T12:03:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T12:03:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:03:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:03:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4049638451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:03:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:03:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:55 smithi193 ceph-mon[33593]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T12:03:55.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:55.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:55.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:55.514 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:55.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:55.616 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:55.650 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:55.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:55.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:55.775 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:55.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:55.877 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:55.915 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:55.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:56.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:03:56.011137+0000 min lat: 0.00138521 max lat: 0.190405 avg lat: 0.0100199 2024-03-24T12:03:56.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:03:56.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 60 16 95797 95781 99.7571 67.9375 0.00861 0.0100199 2024-03-24T12:03:56.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:56.041 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:56.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:56.142 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:56.178 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:56.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:56.280 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:56.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:56 smithi064 ceph-mon[30223]: 4.1e scrub starts 2024-03-24T12:03:56.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:56 smithi064 ceph-mon[30223]: 4.1e scrub ok 2024-03-24T12:03:56.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:56 smithi064 ceph-mon[42243]: 4.1e scrub starts 2024-03-24T12:03:56.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:56 smithi064 ceph-mon[42243]: 4.1e scrub ok 2024-03-24T12:03:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:56.309 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:56.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:56 smithi193 ceph-mon[33593]: 4.1e scrub starts 2024-03-24T12:03:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:56 smithi193 ceph-mon[33593]: 4.1e scrub ok 2024-03-24T12:03:56.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:56.446 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:56.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:56.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:56.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:56.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:56.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:56.812 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:56.852 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:56.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:56.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:56.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:57.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:57.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 61 16 96898 96882 99.2497 68.8125 0.0139313 0.0100713 2024-03-24T12:03:57.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:57.139 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:57.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:57.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:57.280 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:57.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: pgmap v526: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: 2.1e scrub starts 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: 2.1e scrub ok 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: 6.19 scrub starts 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: 6.19 scrub ok 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: 5.1f deep-scrub starts 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: 5.1f deep-scrub ok 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-mon[33593]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T12:03:57.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: pgmap v526: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: 2.1e scrub starts 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: 2.1e scrub ok 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: 6.19 scrub starts 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: 6.19 scrub ok 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: 5.1f deep-scrub starts 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: 5.1f deep-scrub ok 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:57.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[30223]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: pgmap v526: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: 2.1e scrub starts 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: 2.1e scrub ok 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: 6.19 scrub starts 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: 6.19 scrub ok 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: 5.1f deep-scrub starts 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: 5.1f deep-scrub ok 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:57.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-mon[42243]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T12:03:57.421 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:57.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:03:57.523 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:57.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:57.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:57.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:57.699 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:57.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:57.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:57.803 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:03:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:03:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:57.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:03:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:03:57.842 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:57.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:57.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:57.978 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:57.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:58.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 62 16 98103 98087 98.8635 75.3125 0.0209857 0.0101036 2024-03-24T12:03:58.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:58.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:58.220 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:03:58.254 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:58.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:03:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:58 smithi193 ceph-mon[33593]: 5.c scrub starts 2024-03-24T12:03:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:58 smithi193 ceph-mon[33593]: 5.c scrub ok 2024-03-24T12:03:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:58 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T12:03:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:58 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T12:03:58.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:58.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:58.403 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:58 smithi064 ceph-mon[30223]: 5.c scrub starts 2024-03-24T12:03:58.403 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:58 smithi064 ceph-mon[30223]: 5.c scrub ok 2024-03-24T12:03:58.403 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:58 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T12:03:58.403 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:58 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T12:03:58.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:58 smithi064 ceph-mon[42243]: 5.c scrub starts 2024-03-24T12:03:58.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:58 smithi064 ceph-mon[42243]: 5.c scrub ok 2024-03-24T12:03:58.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:58 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T12:03:58.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:58 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T12:03:58.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:03:58.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:58.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:03:58.645 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:03:58.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:58.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:03:58.789 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:58.819 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:58.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:58.920 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:58.959 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:58.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:59.009 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 63 16 99267 99251 98.4489 72.75 0.0205833 0.0101525 2024-03-24T12:03:59.060 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:03:59.098 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:59.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:03:59.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:03:59.233 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:59.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:03:59.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: pgmap v528: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-03-24T12:03:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T12:03:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T12:03:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: 6.a scrub starts 2024-03-24T12:03:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: 6.a scrub ok 2024-03-24T12:03:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T12:03:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T12:03:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T12:03:59.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:03:59 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T12:03:59.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:59.372 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:59.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:59.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:03:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:59.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: pgmap v528: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: 6.a scrub starts 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: 6.a scrub ok 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T12:03:59.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: pgmap v528: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: 6.a scrub starts 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: 6.a scrub ok 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T12:03:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:03:59 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T12:03:59.611 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:59.649 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:59.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:59.751 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:03:59.784 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:03:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:59.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:03:59.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-24T12:03:59.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:03:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:03:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:03:59.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:00.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 64 16 100585 100569 98.1977 82.375 0.0131836 0.0101782 2024-03-24T12:04:00.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:00.084 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:00.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:00.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:00.229 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:00.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:00 smithi064 ceph-mon[30223]: 5.5 deep-scrub starts 2024-03-24T12:04:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:00 smithi064 ceph-mon[30223]: 5.5 deep-scrub ok 2024-03-24T12:04:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:00 smithi064 ceph-mon[42243]: 5.5 deep-scrub starts 2024-03-24T12:04:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:00 smithi064 ceph-mon[42243]: 5.5 deep-scrub ok 2024-03-24T12:04:00.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:00.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:00.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:00.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:00.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:00 smithi193 ceph-mon[33593]: 5.5 deep-scrub starts 2024-03-24T12:04:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:00 smithi193 ceph-mon[33593]: 5.5 deep-scrub ok 2024-03-24T12:04:00.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:00.683 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:00.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:00.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:00.816 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:04:00.817 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:04:00.818 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:04:00.818 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:04:00.818 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:04:00.818 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:04:00.818 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:04:00.821 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:00.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:00.922 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:00.960 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:00.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:00.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:01.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 65 16 101712 101696 97.7705 70.4375 0.00857895 0.0102232 2024-03-24T12:04:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:01.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:01.119 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:01.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:01.221 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:01.262 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:01.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:01.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:01.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:01.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:01.504 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:01.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[30223]: pgmap v529: 168 pgs: 168 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-03-24T12:04:01.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[30223]: 9.b deep-scrub starts 2024-03-24T12:04:01.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[30223]: 1.9 deep-scrub starts 2024-03-24T12:04:01.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[30223]: 1.9 deep-scrub ok 2024-03-24T12:04:01.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2050217341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:01.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[30223]: 2.a deep-scrub starts 2024-03-24T12:04:01.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[30223]: 2.a deep-scrub ok 2024-03-24T12:04:01.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[42243]: pgmap v529: 168 pgs: 168 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-03-24T12:04:01.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[42243]: 9.b deep-scrub starts 2024-03-24T12:04:01.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[42243]: 1.9 deep-scrub starts 2024-03-24T12:04:01.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[42243]: 1.9 deep-scrub ok 2024-03-24T12:04:01.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2050217341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:01.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[42243]: 2.a deep-scrub starts 2024-03-24T12:04:01.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:01 smithi064 ceph-mon[42243]: 2.a deep-scrub ok 2024-03-24T12:04:01.542 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:01.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:01 smithi193 ceph-mon[33593]: pgmap v529: 168 pgs: 168 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-03-24T12:04:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:01 smithi193 ceph-mon[33593]: 9.b deep-scrub starts 2024-03-24T12:04:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:01 smithi193 ceph-mon[33593]: 1.9 deep-scrub starts 2024-03-24T12:04:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:01 smithi193 ceph-mon[33593]: 1.9 deep-scrub ok 2024-03-24T12:04:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:01 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2050217341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:01 smithi193 ceph-mon[33593]: 2.a deep-scrub starts 2024-03-24T12:04:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:01 smithi193 ceph-mon[33593]: 2.a deep-scrub ok 2024-03-24T12:04:01.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:01.683 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:01.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:01.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:01.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:01.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:01.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:01.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:01.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:01.958 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:02.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 66 16 102848 102832 97.3648 71 0.0142205 0.0102659 2024-03-24T12:04:02.057 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:02.093 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:02.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:02.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:02.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:02.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:02.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:02.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:02.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:02.513 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:02.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:02.615 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:02.651 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:02.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:02.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:04:02.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:02.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:02.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:02.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:03.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 67 16 103975 103959 96.9628 70.4375 0.0098774 0.0103074 2024-03-24T12:04:03.049 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:03.087 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:03.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:03.189 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:03.230 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:03.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:03.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:03.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:03.425 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[30223]: pgmap v530: 168 pgs: 168 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[30223]: 6.d scrub starts 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[30223]: 6.d scrub ok 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[42243]: pgmap v530: 168 pgs: 168 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T12:04:03.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T12:04:03.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[42243]: 6.d scrub starts 2024-03-24T12:04:03.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[42243]: 6.d scrub ok 2024-03-24T12:04:03.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T12:04:03.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T12:04:03.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:03 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T12:04:03.474 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:03.512 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:03.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:03 smithi193 ceph-mon[33593]: pgmap v530: 168 pgs: 168 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-24T12:04:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:03 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T12:04:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:03 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T12:04:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:03 smithi193 ceph-mon[33593]: 6.d scrub starts 2024-03-24T12:04:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:03 smithi193 ceph-mon[33593]: 6.d scrub ok 2024-03-24T12:04:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:03 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T12:04:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:03 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T12:04:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:03 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T12:04:03.614 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:03.652 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:03.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:03.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:03.792 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:03.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:03.893 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:03.933 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:03.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:03.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:04.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:04:04.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:04.082 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 68 16 105140 105124 96.6076 72.8125 0.00381482 0.0103451 2024-03-24T12:04:04.123 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:04.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:04.225 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:04.260 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:04.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:04.325 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:04.362 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:04.399 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:04.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:04.438 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T12:04:04.438 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T12:04:04.438 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T12:04:04.438 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T12:04:04.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T12:04:04.439 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T12:04:04.439 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T12:04:04.439 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T12:04:04.439 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T12:04:04.440 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:04 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T12:04:04.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:04.537 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:04.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:04 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T12:04:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:04 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T12:04:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:04 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T12:04:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:04 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T12:04:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:04 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T12:04:04.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:04.685 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:04.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:04.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:04.828 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:04.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:04.929 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:04.970 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:04.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:05.011 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 69 15 106274 106259 96.2355 70.9375 0.0197924 0.010386 2024-03-24T12:04:05.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:05.115 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:05.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:05.217 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:05.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:05 smithi064 ceph-mon[42243]: pgmap v531: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-24T12:04:05.246 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:05 smithi064 ceph-mon[30223]: pgmap v531: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-24T12:04:05.254 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:05.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:05.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:05.395 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:05.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:05.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:05.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:05 smithi193 ceph-mon[33593]: pgmap v531: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-24T12:04:05.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:05.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:05.761 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:05.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:05.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:05.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:05.978 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:05.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:06.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 70 16 107377 107361 95.8446 68.875 0.0202331 0.0104285 2024-03-24T12:04:06.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:06.117 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:06.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:06.149 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-24T12:04:06.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:06.253 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:06.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:06.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:06.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:06 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T12:04:06.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:06 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T12:04:06.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:06 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/566235992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:04:06.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:06 smithi064 ceph-mon[30223]: from='client.35381 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:04:06.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:06 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T12:04:06.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:06 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T12:04:06.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:06 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/566235992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:04:06.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:06 smithi064 ceph-mon[42243]: from='client.35381 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:04:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:06.401 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:06.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:06.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:06.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:06 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T12:04:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:06 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T12:04:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:06 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/566235992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:04:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:06 smithi193 ceph-mon[33593]: from='client.35381 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:04:06.638 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:06.677 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:06.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:06.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:06.807 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:06.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:06.909 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:06.947 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:06.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:07.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 71 16 108510 108494 95.492 70.8125 0.0127189 0.010467 2024-03-24T12:04:07.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:07.085 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:07.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:07.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:07.225 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:07.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:07.327 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:07.362 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:07.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:07.399 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:07 smithi193 ceph-mon[33593]: pgmap v532: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:04:07.400 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:07 smithi193 ceph-mon[33593]: from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:04:07.400 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:07 smithi193 ceph-mon[33593]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T12:04:07.400 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:07 smithi193 ceph-mon[33593]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T12:04:07.465 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:07.504 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:07.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:07.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-mon[30223]: pgmap v532: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:04:07.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-mon[30223]: from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:04:07.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-mon[30223]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T12:04:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-mon[30223]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T12:04:07.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:04:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-mon[42243]: pgmap v532: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:04:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-mon[42243]: from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:04:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-mon[42243]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T12:04:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:07 smithi064 ceph-mon[42243]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T12:04:07.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:07.642 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:07.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:07.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:07.778 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:07.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:07.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:04:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:04:07.880 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:07.920 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:07.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:08.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 72 16 109626 109610 95.1344 69.75 0.00815924 0.0105056 2024-03-24T12:04:08.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:08.061 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:08.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:08.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-24T12:04:08.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:08.203 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:08.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:08.222 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:04:08.222 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:04:08.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35306.0:107765 9.b e276/246)", 2024-03-24T12:04:08.222 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.374667+0000", 2024-03-24T12:04:08.222 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.019615253999999999, 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019635487, 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.374667+0000", 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.374667+0000", 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.374668+0000", 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3489999999999999e-06 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.374675+0000", 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.207e-06 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.374689+0000", 2024-03-24T12:04:08.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.374697+0000", 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8120000000000008e-06 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107789 9.5 9:acde188d:::benchmark_data_smithi064_148274_object107788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.383960+0000", 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.010322168, 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010413164000000001, 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107789 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.383960+0000", 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.383960+0000", 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.383963+0000", 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4549999999999998e-06 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.383988+0000", 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5400000000000001e-05 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384005+0000", 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7252999999999999e-05 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384014+0000", 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6799999999999999e-06 2024-03-24T12:04:08.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384075+0000", 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1088999999999997e-05 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384223+0000", 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014795800000000001 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384374+0000", 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015077299999999999 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390963+0000", 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065894009999999999 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391006+0000", 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3337000000000002e-05 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107790 9.9 9:920b5f42:::benchmark_data_smithi064_148274_object107789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.384815+0000", 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0094669739999999995, 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096210189999999998, 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107790 2024-03-24T12:04:08.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384815+0000", 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384815+0000", 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384817+0000", 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.593e-06 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384840+0000", 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3138000000000001e-05 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384858+0000", 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8241999999999999e-05 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384887+0000", 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8728e-05 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.385095+0000", 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020806700000000001 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.385239+0000", 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014381499999999999 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.385465+0000", 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022566700000000001 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394219+0000", 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087541660000000007 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394269+0000", 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9759000000000003e-05 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394349+0000", 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0222000000000005e-05 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394398+0000", 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9326999999999997e-05 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107793 9.5 9:a4c33b25:::benchmark_data_smithi064_148274_object107792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.236 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.384909+0000", 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0093729160000000002, 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095938199999999994, 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107793 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384909+0000", 2024-03-24T12:04:08.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384909+0000", 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384910+0000", 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9100000000000003e-07 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384929+0000", 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9253999999999999e-05 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384937+0000", 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6559999999999992e-06 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384943+0000", 2024-03-24T12:04:08.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9000000000000003e-06 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.387526+0000", 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025826849999999999 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.387661+0000", 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013486100000000001 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:04:08.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.387831+0000", 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170874 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107795 9.5 9:a9ead316:::benchmark_data_smithi064_148274_object107794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.384951+0000", 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0093312499999999993, 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00959694, 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.241 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107795 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384951+0000", 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384951+0000", 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384952+0000", 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2500000000000005e-07 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384969+0000", 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7657999999999999e-05 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384975+0000", 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0859999999999998e-06 2024-03-24T12:04:08.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384981+0000", 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.31e-06 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391037+0000", 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006056311 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391156+0000", 2024-03-24T12:04:08.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391307+0000", 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150984 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107796 9.9 9:98d6a30f:::benchmark_data_smithi064_148274_object107795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.384987+0000", 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0092953319999999999, 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096047770000000001, 2024-03-24T12:04:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107796 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384987+0000", 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384987+0000", 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.384988+0000", 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8699999999999995e-07 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.385007+0000", 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9321e-05 2024-03-24T12:04:08.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.385013+0000", 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7259999999999999e-06 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.385018+0000", 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0819999999999998e-06 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390124+0000", 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051065609999999999 2024-03-24T12:04:08.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390494+0000", 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036926199999999998 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390653+0000", 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000159889 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:08.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394304+0000", 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036509580000000002 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394338+0000", 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3376e-05 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107797 9.b 9.c006250b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.388293+0000", 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0059892390000000004, 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006352066, 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107797 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388293+0000", 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388293+0000", 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388295+0000", 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7999999999999999e-06 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388319+0000", 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3802999999999999e-05 2024-03-24T12:04:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388343+0000", 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3852000000000001e-05 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388353+0000", 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0868e-05 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "rep_scrubmap(9.b e246 from shard 5)", 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.388743+0000", 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0055397989999999998, 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005959776, 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388743+0000", 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388743+0000", 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388744+0000", 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3790000000000001e-06 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388758+0000", 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3781000000000001e-05 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388768+0000", 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0173999999999999e-05 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.388775+0000", 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8900000000000001e-06 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35306.0:107790 9.9 e276/246 ondisk, result = 0)", 2024-03-24T12:04:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.390507+0000", 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.003775615, 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042230009999999997, 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390507+0000", 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390507+0000", 2024-03-24T12:04:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390508+0000", 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1820000000000001e-06 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390513+0000", 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6829999999999999e-06 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390522+0000", 2024-03-24T12:04:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.611e-06 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390529+0000", 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8970000000000002e-06 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394258+0000", 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037287760000000001 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394264+0000", 2024-03-24T12:04:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8900000000000004e-06 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107798 9.5 9.4fa89e05 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.390545+0000", 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00373771, 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042199819999999997, 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107798 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390545+0000", 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390545+0000", 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390546+0000", 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.395e-06 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390568+0000", 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2354999999999999e-05 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390577+0000", 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5739999999999996e-06 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390584+0000", 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7440000000000001e-06 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35306.0:107789 9.5 e276/246)", 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.390959+0000", 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00332292, 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003836285, 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.263 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390959+0000", 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390959+0000", 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390960+0000", 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1100000000000004e-07 2024-03-24T12:04:08.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390964+0000", 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0980000000000004e-06 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390973+0000", 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9619999999999999e-06 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.390980+0000", 2024-03-24T12:04:08.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4080000000000003e-06 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35306.0:107793 9.5 e276/246)", 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.391016+0000", 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0032659910000000002, 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038059489999999999, 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391016+0000", 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391016+0000", 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391017+0000", 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.37e-07 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391020+0000", 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2030000000000001e-06 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391030+0000", 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7079999999999992e-06 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391036+0000", 2024-03-24T12:04:08.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8649999999999998e-06 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35306.0:107793 9.5 e276/246)", 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.391181+0000", 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0031008849999999998, 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036676339999999999, 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391181+0000", 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391181+0000", 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391182+0000", 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8899999999999998e-07 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391194+0000", 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2004000000000001e-05 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391204+0000", 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1439999999999992e-06 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391210+0000", 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6069999999999996e-06 2024-03-24T12:04:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "rep_scrubmap(9.b e246 from shard 6)", 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.391973+0000", 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0023096449999999999, 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029255570000000001, 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391973+0000", 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391973+0000", 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391974+0000", 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.125e-06 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391988+0000", 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3920000000000001e-05 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.391997+0000", 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4890000000000001e-06 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.392003+0000", 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.156e-06 2024-03-24T12:04:08.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35306.0:107769 9.b e276/246)", 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.392014+0000", 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0022681089999999999, 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029315399999999998, 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.392014+0000", 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.392014+0000", 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.392015+0000", 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0999999999999999e-07 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.392018+0000", 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.88e-06 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.392022+0000", 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8280000000000002e-06 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.392027+0000", 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4660000000000001e-06 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107801 9.5 9.391b6795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.393638+0000", 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00064401900000000002, 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001363207, 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107801 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.393638+0000", 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.393638+0000", 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.393640+0000", 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1119999999999999e-06 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.393690+0000", 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842000000000002e-05 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.393702+0000", 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936000000000001e-05 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.393711+0000", 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5590000000000006e-06 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107765 9.b 9:d17ab0f1:::benchmark_data_smithi064_148274_object107764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.360426+0000", 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.033856356999999997, 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034645426999999999, 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:04:08.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107765 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360426+0000", 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360426+0000", 2024-03-24T12:04:08.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360428+0000", 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6869999999999999e-06 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360448+0000", 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0322e-05 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360456+0000", 2024-03-24T12:04:08.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2279999999999998e-06 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360463+0000", 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360634+0000", 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170846 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:08.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360778+0000", 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000144265 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.360970+0000", 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019258900000000001 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.369755+0000", 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087848200000000005 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:08.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.369773+0000", 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7946000000000001e-05 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35306.0:107795 9.5 e276/246)", 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.394228+0000", 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.4756999999999998e-05, 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093667499999999999, 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394228+0000", 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394228+0000", 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394228+0000", 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2600000000000001e-07 2024-03-24T12:04:08.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394233+0000", 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3359999999999997e-06 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394241+0000", 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0029999999999994e-06 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394247+0000", 2024-03-24T12:04:08.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8799999999999996e-06 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:107769 9.b 9:d98f08a3:::benchmark_data_smithi064_148274_object107768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.363085+0000", 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.031196867, 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032139182000000002, 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:04:08.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107769 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.363085+0000", 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.363085+0000", 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.363088+0000", 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1019999999999999e-06 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.363113+0000", 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4995999999999999e-05 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.363125+0000", 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.258e-05 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.363135+0000", 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5880000000000004e-06 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.365792+0000", 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026568989999999999 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.365954+0000", 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016276499999999999 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.366130+0000", 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017516400000000001 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.369840+0000", 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037105839999999998 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.369883+0000", 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2429000000000003e-05 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35306.0:107795 9.5 e276/246)", 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:04:06.394319+0000", 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4294967295.9999628, 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099142400000000008, 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:08.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394319+0000", 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394319+0000", 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394320+0000", 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.052e-06 2024-03-24T12:04:08.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394325+0000", 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5279999999999997e-06 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394334+0000", 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6070000000000008e-06 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:04:06.394340+0000", 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8900000000000001e-06 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:08.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:08.297 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:04:08.297 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 20 2024-03-24T12:04:08.297 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:04:08.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:08.346 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:08.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:08.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:08.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:08.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:08.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:08 smithi064 ceph-mon[30223]: 4.4 scrub starts 2024-03-24T12:04:08.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:08 smithi064 ceph-mon[30223]: 4.4 scrub ok 2024-03-24T12:04:08.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:08 smithi064 ceph-mon[30223]: 9.0 scrub starts 2024-03-24T12:04:08.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:08 smithi064 ceph-mon[42243]: 4.4 scrub starts 2024-03-24T12:04:08.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:08 smithi064 ceph-mon[42243]: 4.4 scrub ok 2024-03-24T12:04:08.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:08 smithi064 ceph-mon[42243]: 9.0 scrub starts 2024-03-24T12:04:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:08 smithi193 ceph-mon[33593]: 4.4 scrub starts 2024-03-24T12:04:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:08 smithi193 ceph-mon[33593]: 4.4 scrub ok 2024-03-24T12:04:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:08 smithi193 ceph-mon[33593]: 9.0 scrub starts 2024-03-24T12:04:08.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:08.639 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:08.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:08.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:08.776 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:08.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:08.878 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:08.915 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:08.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:09.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 73 16 110653 110637 94.7104 64.1875 0.0119659 0.0105525 2024-03-24T12:04:09.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:04:09.016 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:09.054 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:09.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:09.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:09.205 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:09.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:09.231 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[30223]: pgmap v535: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[30223]: 9.3 scrub starts 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[30223]: 3.2 scrub starts 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[30223]: 3.2 scrub ok 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[42243]: pgmap v535: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[42243]: 9.3 scrub starts 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[42243]: 3.2 scrub starts 2024-03-24T12:04:09.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:09 smithi064 ceph-mon[42243]: 3.2 scrub ok 2024-03-24T12:04:09.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:09.341 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:09.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:09.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:09.476 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:09.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:09.578 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:09 smithi193 ceph-mon[33593]: pgmap v535: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:04:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:09 smithi193 ceph-mon[33593]: 9.3 scrub starts 2024-03-24T12:04:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:09 smithi193 ceph-mon[33593]: 3.2 scrub starts 2024-03-24T12:04:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:09 smithi193 ceph-mon[33593]: 3.2 scrub ok 2024-03-24T12:04:09.613 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:09.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:09.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:09.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:09.753 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:09.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:09.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:09.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:09.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:09.998 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:10.010 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 74 16 111461 111445 94.1129 50.5 0.0207833 0.010618 2024-03-24T12:04:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:10.044 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:10.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:10.146 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:10.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:10.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:10.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:10 smithi064 ceph-mon[30223]: 9.9 deep-scrub starts 2024-03-24T12:04:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:10 smithi064 ceph-mon[30223]: 9.e scrub starts 2024-03-24T12:04:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:10 smithi064 ceph-mon[30223]: 9.0 scrub ok 2024-03-24T12:04:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:10 smithi064 ceph-mon[30223]: pgmap v536: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T12:04:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:10 smithi064 ceph-mon[42243]: 9.9 deep-scrub starts 2024-03-24T12:04:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:10 smithi064 ceph-mon[42243]: 9.e scrub starts 2024-03-24T12:04:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:10 smithi064 ceph-mon[42243]: 9.0 scrub ok 2024-03-24T12:04:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:10 smithi064 ceph-mon[42243]: pgmap v536: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T12:04:10.320 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:10.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:10.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:10.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:10.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:10.557 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:10 smithi193 ceph-mon[33593]: 9.9 deep-scrub starts 2024-03-24T12:04:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:10 smithi193 ceph-mon[33593]: 9.e scrub starts 2024-03-24T12:04:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:10 smithi193 ceph-mon[33593]: 9.0 scrub ok 2024-03-24T12:04:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:10 smithi193 ceph-mon[33593]: pgmap v536: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T12:04:10.593 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:10.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:10.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:10.708 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:10.752 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:10.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:10.820 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:04:10.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:10.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:04:10.896 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:10.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:10.897 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:04:10.928 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:04:10.965 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:04:10.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:11.006 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:04:11.011 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 75 16 112278 112262 93.5385 51.0625 0.0178872 0.0106831 2024-03-24T12:04:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:11.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:11.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:04:11.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:04:11.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:11.177 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:11.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:11.279 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:11.313 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:11.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:11.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:04:11.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:04:11.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[30223]: 9.3 scrub ok 2024-03-24T12:04:11.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[30223]: 9.d scrub starts 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1066794816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[42243]: 9.3 scrub ok 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[42243]: 9.d scrub starts 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T12:04:11.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:11 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1066794816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:11.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:11.450 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:11.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:11.552 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:11 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:04:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:11 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:04:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:11 smithi193 ceph-mon[33593]: 9.3 scrub ok 2024-03-24T12:04:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:11 smithi193 ceph-mon[33593]: 9.d scrub starts 2024-03-24T12:04:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:11 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T12:04:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:11 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T12:04:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1066794816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:11.589 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:11.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:11.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:11.731 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:11.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:11.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:11.869 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:11.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:11.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:12.007 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:12.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:12.011 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 76 16 113151 113135 93.0257 54.5625 0.0320499 0.0107432 2024-03-24T12:04:12.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:12.141 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:12.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:12.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:12.278 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:12.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:12.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:12.413 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:12.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[30223]: 9.e scrub ok 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[30223]: 5.16 scrub starts 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[30223]: 5.16 scrub ok 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[30223]: pgmap v537: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 166 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 787 op/s 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[42243]: 9.e scrub ok 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[42243]: 5.16 scrub starts 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[42243]: 5.16 scrub ok 2024-03-24T12:04:12.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[42243]: pgmap v537: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 166 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 787 op/s 2024-03-24T12:04:12.492 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:12.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:12.554 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:12.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:12 smithi193 ceph-mon[33593]: 9.e scrub ok 2024-03-24T12:04:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:12 smithi193 ceph-mon[33593]: 5.16 scrub starts 2024-03-24T12:04:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:12 smithi193 ceph-mon[33593]: 5.16 scrub ok 2024-03-24T12:04:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:12 smithi193 ceph-mon[33593]: pgmap v537: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 166 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 787 op/s 2024-03-24T12:04:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:12.655 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:12.694 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:12.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:12.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:12.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:12.841 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:12.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:12.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:12.976 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:12.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:13.011 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 77 16 114010 113994 92.5147 53.6875 0.0101008 0.0108012 2024-03-24T12:04:13.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:13.115 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:13.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:13.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:13.272 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:13.276 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:13.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:13.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:13.420 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:13.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:13.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: 2.18 scrub starts 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: 2.18 scrub ok 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: 9.d scrub ok 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T12:04:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[30223]: 9.5 scrub starts 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: 2.18 scrub starts 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: 2.18 scrub ok 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: 9.d scrub ok 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T12:04:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:13 smithi064 ceph-mon[42243]: 9.5 scrub starts 2024-03-24T12:04:13.557 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:13.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:04:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:04:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: 2.18 scrub starts 2024-03-24T12:04:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: 2.18 scrub ok 2024-03-24T12:04:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T12:04:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:04:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:04:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: 9.d scrub ok 2024-03-24T12:04:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T12:04:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T12:04:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:13 smithi193 ceph-mon[33593]: 9.5 scrub starts 2024-03-24T12:04:13.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:13.691 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:13.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:13.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:13.834 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:13.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:13.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:13.975 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:13.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:14.011 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 78 15 114891 114876 92.0352 55.125 0.0887505 0.0108573 2024-03-24T12:04:14.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:14.110 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:14.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:14.184 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:04:14.212 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:14.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:04:14.248 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:14.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:14.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T12:04:14.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T12:04:14.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3812667815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:04:14.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[30223]: from='client.25405 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:04:14.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[30223]: pgmap v539: 168 pgs: 3 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 163 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3812667815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[42243]: from='client.25405 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[42243]: pgmap v539: 168 pgs: 3 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 163 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T12:04:14.279 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:14 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T12:04:14.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:14.392 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:14.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:14.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:14.532 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:14.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:14 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T12:04:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:14 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T12:04:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3812667815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:04:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:14 smithi193 ceph-mon[33593]: from='client.25405 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:04:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:14 smithi193 ceph-mon[33593]: pgmap v539: 168 pgs: 3 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 163 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-24T12:04:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:14 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T12:04:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:14 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T12:04:14.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:14.676 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:14.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:14.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:14.796 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:04:14.796 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:04:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:14.825 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:14.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:14.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:14.962 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:14.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:15.011 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 79 16 115838 115822 91.6186 59.125 0.00835666 0.0109096 2024-03-24T12:04:15.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:15.098 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:15.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:15.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:15.235 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:15.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:15.336 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:15.371 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:15.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:15.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:15.508 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:15.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[30223]: 6.1d deep-scrub starts 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[30223]: from='client.25405 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[30223]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[30223]: 6.1d deep-scrub ok 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[42243]: 6.1d deep-scrub starts 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[42243]: from='client.25405 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:04:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[42243]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T12:04:15.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[42243]: 6.1d deep-scrub ok 2024-03-24T12:04:15.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T12:04:15.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:04:15.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:15 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:04:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:15 smithi193 ceph-mon[33593]: 6.1d deep-scrub starts 2024-03-24T12:04:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:15 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:15 smithi193 ceph-mon[33593]: from='client.25405 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:04:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:15 smithi193 ceph-mon[33593]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T12:04:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:15 smithi193 ceph-mon[33593]: 6.1d deep-scrub ok 2024-03-24T12:04:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:15 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T12:04:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:15 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:04:15.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:15 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:04:15.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:15.644 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:15.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:15.746 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:15.784 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:15.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:15.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:15.918 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:15.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:16.012 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:04:16.013335+0000 min lat: 0.00138521 max lat: 0.190405 avg lat: 0.0109561 2024-03-24T12:04:16.012 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:04:16.012 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 80 16 116810 116794 91.2327 60.75 0.00962366 0.0109561 2024-03-24T12:04:16.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:16.054 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:16.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:16.156 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:16.191 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:16.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:16.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:16.327 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:16.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:16.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:16.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:16.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:16.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:16.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:16 smithi064 ceph-mon[30223]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T12:04:16.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:16 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:04:16.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:16 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:04:16.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:16 smithi064 ceph-mon[30223]: pgmap v542: 168 pgs: 3 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 163 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 786 op/s 2024-03-24T12:04:16.538 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:16 smithi064 ceph-mon[42243]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T12:04:16.538 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:16 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:04:16.538 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:16 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:04:16.538 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:16 smithi064 ceph-mon[42243]: pgmap v542: 168 pgs: 3 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 163 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 786 op/s 2024-03-24T12:04:16.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:16 smithi193 ceph-mon[33593]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T12:04:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:16 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:04:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:16 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:04:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:16 smithi193 ceph-mon[33593]: pgmap v542: 168 pgs: 3 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 163 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 786 op/s 2024-03-24T12:04:16.626 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:16.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:16.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:16.763 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:16.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:16.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:16.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:04:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:16.911 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:16.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:16.940 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:17.012 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 81 16 117787 117771 90.8601 61.0625 0.00990871 0.010999 2024-03-24T12:04:17.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:17.061 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:17.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:17.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:17.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:04:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:17.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:17.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:17.387 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:17.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:17.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:17 smithi064 ceph-mon[30223]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T12:04:17.433 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:04:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:04:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:17.433 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:17 smithi064 ceph-mon[42243]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T12:04:17.490 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:17.519 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:17.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:17 smithi193 ceph-mon[33593]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T12:04:17.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:04:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:04:17.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:17.659 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:17.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:17.759 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:17.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:17.799 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:17.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:17.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:17.943 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:17.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:18.012 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 82 16 118577 118561 90.3542 49.375 0.0047352 0.0110606 2024-03-24T12:04:18.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:18.087 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:18.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:18.188 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:18.220 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:18.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:18.321 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:18.354 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:18.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:18.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:18.492 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:18.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:18 smithi064 ceph-mon[30223]: pgmap v544: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 986 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-24T12:04:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:18 smithi064 ceph-mon[42243]: pgmap v544: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 986 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-24T12:04:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:18.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:18 smithi193 ceph-mon[33593]: pgmap v544: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 986 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-24T12:04:18.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:18.594 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:18.628 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:18.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:18.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:18.774 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:18.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:18.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:18.908 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:18.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:19.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:19.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 83 16 119471 119455 89.9387 55.875 0.00907676 0.0111129 2024-03-24T12:04:19.053 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:19.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:19.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:19.198 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:19.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:19.251 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-03-24T12:04:19.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:19.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:19 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1759099692' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:04:19.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:19 smithi064 ceph-mon[42243]: from='client.25408 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:04:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:19 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1759099692' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:04:19.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:19 smithi064 ceph-mon[30223]: from='client.25408 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:04:19.340 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:19.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:19.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:19.475 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:19.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:19.576 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:19 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1759099692' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:04:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:19 smithi193 ceph-mon[33593]: from='client.25408 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:04:19.617 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:19.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:19.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:19.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:19.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:19.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:19.898 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:19.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:20.000 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:20.012 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 84 16 120360 120344 89.5294 55.5625 0.00527366 0.0111628 2024-03-24T12:04:20.040 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:20.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:20.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:20.184 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:20.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:20.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:20.329 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:20.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:20.430 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:20.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:20.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:20.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:20 smithi064 ceph-mon[42243]: from='client.25408 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:04:20.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:20 smithi064 ceph-mon[42243]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T12:04:20.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:20 smithi064 ceph-mon[42243]: pgmap v546: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 986 B/s rd, 56 MiB/s wr, 897 op/s 2024-03-24T12:04:20.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2671683240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:20.505 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:20 smithi064 ceph-mon[30223]: from='client.25408 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:04:20.505 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:20 smithi064 ceph-mon[30223]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T12:04:20.505 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:20 smithi064 ceph-mon[30223]: pgmap v546: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 986 B/s rd, 56 MiB/s wr, 897 op/s 2024-03-24T12:04:20.505 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:20 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2671683240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:20.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-24T12:04:20.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:20 smithi193 ceph-mon[33593]: from='client.25408 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:04:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:20 smithi193 ceph-mon[33593]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T12:04:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:20 smithi193 ceph-mon[33593]: pgmap v546: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 986 B/s rd, 56 MiB/s wr, 897 op/s 2024-03-24T12:04:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:20 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2671683240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:20.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:20.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:20.775 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:20.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:20.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:04:20.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:04:20.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:04:20.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:04:20.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:04:20.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:04:20.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:04:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:04:20.876 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:20.912 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:20.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:21.012 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 85 16 121332 121316 89.1908 60.75 0.0685368 0.0112042 2024-03-24T12:04:21.013 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:21.046 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:21.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:21.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:21.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:21.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:21.321 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:21.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:21.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:21.461 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:21.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:21.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:21.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:21.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:21.713 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:21.714 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:21.739 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:21.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:21.794 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:21.840 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:21.876 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:21.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:21.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:21.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:22.012 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:22.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:22.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 86 16 122269 122253 88.8334 58.5625 0.00449618 0.0112505 2024-03-24T12:04:22.115 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:22.154 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:22.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:22.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:22.293 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:22.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:22.394 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:22.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:22.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:22.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:22.571 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:22.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:22.673 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:22.706 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:22.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:22.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:22.808 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:22.846 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:22.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:22.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:22.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:23.013 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 87 16 123254 123238 88.5199 61.5625 0.00486061 0.0112907 2024-03-24T12:04:23.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:23.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:22 smithi193 ceph-mon[33593]: pgmap v547: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 829 B/s rd, 53 MiB/s wr, 843 op/s 2024-03-24T12:04:23.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:22 smithi193 ceph-mon[33593]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T12:04:23.122 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:23.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:23.224 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:23.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:23.261 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:23.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:22 smithi064 ceph-mon[30223]: pgmap v547: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 829 B/s rd, 53 MiB/s wr, 843 op/s 2024-03-24T12:04:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:22 smithi064 ceph-mon[30223]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T12:04:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:22 smithi064 ceph-mon[42243]: pgmap v547: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 829 B/s rd, 53 MiB/s wr, 843 op/s 2024-03-24T12:04:23.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:22 smithi064 ceph-mon[42243]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T12:04:23.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:23.396 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:23.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:23.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:23.539 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:23.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:23.641 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:23.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:04:23.682 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:23.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:23.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:23.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:23.822 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:23.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:23.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:23.968 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:23.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:24.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 88 16 124139 124123 88.1425 55.3125 0.00919292 0.0113394 2024-03-24T12:04:24.069 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:24.105 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:24.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:24.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:24.252 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:24.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:24.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:24.390 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:24.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:24.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:24.525 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:24.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:24.627 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:24.663 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:24.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:24.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:24.798 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:24.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:24.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:24.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:24.939 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:24.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:25.013 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 89 16 125009 124993 87.763 54.375 0.00509029 0.011388 2024-03-24T12:04:25.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:25.071 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:25.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:25.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:25.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:24 smithi193 ceph-mon[33593]: pgmap v549: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T12:04:25.173 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:25.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:25.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:25.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:24 smithi064 ceph-mon[30223]: pgmap v549: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T12:04:25.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:24 smithi064 ceph-mon[42243]: pgmap v549: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T12:04:25.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:25.335 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:25.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:25.437 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:25.472 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:25.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:25.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:25.600 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:25.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:25.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:25.741 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:25.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:25.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:25.865 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:25.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:25.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:25.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:26.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:26.013 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 90 2 125810 125808 87.3538 50.9375 0.0561467 0.0114425 2024-03-24T12:04:26.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:26.127 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:26.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:26.229 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:26.256 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:26.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:26.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:25 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:04:26.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:25 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:04:26.267 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:25 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:04:26.267 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:25 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:04:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:25 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:04:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:25 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:04:26.358 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:26.383 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:26.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:26.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:26.511 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:26.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:26.613 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:26.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:26.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:26.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:26.764 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:26.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:26.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:26.891 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:26.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:26.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total time run: 90.0345 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total writes made: 125810 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Write size: 65536 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object size: 65536 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Bandwidth (MB/sec): 87.3346 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Bandwidth: 34.0909 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max bandwidth (MB/sec): 214 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min bandwidth (MB/sec): 49.375 2024-03-24T12:04:27.014 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average IOPS: 1397 2024-03-24T12:04:27.015 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev IOPS: 545.454 2024-03-24T12:04:27.015 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max IOPS: 3424 2024-03-24T12:04:27.015 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min IOPS: 790 2024-03-24T12:04:27.015 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average Latency(s): 0.011443 2024-03-24T12:04:27.015 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Latency(s): 0.0117794 2024-03-24T12:04:27.015 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max latency(s): 0.204802 2024-03-24T12:04:27.015 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min latency(s): 0.00138521 2024-03-24T12:04:27.019 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:27.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:27.020 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:04:27.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:27.154 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:27.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:27.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:27.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[30223]: pgmap v550: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 827 op/s 2024-03-24T12:04:27.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[30223]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T12:04:27.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:04:27.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:27.279 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[30223]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T12:04:27.280 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[42243]: pgmap v550: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 827 op/s 2024-03-24T12:04:27.280 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[42243]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T12:04:27.280 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:04:27.280 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:27.280 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:26 smithi064 ceph-mon[42243]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T12:04:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:27.302 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:27.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:27.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:26 smithi193 ceph-mon[33593]: pgmap v550: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 827 op/s 2024-03-24T12:04:27.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:26 smithi193 ceph-mon[33593]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T12:04:27.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:04:27.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:27.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:26 smithi193 ceph-mon[33593]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T12:04:27.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:27.440 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:27.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:27.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:27.543 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:04:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:04:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:27.582 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:27.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:27.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:27.731 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:27.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:27.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:27.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:04:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:04:27.870 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:27.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:27.876 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:04:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215973 7.16 7:6b3af921:::benchmark_data_smithi064_87013_object95016:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.912919+0000", 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 402.959674576, 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1901695670000001, 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215973 2024-03-24T12:04:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912919+0000", 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912919+0000", 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912921+0000", 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0609999999999998e-06 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912926+0000", 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6949999999999997e-06 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912939+0000", 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2233e-05 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912950+0000", 2024-03-24T12:04:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1226000000000001e-05 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913010+0000", 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0679e-05 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913018+0000", 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4749999999999996e-06 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.005474+0000", 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092455561000000006 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.005481+0000", 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9230000000000002e-06 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.103005+0000", 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.097523698 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.103088+0000", 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3016000000000006e-05 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215986 7.17 7:eb4ea6f0:::benchmark_data_smithi064_87013_object95029:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.912967+0000", 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 402.95962576900001, 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.216837122, 2024-03-24T12:04:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215986 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912967+0000", 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912967+0000", 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912968+0000", 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5099999999999999e-07 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912972+0000", 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4989999999999999e-06 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912978+0000", 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6069999999999996e-06 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912984+0000", 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1190000000000002e-06 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913051+0000", 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6143999999999996e-05 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913058+0000", 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5039999999999996e-06 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006365+0000", 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093306807000000005 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006374+0000", 2024-03-24T12:04:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2490000000000007e-06 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.129717+0000", 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233432430000001 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.129805+0000", 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199000000000003e-05 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215990 7.19 7:9bd879f2:::benchmark_data_smithi064_87013_object95033:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.912998+0000", 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 402.95959539199998, 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.224418563, 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215990 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912998+0000", 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912998+0000", 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912998+0000", 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.99e-07 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913001+0000", 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8600000000000001e-06 2024-03-24T12:04:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913007+0000", 2024-03-24T12:04:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9730000000000002e-06 2024-03-24T12:04:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913014+0000", 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7410000000000003e-06 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913068+0000", 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4457999999999999e-05 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913075+0000", 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5060000000000001e-06 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.005232+0000", 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092157417000000005 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.005242+0000", 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9110000000000005e-06 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.115803+0000", 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1105610480000001 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.115949+0000", 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014587500000000001 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.116297+0000", 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034831400000000001 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.127609+0000", 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011311524 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.127663+0000", 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3974000000000002e-05 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.127712+0000", 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9061999999999999e-05 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.127759+0000", 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6496999999999999e-05 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.127805+0000", 2024-03-24T12:04:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6844000000000001e-05 2024-03-24T12:04:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.127901+0000", 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5229000000000006e-05 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.127965+0000", 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4034000000000004e-05 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.137416+0000", 2024-03-24T12:04:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094516970000000002 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215991 7.14 7:2c9458b2:::benchmark_data_smithi064_87013_object95034:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.913027+0000", 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 402.95956661899999, 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.232752786, 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215991 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913027+0000", 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913027+0000", 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913027+0000", 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4199999999999995e-07 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913030+0000", 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913035+0000", 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5450000000000003e-06 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913041+0000", 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6049999999999997e-06 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913094+0000", 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3032000000000003e-05 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913101+0000", 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7279999999999998e-06 2024-03-24T12:04:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.011526+0000", 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098424864000000001 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.011536+0000", 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0372e-05 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.131473+0000", 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.119936488 2024-03-24T12:04:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.131592+0000", 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119271 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.131837+0000", 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000245603 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.138572+0000", 2024-03-24T12:04:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067350079999999998 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.138618+0000", 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5374999999999998e-05 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.140247+0000", 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016291680000000001 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.141009+0000", 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076165 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.141481+0000", 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000472862 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.145575+0000", 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040936640000000003 2024-03-24T12:04:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.145637+0000", 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2096999999999999e-05 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.145779+0000", 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142108 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:216008 7.14 7:2b7aeccd:::benchmark_data_smithi064_87013_object95051:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.913053+0000", 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 402.95954061200001, 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2329523339999999, 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216008 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913053+0000", 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913053+0000", 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913053+0000", 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4899999999999995e-07 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913056+0000", 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.745e-06 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913063+0000", 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9569999999999996e-06 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913067+0000", 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5399999999999997e-06 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913142+0000", 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4140999999999994e-05 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913148+0000", 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2870000000000001e-06 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.011700+0000", 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098552612999999997 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.011709+0000", 2024-03-24T12:04:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4290000000000001e-06 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.133536+0000", 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1218273919999999 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.133651+0000", 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114898 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.133869+0000", 2024-03-24T12:04:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217419 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.140367+0000", 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064981079999999998 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.140598+0000", 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000231439 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.140970+0000", 2024-03-24T12:04:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037213399999999998 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.141074+0000", 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104196 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.141490+0000", 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041525299999999998 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.145820+0000", 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043303370000000001 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.145876+0000", 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5753e-05 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.146005+0000", 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012904400000000001 2024-03-24T12:04:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:216010 7.14 7:2ce6e05a:::benchmark_data_smithi064_87013_object95053:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.913086+0000", 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 402.95950756500002, 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2486825459999999, 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216010 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913086+0000", 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913086+0000", 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913086+0000", 2024-03-24T12:04:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5400000000000001e-07 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913089+0000", 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.706e-06 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913095+0000", 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-06 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913100+0000", 2024-03-24T12:04:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7960000000000003e-06 2024-03-24T12:04:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913190+0000", 2024-03-24T12:04:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9864000000000006e-05 2024-03-24T12:04:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-03-24T12:04:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913193+0000", 2024-03-24T12:04:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1039999999999998e-06 2024-03-24T12:04:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.012048+0000", 2024-03-24T12:04:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098855267999999996 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.012057+0000", 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5820000000000002e-06 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.136195+0000", 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1241384059999999 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.136327+0000", 2024-03-24T12:04:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013161299999999999 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.136567+0000", 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024027200000000001 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.141043+0000", 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044759780000000002 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.141124+0000", 2024-03-24T12:04:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1159e-05 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.141453+0000", 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032867200000000001 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.141498+0000", 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4666000000000002e-05 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.146046+0000", 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045481979999999998 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.161552+0000", 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015505896 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.161623+0000", 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1357000000000001e-05 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.161768+0000", 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145045 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:216011 7.19 7:9e3c7e6e:::benchmark_data_smithi064_87013_object95054:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.913106+0000", 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 402.95948719199998, 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2253982050000001, 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216011 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913106+0000", 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913106+0000", 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913107+0000", 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8599999999999998e-07 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913109+0000", 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3829999999999999e-06 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913113+0000", 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9319999999999998e-06 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913117+0000", 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.031e-06 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913159+0000", 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1764999999999997e-05 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.913163+0000", 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5850000000000001e-06 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.005833+0000", 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092669345 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.005841+0000", 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7280000000000001e-06 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.124353+0000", 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1185116289999999 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.124798+0000", 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044507900000000003 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.125143+0000", 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034522699999999998 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.137473+0000", 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012329764999999999 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.137523+0000", 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9579000000000001e-05 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.137569+0000", 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6298000000000003e-05 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.138097+0000", 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052786599999999999 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.138113+0000", 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6093e-05 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.138151+0000", 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8621999999999997e-05 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.138208+0000", 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6467e-05 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.138504+0000", 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000296225 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221641 7.13 7:cf56b4e9:::benchmark_data_smithi064_87013_object100684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.068966+0000", 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.80362708000001, 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99779801899999998, 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221641 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.068966+0000", 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.068966+0000", 2024-03-24T12:04:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.068968+0000", 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9589999999999998e-06 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.068973+0000", 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0699999999999997e-06 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.068982+0000", 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3780000000000007e-06 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.068990+0000", 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9160000000000001e-06 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.069145+0000", 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000155057 2024-03-24T12:04:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.069155+0000", 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0029e-05 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.062508+0000", 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99335277399999999 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.062635+0000", 2024-03-24T12:04:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012677599999999999 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.062847+0000", 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021199899999999999 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066460+0000", 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036134510000000002 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066499+0000", 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8114000000000001e-05 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066661+0000", 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016192800000000001 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066720+0000", 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9948999999999998e-05 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066764+0000", 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3618999999999997e-05 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221658 7.13 7:cde30260:::benchmark_data_smithi064_87013_object100701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.082961+0000", 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.78963182000001, 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98398448599999999, 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221658 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.082961+0000", 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.082961+0000", 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.082962+0000", 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5099999999999999e-07 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.082965+0000", 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.881e-06 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.082972+0000", 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5479999999999998e-06 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.082978+0000", 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.015e-06 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.083033+0000", 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5859000000000002e-05 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.083043+0000", 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5589999999999994e-06 2024-03-24T12:04:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.064323+0000", 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.981279868 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.064435+0000", 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011253500000000001 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.064622+0000", 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186966 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066619+0000", 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001996888 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066814+0000", 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019440500000000001 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066856+0000", 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2645999999999998e-05 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066908+0000", 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2055000000000002e-05 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066946+0000", 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7509999999999998e-05 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221672 7.13 7:cc99a477:::benchmark_data_smithi064_87013_object100715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.092571+0000", 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.78002181099998, 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97683013100000005, 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221672 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.092571+0000", 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.092571+0000", 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.092572+0000", 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7400000000000002e-07 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.092576+0000", 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3799999999999998e-06 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.092582+0000", 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8430000000000004e-06 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.092588+0000", 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999998e-06 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.092785+0000", 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019667 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.092794+0000", 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4840000000000004e-06 2024-03-24T12:04:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.065400+0000", 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97260630999999997 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.065499+0000", 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9217999999999998e-05 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.065655+0000", 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015546200000000001 2024-03-24T12:04:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066775+0000", 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011200260000000001 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.069214+0000", 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024389720000000002 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.069255+0000", 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.104e-05 2024-03-24T12:04:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.069352+0000", 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7612e-05 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.069401+0000", 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9200000000000003e-05 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221679 7.13 7:cb37663d:::benchmark_data_smithi064_87013_object100722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.097732+0000", 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.77486165699997, 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97733617399999995, 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221679 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.097732+0000", 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.097732+0000", 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.097734+0000", 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1459999999999999e-06 2024-03-24T12:04:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.097739+0000", 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3140000000000003e-06 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.097749+0000", 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0207000000000001e-05 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.097757+0000", 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9710000000000004e-06 2024-03-24T12:04:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.097810+0000", 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3066999999999999e-05 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.097818+0000", 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9449999999999993e-06 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.066976+0000", 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96915831100000005 2024-03-24T12:04:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.067194+0000", 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217223 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.067346+0000", 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015213999999999999 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070421+0000", 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030746810000000001 2024-03-24T12:04:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.074847+0000", 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044267910000000002 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.074914+0000", 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6836999999999997e-05 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.075015+0000", 2024-03-24T12:04:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101328 2024-03-24T12:04:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.075068+0000", 2024-03-24T12:04:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2213000000000003e-05 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221680 7.13 7:cce43669:::benchmark_data_smithi064_87013_object100723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.098269+0000", 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.77432377999997, 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98336049000000003, 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221680 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.098269+0000", 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.098269+0000", 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.098271+0000", 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.364e-06 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.098276+0000", 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1800000000000004e-06 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.098286+0000", 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0185e-05 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.098294+0000", 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2560000000000002e-06 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.098355+0000", 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0967000000000001e-05 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.098363+0000", 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5009999999999998e-06 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.068182+0000", 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.969818921 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.068278+0000", 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6451000000000002e-05 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.068432+0000", 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015332400000000001 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.071541+0000", 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031097009999999999 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081288+0000", 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097469700000000006 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081521+0000", 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000232628 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081578+0000", 2024-03-24T12:04:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7235999999999999e-05 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081630+0000", 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1805999999999998e-05 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221723 7.13 7:c97dee8e:::benchmark_data_smithi064_87013_object100766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.133575+0000", 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.73901775000002, 2024-03-24T12:04:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.94817903000000003, 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221723 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.133575+0000", 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.133575+0000", 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.133576+0000", 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7300000000000005e-07 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.133579+0000", 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0599999999999999e-06 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.133586+0000", 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-06 2024-03-24T12:04:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.133592+0000", 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2389999999999999e-06 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.133649+0000", 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6675999999999997e-05 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.133657+0000", 2024-03-24T12:04:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6740000000000003e-06 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.069432+0000", 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.93577406799999996 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.069522+0000", 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0982000000000004e-05 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.069663+0000", 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140681 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.072568+0000", 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029045899999999999 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081337+0000", 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087695489999999997 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081671+0000", 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033349699999999998 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081718+0000", 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7323000000000001e-05 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081754+0000", 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6318e-05 2024-03-24T12:04:27.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221737 7.13 7:cef7a9be:::benchmark_data_smithi064_87013_object100780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.150766+0000", 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.72182715100001, 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.93111997199999996, 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221737 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.150766+0000", 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.150766+0000", 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.150768+0000", 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5e-06 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.150771+0000", 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.150779+0000", 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8590000000000005e-06 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.150786+0000", 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5379999999999999e-06 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.151612+0000", 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00082592099999999997 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.151620+0000", 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7569999999999993e-06 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070449+0000", 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.91882823300000005 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070553+0000", 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010463000000000001 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070731+0000", 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017797200000000001 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.074861+0000", 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00412927 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081374+0000", 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065138440000000004 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081798+0000", 2024-03-24T12:04:27.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042348100000000002 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081839+0000", 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1273999999999998e-05 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081886+0000", 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6845000000000003e-05 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221806 7.13 7:ca51905f:::benchmark_data_smithi064_87013_object100849:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.217048+0000", 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.65554545800001, 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86992986500000002, 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221806 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.217048+0000", 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.217048+0000", 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.217050+0000", 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1509999999999998e-06 2024-03-24T12:04:27.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.217055+0000", 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2240000000000003e-06 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.217066+0000", 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1253999999999999e-05 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.217075+0000", 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449999999999996e-06 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.217808+0000", 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073309999999999998 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.217817+0000", 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6600000000000001e-06 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.071570+0000", 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.85375367099999999 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.071671+0000", 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101216 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.071820+0000", 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014881000000000001 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.074871+0000", 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030510020000000001 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081409+0000", 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006537785 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.086826+0000", 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054170479999999998 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.086928+0000", 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102174 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.086978+0000", 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9325e-05 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221816 7.13 7:cf5f22ac:::benchmark_data_smithi064_87013_object100859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:52.226479+0000", 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 395.64611379199999, 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86067291099999998, 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221816 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.226479+0000", 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.226479+0000", 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.226481+0000", 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5850000000000001e-06 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.226485+0000", 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4859999999999999e-06 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.226496+0000", 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.006e-05 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.226503+0000", 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3980000000000002e-06 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.226567+0000", 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3971000000000001e-05 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:52.226576+0000", 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8270000000000004e-06 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.072598+0000", 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.84602214499999995 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.072699+0000", 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010121899999999999 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.072851+0000", 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000151519 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.074876+0000", 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020253760000000002 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.081444+0000", 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065680490000000003 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.087028+0000", 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055835429999999998 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.087112+0000", 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4676e-05 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.087152+0000", 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0057e-05 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104586 8.12 8:4d569037:::benchmark_data_smithi064_120708_object104585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.756621+0000", 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.115972575, 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085979316, 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T12:04:27.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104586 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756621+0000", 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756621+0000", 2024-03-24T12:04:27.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756623+0000", 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9750000000000001e-06 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756647+0000", 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4124000000000001e-05 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:27.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756657+0000", 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0020999999999999e-05 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756665+0000", 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4679999999999992e-06 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756720+0000", 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5173000000000002e-05 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:27.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756729+0000", 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7770000000000008e-06 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.842494+0000", 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0857644289999999 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.842600+0000", 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010634899999999999 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:27.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:27.998 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104623 8.11 8:8a3bc892:::benchmark_data_smithi064_120708_object104622:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:04:27.998 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.756688+0000", 2024-03-24T12:04:27.998 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.11590509300001, 2024-03-24T12:04:27.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1197171189999999, 2024-03-24T12:04:27.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:27.998 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:27.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104623 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756688+0000", 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756688+0000", 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756689+0000", 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7000000000000003e-07 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756712+0000", 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2946e-05 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:28.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756719+0000", 2024-03-24T12:04:28.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3520000000000001e-06 2024-03-24T12:04:28.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:28.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756726+0000", 2024-03-24T12:04:28.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3570000000000003e-06 2024-03-24T12:04:28.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:28.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.758953+0000", 2024-03-24T12:04:28.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022276819999999999 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.758962+0000", 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3899999999999993e-06 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.866510+0000", 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.107548763 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.866666+0000", 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000155828 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,5,6,7", 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.867174+0000", 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050804699999999997 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.874682+0000", 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075073939999999997 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.874732+0000", 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0102000000000002e-05 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.875980+0000", 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001248514 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.876045+0000", 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5109000000000003e-05 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.876109+0000", 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3553999999999995e-05 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.876196+0000", 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7076000000000005e-05 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:28.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.876254+0000", 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8328000000000001e-05 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.876405+0000", 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015080700000000001 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104643 8.11 8:8fef732e:::benchmark_data_smithi064_120708_object104642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:04:28.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.756736+0000", 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.115857304, 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1208679690000001, 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104643 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756736+0000", 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756736+0000", 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756737+0000", 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4900000000000005e-07 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756758+0000", 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1699e-05 2024-03-24T12:04:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756765+0000", 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8669999999999996e-06 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756772+0000", 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4459999999999998e-06 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759006+0000", 2024-03-24T12:04:28.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002234195 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759014+0000", 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0630000000000006e-06 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.870123+0000", 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1111095660000001 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.870381+0000", 2024-03-24T12:04:28.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025747399999999999 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,5,6,7", 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.870713+0000", 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033240599999999999 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.874689+0000", 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039756469999999997 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.876155+0000", 2024-03-24T12:04:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001465605 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.877054+0000", 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089986299999999999 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.877102+0000", 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7117000000000003e-05 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.877189+0000", 2024-03-24T12:04:28.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7171999999999998e-05 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.877293+0000", 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104595 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.877455+0000", 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016123600000000001 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.877604+0000", 2024-03-24T12:04:28.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149269 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104644 8.12 8:4deb5854:::benchmark_data_smithi064_120708_object104643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.756780+0000", 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.115813264, 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.105261952, 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:04:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104644 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756780+0000", 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756780+0000", 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:04:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756781+0000", 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4499999999999997e-07 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756802+0000", 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1301000000000002e-05 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756809+0000", 2024-03-24T12:04:28.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7780000000000002e-06 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.756815+0000", 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2210000000000002e-06 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.758792+0000", 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019774079999999999 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.758802+0000", 2024-03-24T12:04:28.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4690000000000003e-06 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.842907+0000", 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0841050400000001 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.843058+0000", 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015127900000000001 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,4,5,6,7", 2024-03-24T12:04:28.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.843399+0000", 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000341455 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.847091+0000", 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036919100000000001 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.847136+0000", 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4956000000000001e-05 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.847537+0000", 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000400694 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.849151+0000", 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016143279999999999 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.850591+0000", 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001439921 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:04:28.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.861799+0000", 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011207869000000001 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.861886+0000", 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6963000000000001e-05 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.862042+0000", 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015571500000000001 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:28.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:28.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:04:28.026 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:04:28.026 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:04:28.031 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:28.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:28.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:28.167 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:28.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:28.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:28.302 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:28.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:28.404 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:28.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:28.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:28.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:28.574 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:28.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:28.676 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:28.715 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:28.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:28.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:28.853 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:28.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:28.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:28.987 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:28.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:29.089 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:29.127 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:29.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:29.217 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:28 smithi064 ceph-mon[30223]: pgmap v553: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-24T12:04:29.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:29.218 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:28 smithi064 ceph-mon[42243]: pgmap v553: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-24T12:04:29.218 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:29.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:29.262 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:29.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:29.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:28 smithi193 ceph-mon[33593]: pgmap v553: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-24T12:04:29.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:29.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:29.397 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:29.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:29.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:29.543 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:29.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:29.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:29.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:29.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:29.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:29.823 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:29.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:29.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:29.969 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:29.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:30.070 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:30.109 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:30.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:30.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:30.243 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:30.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:30.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:30.381 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:30.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:30.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:30.521 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:30.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:30.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:30.647 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:30.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:30.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:30.783 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:30.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:30.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:30.920 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:30.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:30.923 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:04:31.022 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:31.054 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:31.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:31.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:31.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:31.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:31.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:31.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:30 smithi064 ceph-mon[30223]: pgmap v554: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 789 op/s 2024-03-24T12:04:31.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:30 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2601674165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:04:31.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:30 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:04:31.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:30 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:04:31.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:30 smithi064 ceph-mon[42243]: pgmap v554: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 789 op/s 2024-03-24T12:04:31.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:30 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2601674165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:04:31.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:30 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:04:31.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:30 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:04:31.325 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:31.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:30 smithi193 ceph-mon[33593]: pgmap v554: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 789 op/s 2024-03-24T12:04:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:30 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2601674165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:04:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:30 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:04:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:30 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:04:31.427 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:31.464 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:31.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:31.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:31.589 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:31.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:31.690 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:31.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:31.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:31.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:31.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:31.966 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:32.005 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:32.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:32.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:32.148 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:32.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:32.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:32.290 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:32.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:32.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:32.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2601674165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:04:32.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[30223]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T12:04:32.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:04:32.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:04:32.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[30223]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T12:04:32.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:32.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2601674165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:04:32.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[42243]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T12:04:32.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:04:32.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:04:32.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:31 smithi064 ceph-mon[42243]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T12:04:32.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:31 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:32.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2601674165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:04:32.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:31 smithi193 ceph-mon[33593]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T12:04:32.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:31 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:04:32.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:31 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:04:32.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:31 smithi193 ceph-mon[33593]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T12:04:32.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:32.426 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:32.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:32.528 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:32.562 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:32.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:32.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:32.705 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:32.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:32.808 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:32.841 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:32.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:32.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:32.982 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:32.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:33.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:33.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:33.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:33.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:33.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:32 smithi064 ceph-mon[30223]: pgmap v556: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 738 op/s 2024-03-24T12:04:33.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:32 smithi064 ceph-mon[42243]: pgmap v556: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 738 op/s 2024-03-24T12:04:33.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:32 smithi193 ceph-mon[33593]: pgmap v556: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 738 op/s 2024-03-24T12:04:33.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:33.404 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:33.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:33.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:33.538 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:33.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:33.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:33.683 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:33.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:33.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:33.826 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:33.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:33.927 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:33.965 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:33.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:34.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:34.103 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:34.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:34.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:34.240 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:34.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:34.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:34.380 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:34.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:34.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:34.512 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:34.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:34.613 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:34.649 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:34.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:34.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:34.788 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:34.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:34.890 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:34.923 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:34.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:35.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:35.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:34 smithi064 ceph-mon[30223]: pgmap v558: 168 pgs: 168 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 12 MiB/s wr, 955 op/s 2024-03-24T12:04:35.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:34 smithi064 ceph-mon[42243]: pgmap v558: 168 pgs: 168 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 12 MiB/s wr, 955 op/s 2024-03-24T12:04:35.063 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:35.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:35.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:04:35.165 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:35.199 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:35.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:35.266 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:34 smithi193 ceph-mon[33593]: pgmap v558: 168 pgs: 168 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 12 MiB/s wr, 955 op/s 2024-03-24T12:04:35.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:35.333 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:35.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:35.367 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:35.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:35.493 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:35.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:35.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:35.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:35.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:35.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:35.797 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:35.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:35.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:35.939 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:35.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:35.945 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.5 to 1 (10000) 2024-03-24T12:04:36.041 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:36.046 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4285514287' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:04:36.047 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:35 smithi064 ceph-mon[30223]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:04:36.047 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4285514287' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:04:36.047 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:35 smithi064 ceph-mon[42243]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:04:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:36.080 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:36.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:36.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:36.218 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:36.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:36.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:36.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4285514287' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:04:36.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:35 smithi193 ceph-mon[33593]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:04:36.361 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:36.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:36.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:36.504 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:36.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:36.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:36.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:36.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:36.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:04:36.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:36.789 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:36.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:36.891 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:36.940 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:36.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:37.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:37.104 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:37.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:37.205 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:37.239 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:37.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:37.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:36 smithi064 ceph-mon[30223]: pgmap v559: 168 pgs: 168 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 10 MiB/s wr, 779 op/s 2024-03-24T12:04:37.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:36 smithi064 ceph-mon[30223]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:04:37.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:36 smithi064 ceph-mon[30223]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T12:04:37.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:36 smithi064 ceph-mon[30223]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T12:04:37.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:36 smithi064 ceph-mon[42243]: pgmap v559: 168 pgs: 168 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 10 MiB/s wr, 779 op/s 2024-03-24T12:04:37.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:36 smithi064 ceph-mon[42243]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:04:37.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:36 smithi064 ceph-mon[42243]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T12:04:37.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:36 smithi064 ceph-mon[42243]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T12:04:37.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:36 smithi193 ceph-mon[33593]: pgmap v559: 168 pgs: 168 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 10 MiB/s wr, 779 op/s 2024-03-24T12:04:37.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:36 smithi193 ceph-mon[33593]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:04:37.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:36 smithi193 ceph-mon[33593]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T12:04:37.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:36 smithi193 ceph-mon[33593]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T12:04:37.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:37.375 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:37.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:37.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:37.517 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:37.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:37.550 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:04:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:04:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:37.573 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:37.618 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:37.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-24T12:04:37.652 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:37.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:37.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:37.789 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:37.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:37.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:04:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:04:37.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:37.927 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:37.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:38.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:38.075 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:38.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:38.177 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:38.211 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:38.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:38.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:38.354 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:38.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:38.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:38.492 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:38.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:38.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:38.613 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:04:38.638 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:38.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:38.656 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:04:38.695 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:04:38.734 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:04:38.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:38.775 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:38.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:38.776 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:04:38.813 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:04:38.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:04:38.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:38.897 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:04:38.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:04:38.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:04:38.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:04:38.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:04:38.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:04:38.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:04:38.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:04:38.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:04:38.913 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:38.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:39.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:39.050 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:39.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:39.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:39.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:39.189 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:39.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:39.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:38 smithi064 ceph-mon[30223]: pgmap v562: 168 pgs: 168 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1.34k op/s 2024-03-24T12:04:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1101995206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:38 smithi064 ceph-mon[42243]: pgmap v562: 168 pgs: 168 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1.34k op/s 2024-03-24T12:04:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1101995206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:39.331 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:39.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:38 smithi193 ceph-mon[33593]: pgmap v562: 168 pgs: 168 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1.34k op/s 2024-03-24T12:04:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:39.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1101995206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:39.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:39.470 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:39.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:39.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:39.607 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:39.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:39.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:39.742 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:39.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:39.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:39.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:39.979 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:40.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:40.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:40.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:39 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:04:40.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:39 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:04:40.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:39 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:04:40.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:39 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:04:40.119 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:40.154 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:40.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:40.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:40.296 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:40.308 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:40.316 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:40.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:40.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:39 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:04:40.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:39 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:04:40.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:40.455 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:40.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:40.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:40.593 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:40.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:40.695 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:40.732 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:40.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:40.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:40.868 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:40.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:40.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:41.008 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:41.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:41.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:40 smithi064 ceph-mon[30223]: pgmap v563: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T12:04:41.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:40 smithi064 ceph-mon[30223]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T12:04:41.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:40 smithi064 ceph-mon[42243]: pgmap v563: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T12:04:41.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:40 smithi064 ceph-mon[42243]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T12:04:41.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:41.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:41.144 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:41.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:41.239 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:04:41.239 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:04:41.245 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:41.275 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:41.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:41.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:40 smithi193 ceph-mon[33593]: pgmap v563: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T12:04:41.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:40 smithi193 ceph-mon[33593]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T12:04:41.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:41.416 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:41.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:41.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:41.557 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:41.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:41.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:41.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:41.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:04:41.800 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:41.831 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:41.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:41.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:41.977 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:41.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:42.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:42.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:42.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:42.285 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:42.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:42.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:41 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:42.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:41 smithi064 ceph-mon[42243]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T12:04:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:41 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:42.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:41 smithi064 ceph-mon[30223]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T12:04:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:41 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:41 smithi193 ceph-mon[33593]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T12:04:42.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:42.423 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:42.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:42.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:42.564 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:42.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:42.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:42.705 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:42.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:42.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:42.846 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:42.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:42.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:42.981 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:42.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:42.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:43.082 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:43.117 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:43.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:43.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:43.258 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:43.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:42 smithi064 ceph-mon[30223]: pgmap v565: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:04:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:42 smithi064 ceph-mon[42243]: pgmap v565: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:04:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:42 smithi193 ceph-mon[33593]: pgmap v565: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:04:43.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:43.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:43.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:43.509 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:43.546 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:43.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:43.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:43.692 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:43.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:43.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:43.829 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:43.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:43.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:04:43.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:43.962 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:43.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:44.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:44.116 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:44.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:44.218 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:44.264 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:44.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:44.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:44.406 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:44.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:44.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:44.542 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:44.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:44.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:44.680 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:44.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:44.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:44.824 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:44.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:44.924 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:44.959 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:44.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:44.993 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:44 smithi193 ceph-mon[33593]: pgmap v567: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-03-24T12:04:45.061 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:45.102 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:45.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:45.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:44 smithi064 ceph-mon[30223]: pgmap v567: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-03-24T12:04:45.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:44 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:04:45.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:44 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:04:45.112 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:44 smithi064 ceph-mon[42243]: pgmap v567: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-03-24T12:04:45.112 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:44 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:04:45.112 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:44 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:04:45.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:45.251 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:45.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:44 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:04:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:44 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:04:45.353 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:45.386 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:45.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:45.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:45.538 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:45.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:45.585 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:45.588 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:45.640 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:45.670 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:45.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:45.772 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:45.803 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:45.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:45.905 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:45.936 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:45.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:46.039 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:46.041 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-03-24T12:04:46.070 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:46.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:46.171 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:46.205 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:46.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:46 smithi064 ceph-mon[30223]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T12:04:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:46 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3295524532' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:04:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:46 smithi064 ceph-mon[42243]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T12:04:46.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:46 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3295524532' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:04:46.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:46.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:46 smithi193 ceph-mon[33593]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T12:04:46.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:46 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3295524532' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:04:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:46.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:46.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:46.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:46.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:46.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:46.637 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:46.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:46.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:46.776 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:46.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:46.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:46.918 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:46.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:47.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:47.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-mon[30223]: pgmap v569: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:04:47.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3295524532' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:04:47.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-mon[30223]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T12:04:47.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1471835525' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:04:47.049 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-mon[42243]: pgmap v569: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:04:47.049 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3295524532' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:04:47.049 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-mon[42243]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T12:04:47.049 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1471835525' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:04:47.049 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:04:47.052 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:47.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:47.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:47.261 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:47.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:47 smithi193 ceph-mon[33593]: pgmap v569: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:04:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:47 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3295524532' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:04:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:47 smithi193 ceph-mon[33593]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T12:04:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:47 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1471835525' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:04:47.363 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:47.399 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:47.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:47.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:47.541 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:47.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:47.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:47.676 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:47.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:47.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:47.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:04:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:04:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:47.821 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:47.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:47.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:04:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:04:47.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:47.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:48.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-24T12:04:48.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:48.102 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:48.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:48.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:48.234 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:48.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:48.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:48.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1471835525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:04:48.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[30223]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T12:04:48.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[30223]: 1.1 scrub starts 2024-03-24T12:04:48.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[30223]: 1.1 scrub ok 2024-03-24T12:04:48.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:48.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1471835525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:04:48.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[42243]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T12:04:48.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[42243]: 1.1 scrub starts 2024-03-24T12:04:48.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:48 smithi064 ceph-mon[42243]: 1.1 scrub ok 2024-03-24T12:04:48.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:48 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:04:48.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:48 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1471835525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:04:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:48 smithi193 ceph-mon[33593]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T12:04:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:48 smithi193 ceph-mon[33593]: 1.1 scrub starts 2024-03-24T12:04:48.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:48 smithi193 ceph-mon[33593]: 1.1 scrub ok 2024-03-24T12:04:48.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:48.375 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:48.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:48.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:48.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:48.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:48.669 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:48.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:48.770 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:48.794 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:48.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:48.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:48.934 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:48.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:49.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:49.073 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:49.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:49.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:49.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:49.217 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:49.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:49.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: pgmap v572: 168 pgs: 168 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T12:04:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T12:04:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T12:04:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:04:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T12:04:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T12:04:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T12:04:49.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: pgmap v572: 168 pgs: 168 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T12:04:49.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:49 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T12:04:49.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: pgmap v572: 168 pgs: 168 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T12:04:49.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:49 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T12:04:49.346 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:49.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:49.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:49.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:49.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:49.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:49.634 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:49.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:49.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:49.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:49.872 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:49.906 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:49.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:50.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:50.045 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:50.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:50.147 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:50.178 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:50.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:50.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:50.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:50 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T12:04:50.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:50 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T12:04:50.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:50 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T12:04:50.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:50 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T12:04:50.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:50.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:50.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:50 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T12:04:50.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:50 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T12:04:50.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:50.459 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:50.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:50.561 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:50.594 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:50.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:50.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:50.805 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:50.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:50.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:50.945 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:50.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:51.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:51.081 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:51.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:51.183 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:51.217 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:51.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:51.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:51 smithi193 ceph-mon[33593]: pgmap v573: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:04:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:51 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:04:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:51 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:04:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:51 smithi193 ceph-mon[33593]: 4.19 scrub starts 2024-03-24T12:04:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:51 smithi193 ceph-mon[33593]: 4.19 scrub ok 2024-03-24T12:04:51.350 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:51.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:51.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:51.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:51.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[30223]: pgmap v573: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:04:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:04:51.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:04:51.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[30223]: 4.19 scrub starts 2024-03-24T12:04:51.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[30223]: 4.19 scrub ok 2024-03-24T12:04:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[42243]: pgmap v573: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:04:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:04:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:04:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[42243]: 4.19 scrub starts 2024-03-24T12:04:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:51 smithi064 ceph-mon[42243]: 4.19 scrub ok 2024-03-24T12:04:51.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:51.620 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:51.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:51.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:51.758 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:51.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:51.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:51.895 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:51.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:51.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:52.036 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:52.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:52.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:52.204 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:52.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:52.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:52.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:52 smithi193 ceph-mon[33593]: 1.14 scrub starts 2024-03-24T12:04:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:52 smithi193 ceph-mon[33593]: 1.14 scrub ok 2024-03-24T12:04:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:52 smithi193 ceph-mon[33593]: 9.b scrub starts 2024-03-24T12:04:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:52 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:04:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:52 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:04:52.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:52 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:04:52.341 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:52.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:52.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[30223]: 1.14 scrub starts 2024-03-24T12:04:52.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[30223]: 1.14 scrub ok 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[30223]: 9.b scrub starts 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[42243]: 1.14 scrub starts 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[42243]: 1.14 scrub ok 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[42243]: 9.b scrub starts 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:04:52.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:52 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:04:52.443 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:52.474 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:52.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:52.575 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:52.610 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:52.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:52.711 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:52.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:52.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:52.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:52.878 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:52.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:52.979 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:53.021 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:53.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:53.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:53.159 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:53.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:53.261 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:53.296 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:53.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[42243]: pgmap v574: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:04:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:04:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:04:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T12:04:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T12:04:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[30223]: pgmap v574: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:04:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:04:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:04:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T12:04:53.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:53 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T12:04:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:53 smithi193 ceph-mon[33593]: pgmap v574: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:04:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:53 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:04:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:53 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:04:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:53 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T12:04:53.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:53 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T12:04:53.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:53.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:53.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:53.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:04:53.571 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:53.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:04:53.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:53.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:53.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:04:53.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:04:53.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:04:53.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:04:53.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:04:53.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:04:53.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:04:53.814 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:53.826 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:04:53.831 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:04:53.831 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:04:53.831 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:04:53.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:04:53.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:04:53.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:04:53.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:04:53.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:04:53.848 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:53.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:53.950 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:53.987 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:53.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:53.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:54.089 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:54.124 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:54.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:54.166 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:04:54.166 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:04:54.166 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:54 smithi064 ceph-mon[30223]: 9.b scrub ok 2024-03-24T12:04:54.167 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1841888841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:54.167 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:54 smithi064 ceph-mon[42243]: 9.b scrub ok 2024-03-24T12:04:54.167 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1841888841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:54.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:54.261 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:54.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:54 smithi193 ceph-mon[33593]: 9.b scrub ok 2024-03-24T12:04:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1841888841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:04:54.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:54.396 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:54.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:54.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:54.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:54.705 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:54.739 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:54.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:54.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:54.878 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:54.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:54.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:55.014 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:55.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:55.116 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:55.152 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:55.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:55.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:55.294 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:55.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:55.396 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:55.434 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:55.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:55.534 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: pgmap v575: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 2.2 scrub starts 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 2.2 scrub ok 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 9.9 scrub starts 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T12:04:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: pgmap v575: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 2.2 scrub starts 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 2.2 scrub ok 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 9.9 scrub starts 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T12:04:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:55 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T12:04:55.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:55.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: pgmap v575: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T12:04:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T12:04:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 2.2 scrub starts 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 2.2 scrub ok 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 9.9 scrub starts 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T12:04:55.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:55 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T12:04:55.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:55.699 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:55.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:55.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:55.836 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:55.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:55.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:55.979 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:55.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:56.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:56.108 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:56.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:56.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:56.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:56.252 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:56.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:56.354 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:56.394 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:56.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:56.496 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:56.534 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:56.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:56.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:04:56.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:04:56.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T12:04:56.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T12:04:56.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[30223]: 6.b scrub starts 2024-03-24T12:04:56.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:04:56.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:04:56.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T12:04:56.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T12:04:56.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:56 smithi064 ceph-mon[42243]: 6.b scrub starts 2024-03-24T12:04:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:56 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:04:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:56 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:04:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:56 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T12:04:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:56 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T12:04:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:56 smithi193 ceph-mon[33593]: 6.b scrub starts 2024-03-24T12:04:56.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:56.672 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:56.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:56.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:04:56.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:56.862 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:56.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:56.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:04:57.004 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:57.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:04:57.106 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:57.143 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:57.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:57.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:57.280 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:57.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:57.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[30223]: pgmap v576: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[30223]: 6.b scrub ok 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[30223]: 9.7 scrub starts 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[30223]: 9.9 scrub ok 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[30223]: 1.6 scrub starts 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[30223]: 1.6 scrub ok 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[42243]: pgmap v576: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[42243]: 6.b scrub ok 2024-03-24T12:04:57.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[42243]: 9.7 scrub starts 2024-03-24T12:04:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[42243]: 9.9 scrub ok 2024-03-24T12:04:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[42243]: 1.6 scrub starts 2024-03-24T12:04:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[42243]: 1.6 scrub ok 2024-03-24T12:04:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:57.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:57.420 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:57.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:57.451 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:57 smithi193 ceph-mon[33593]: pgmap v576: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-24T12:04:57.451 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:57 smithi193 ceph-mon[33593]: 6.b scrub ok 2024-03-24T12:04:57.451 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:57 smithi193 ceph-mon[33593]: 9.7 scrub starts 2024-03-24T12:04:57.451 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:57 smithi193 ceph-mon[33593]: 9.9 scrub ok 2024-03-24T12:04:57.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:57 smithi193 ceph-mon[33593]: 1.6 scrub starts 2024-03-24T12:04:57.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:57 smithi193 ceph-mon[33593]: 1.6 scrub ok 2024-03-24T12:04:57.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:57.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:57.565 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:57.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:57.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:57.681 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:04:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:04:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:57.702 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:57.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:57.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:57.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:04:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:04:57.860 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:57.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:57.899 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:04:57.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:58.001 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:58.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:58.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:58.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:58.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:58.290 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:58.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:58.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:58.441 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:58.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:58.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:58.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:58.575 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:58.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:58.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:58.716 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:58.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:58.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:58.855 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:58.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:58.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:58.992 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:58.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:59.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:04:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:59.129 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:59.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:04:59.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:04:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:59.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:04:59.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:04:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:59.404 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:59.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:04:59.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:04:59.524 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[30223]: pgmap v577: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 991 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T12:04:59.524 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:59.524 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T12:04:59.524 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T12:04:59.524 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T12:04:59.524 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T12:04:59.524 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[30223]: 9.7 scrub ok 2024-03-24T12:04:59.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[42243]: pgmap v577: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 991 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T12:04:59.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:59.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T12:04:59.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T12:04:59.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T12:04:59.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T12:04:59.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:04:59 smithi064 ceph-mon[42243]: 9.7 scrub ok 2024-03-24T12:04:59.549 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:59.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:04:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:59 smithi193 ceph-mon[33593]: pgmap v577: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 991 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T12:04:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:59 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T12:04:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:59 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T12:04:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:59 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T12:04:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:59 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T12:04:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:04:59 smithi193 ceph-mon[33593]: 9.7 scrub ok 2024-03-24T12:04:59.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:59.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:04:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:59.690 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:59.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:04:59.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:04:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:04:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:59.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:04:59.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:04:59.951 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:04:59.983 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:04:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:04:59.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:00.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:00.179 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:00.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:00.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:00.281 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:05:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:05:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[30223]: 2.12 scrub starts 2024-03-24T12:05:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[30223]: 2.12 scrub ok 2024-03-24T12:05:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[30223]: 9.5 scrub starts 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[42243]: 2.12 scrub starts 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[42243]: 2.12 scrub ok 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:05:00.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:00 smithi064 ceph-mon[42243]: 9.5 scrub starts 2024-03-24T12:05:00.316 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:00.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:00.418 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:00.454 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:00.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:00.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:00 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:05:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:00 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:05:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:00 smithi193 ceph-mon[33593]: 2.12 scrub starts 2024-03-24T12:05:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:00 smithi193 ceph-mon[33593]: 2.12 scrub ok 2024-03-24T12:05:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:00 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:05:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:00 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:05:00.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:00 smithi193 ceph-mon[33593]: 9.5 scrub starts 2024-03-24T12:05:00.641 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:00.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:00.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:00.779 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:00.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:00.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:00.919 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:00.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:01.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:01.066 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:01.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:01.168 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:01.210 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:01.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:01.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:01 smithi064 ceph-mon[42243]: pgmap v578: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:01.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:01 smithi064 ceph-mon[42243]: 9.a scrub starts 2024-03-24T12:05:01.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:01 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T12:05:01.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:01 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T12:05:01.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:01 smithi064 ceph-mon[30223]: pgmap v578: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:01.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:01 smithi064 ceph-mon[30223]: 9.a scrub starts 2024-03-24T12:05:01.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:01 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T12:05:01.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:01 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T12:05:01.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:01.356 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:01.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:01.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:01.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:01.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:01 smithi193 ceph-mon[33593]: pgmap v578: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:01 smithi193 ceph-mon[33593]: 9.a scrub starts 2024-03-24T12:05:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:01 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T12:05:01.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:01 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T12:05:01.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:01.641 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:01.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:01.746 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:01.781 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:01.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215974 7.12 7:49827d29:::benchmark_data_smithi064_87013_object95017:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.912925+0000", 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 436.91644057000002, 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1524356259999999, 2024-03-24T12:05:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215974 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912925+0000", 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912925+0000", 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912927+0000", 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9910000000000001e-06 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912932+0000", 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5979999999999999e-06 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.004832+0000", 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091900446999999996 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.004846+0000", 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4114000000000001e-05 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009898+0000", 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050523190000000004 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009908+0000", 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.065275+0000", 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0553668060000001 2024-03-24T12:05:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.065361+0000", 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5686999999999995e-05 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30713 8.7 8:e176b16f:::benchmark_data_smithi064_120708_object30712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.805040+0000", 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 349.02432588599999, 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.131868809, 2024-03-24T12:05:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30713 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805040+0000", 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805040+0000", 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805041+0000", 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805077+0000", 2024-03-24T12:05:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5411999999999999e-05 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805092+0000", 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.575e-05 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.805101+0000", 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8470000000000002e-06 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.806250+0000", 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011484959999999999 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.806361+0000", 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011084 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.806848+0000", 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000487164 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.810513+0000", 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036655379999999999 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814134+0000", 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003620393 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936775+0000", 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12264096400000001 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936843+0000", 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8485999999999994e-05 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936908+0000", 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5298999999999999e-05 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30732 8.7 8:e01cf41c:::benchmark_data_smithi064_120708_object30731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.811503+0000", 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 349.01786247799998, 2024-03-24T12:05:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763239600000001, 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30732 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811503+0000", 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811503+0000", 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811504+0000", 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.02e-07 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811524+0000", 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.986e-05 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811531+0000", 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0079999999999996e-06 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.811536+0000", 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5430000000000001e-06 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814205+0000", 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026690009999999998 2024-03-24T12:05:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814384+0000", 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017845 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814516+0000", 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013214100000000001 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936989+0000", 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.122473472 2024-03-24T12:05:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937025+0000", 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5799000000000001e-05 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939021+0000", 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019962000000000001 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939102+0000", 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939135+0000", 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3544999999999998e-05 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30738 8.a 8:522e4dbb:::benchmark_data_smithi064_120708_object30737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.814514+0000", 2024-03-24T12:05:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 349.01485172299999, 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13393280499999999, 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30738 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814514+0000", 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814514+0000", 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814516+0000", 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1040000000000001e-06 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814539+0000", 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3587999999999998e-05 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814552+0000", 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.222e-05 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814560+0000", 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7499999999999992e-06 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814623+0000", 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2510999999999998e-05 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936947+0000", 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.122324012 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937574+0000", 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000626741 2024-03-24T12:05:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.940677+0000", 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031032949999999998 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.943485+0000", 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028083100000000001 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.947014+0000", 2024-03-24T12:05:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035284399999999999 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.948401+0000", 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013869589999999999 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.948447+0000", 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5874999999999997e-05 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30739 8.3 8:cfd8eaff:::benchmark_data_smithi064_120708_object30738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.814697+0000", 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 349.01466807600002, 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130831052, 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30739 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814697+0000", 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814697+0000", 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814699+0000", 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814720+0000", 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1676e-05 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814729+0000", 2024-03-24T12:05:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6400000000000003e-06 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.814737+0000", 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7049999999999999e-06 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937094+0000", 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.122357138 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937377+0000", 2024-03-24T12:05:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028292399999999999 2024-03-24T12:05:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.937530+0000", 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153752 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939685+0000", 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021549360000000001 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.942092+0000", 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024068639999999999 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945380+0000", 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003288146 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945480+0000", 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9499000000000004e-05 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945528+0000", 2024-03-24T12:05:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8529000000000002e-05 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:30741 8.3 8:c156d5da:::benchmark_data_smithi064_120708_object30740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:12.815142+0000", 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 349.01422330499997, 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13056442700000001, 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30741 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.815142+0000", 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.815142+0000", 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.815144+0000", 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.815165+0000", 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1237999999999999e-05 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936625+0000", 2024-03-24T12:05:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121460399 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.936650+0000", 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4215000000000001e-05 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939720+0000", 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030704870000000002 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939835+0000", 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115309 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.939996+0000", 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016097600000000001 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.942080+0000", 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002083807 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.943585+0000", 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001505008 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945578+0000", 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019926800000000001 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945667+0000", 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9141000000000006e-05 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:12.945707+0000", 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9694000000000003e-05 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:31212 8.3 8:cc964d41:::benchmark_data_smithi064_120708_object31211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:13.148461+0000", 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 348.68090488199999, 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.129072409, 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31212 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.148461+0000", 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.148461+0000", 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.148462+0000", 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6279999999999999e-06 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.148485+0000", 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2901999999999999e-05 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.258790+0000", 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110304771 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.258810+0000", 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0080000000000001e-05 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.260729+0000", 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019188930000000001 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.260872+0000", 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014303800000000001 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.261004+0000", 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013254 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.263354+0000", 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023496070000000001 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.263481+0000", 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012694099999999999 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.265111+0000", 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016302350000000001 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.265281+0000", 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016961799999999999 2024-03-24T12:05:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:13.277533+0000", 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012252156 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:34129 8.3 8:ccbd3bd8:::benchmark_data_smithi064_120708_object34128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.762014+0000", 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 347.067351831, 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16982466299999999, 2024-03-24T12:05:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34129 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762014+0000", 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762014+0000", 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762015+0000", 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.734e-06 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762050+0000", 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4314999999999999e-05 2024-03-24T12:05:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762061+0000", 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0957e-05 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762069+0000", 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5369999999999997e-06 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.764178+0000", 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002108651 2024-03-24T12:05:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.764606+0000", 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042792699999999998 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.764779+0000", 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017316099999999999 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928527+0000", 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.163748002 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928538+0000", 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1559000000000001e-05 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931737+0000", 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031986919999999999 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931793+0000", 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6141999999999999e-05 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931838+0000", 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4985999999999999e-05 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:34130 8.b e176/170 8:dce7680e:::benchmark_data_smithi064_120708_object34129:head v 176'2067, mlcod=176'2065)", 2024-03-24T12:05:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.762745+0000", 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 347.066620801, 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16556532800000001, 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762745+0000", 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762745+0000", 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762746+0000", 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.001e-06 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762774+0000", 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8359000000000001e-05 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762783+0000", 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6500000000000002e-06 2024-03-24T12:05:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.762789+0000", 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2840000000000003e-06 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.763014+0000", 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022540099999999999 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.763033+0000", 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8423999999999998e-05 2024-03-24T12:05:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766313+0000", 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00327999 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928310+0000", 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.161997219 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:34127 8.f e176/170 8:f45926e9:::benchmark_data_smithi064_120708_object34126:head v 176'2120, mlcod=176'2115)", 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.765432+0000", 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 347.06393345399999, 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16544773300000001, 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765432+0000", 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765432+0000", 2024-03-24T12:05:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765433+0000", 2024-03-24T12:05:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7400000000000003e-07 2024-03-24T12:05:01.889 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765457+0000", 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.455e-05 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765465+0000", 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6709999999999999e-06 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765471+0000", 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8109999999999999e-06 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765567+0000", 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5957000000000003e-05 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.765586+0000", 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9292000000000001e-05 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.930801+0000", 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16521519200000001 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.930880+0000", 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8685999999999998e-05 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35033.0:34132 8.f e176/170 8:f6dfb55d:::benchmark_data_smithi064_120708_object34131:head v 176'2121, mlcod=176'2115)", 2024-03-24T12:05:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.766181+0000", 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 347.06318426500002, 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16485608900000001, 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766181+0000", 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766181+0000", 2024-03-24T12:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766182+0000", 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0950000000000001e-06 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766206+0000", 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3589000000000001e-05 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766215+0000", 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9779999999999994e-06 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.766222+0000", 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6819999999999997e-06 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928430+0000", 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.162208239 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928454+0000", 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4677000000000001e-05 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931008+0000", 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025538549999999998 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931037+0000", 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8974000000000001e-05 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:34140 8.a 8:540add6c:::benchmark_data_smithi064_120708_object34139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.767290+0000", 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 347.06207513999999, 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18283492000000001, 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34140 2024-03-24T12:05:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.767290+0000", 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.767290+0000", 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.767293+0000", 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7209999999999999e-06 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.767316+0000", 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2836999999999998e-05 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928285+0000", 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16096931 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928302+0000", 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7053000000000001e-05 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928700+0000", 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039811700000000002 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928828+0000", 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127329 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:05:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.929223+0000", 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039511 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.930744+0000", 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015214829999999999 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.932965+0000", 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022207749999999999 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.949980+0000", 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017014486999999998 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.950081+0000", 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010141699999999999 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.950125+0000", 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4280999999999998e-05 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35033.0:34129 8.3 e176/170 ondisk, result = 0)", 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.768222+0000", 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 347.06114370500001, 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16362191400000001, 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.768222+0000", 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.768222+0000", 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.768223+0000", 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.139e-06 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.768227+0000", 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9809999999999997e-06 2024-03-24T12:05:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928596+0000", 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16036869000000001 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928604+0000", 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1880000000000002e-06 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931725+0000", 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003121268 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931732+0000", 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8900000000000001e-06 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.931844+0000", 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111758 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.35033.0:34129 8.3 e176/170 ondisk, result = 0)", 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:14.769164+0000", 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 347.06020134699997, 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15937512700000001, 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.769164+0000", 2024-03-24T12:05:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.769164+0000", 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.769165+0000", 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.105e-06 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.769169+0000", 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9530000000000001e-06 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928286+0000", 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15911722 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928301+0000", 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4228e-05 2024-03-24T12:05:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928515+0000", 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021393700000000001 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928521+0000", 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6579999999999996e-06 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:14.928539+0000", 2024-03-24T12:05:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8026e-05 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:35795 8.7 8:eacf5fe1:::benchmark_data_smithi064_120708_object35794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:15.841335+0000", 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.98803065999999, 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14967005899999999, 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35795 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841335+0000", 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841335+0000", 2024-03-24T12:05:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841336+0000", 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841359+0000", 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3053000000000001e-05 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841366+0000", 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4250000000000001e-06 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841373+0000", 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1049999999999997e-06 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841459+0000", 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6024000000000002e-05 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841630+0000", 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017069700000000001 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.841788+0000", 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158407 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.843971+0000", 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00218286 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.844208+0000", 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023623000000000001 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.989481+0000", 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14527379400000001 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.990963+0000", 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00148116 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:15.991005+0000", 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2361999999999999e-05 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104598 8.17 8:eb9d3bdd:::benchmark_data_smithi064_120708_object104597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.755205+0000", 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.07416038899999, 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.088392112, 2024-03-24T12:05:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104598 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755205+0000", 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755205+0000", 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755208+0000", 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6960000000000001e-06 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755325+0000", 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117261 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755338+0000", 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3380000000000001e-05 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755352+0000", 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3876999999999999e-05 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755418+0000", 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5297999999999997e-05 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755425+0000", 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2949999999999996e-06 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.824824+0000", 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.069399319 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.825008+0000", 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018328800000000001 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.825376+0000", 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036875500000000002 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.830186+0000", 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048092509999999996 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.830236+0000", 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0263000000000003e-05 2024-03-24T12:05:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832844+0000", 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026079599999999999 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.833622+0000", 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000778238 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.843390+0000", 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097680479999999997 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.843462+0000", 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1722000000000002e-05 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.843597+0000", 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135461 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104612 8.17 8:e8feba9a:::benchmark_data_smithi064_120708_object104611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.755373+0000", 2024-03-24T12:05:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.073992073, 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0905727569999999, 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104612 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755373+0000", 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755373+0000", 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755374+0000", 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0400000000000005e-07 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755396+0000", 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1469000000000001e-05 2024-03-24T12:05:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755405+0000", 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7530000000000007e-06 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755412+0000", 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2080000000000003e-06 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755458+0000", 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6656999999999997e-05 2024-03-24T12:05:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755467+0000", 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9339999999999995e-06 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.827928+0000", 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.07246029 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.828092+0000", 2024-03-24T12:05:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164038 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.828454+0000", 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036203500000000003 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832895+0000", 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044415230000000002 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832911+0000", 2024-03-24T12:05:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5593000000000001e-05 2024-03-24T12:05:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832949+0000", 2024-03-24T12:05:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8111999999999997e-05 2024-03-24T12:05:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.833669+0000", 2024-03-24T12:05:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071961200000000001 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.845732+0000", 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012063212 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.845802+0000", 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0227000000000004e-05 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.845946+0000", 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014418999999999999 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104620 8.17 8:ed89d4bc:::benchmark_data_smithi064_120708_object104619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.755421+0000", 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.07394431, 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.094105661, 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104620 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755421+0000", 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755421+0000", 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755422+0000", 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3099999999999997e-07 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755444+0000", 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1926e-05 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755451+0000", 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1119999999999998e-06 2024-03-24T12:05:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755457+0000", 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1990000000000003e-06 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755499+0000", 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1832999999999997e-05 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:05:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.755508+0000", 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.037e-06 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.830590+0000", 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0750823780000001 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.830765+0000", 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174134 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831252+0000", 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048734400000000002 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.833578+0000", 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002326355 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.833636+0000", 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7837000000000002e-05 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.838464+0000", 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048280160000000001 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.842219+0000", 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037549889999999998 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.849143+0000", 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069238360000000001 2024-03-24T12:05:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.849381+0000", 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000237722 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.849527+0000", 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000146212 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:144410 8.17 8:eb4d00be:::benchmark_data_smithi064_120708_object22384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.16290160599999, 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17982219199999999, 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 144410 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666465+0000", 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.615e-06 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666470+0000", 2024-03-24T12:05:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2420000000000002e-06 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666477+0000", 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6329999999999996e-06 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666484+0000", 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2450000000000003e-06 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669073+0000", 2024-03-24T12:05:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002589755 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669197+0000", 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012349999999999999 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669550+0000", 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035300000000000002 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.710796+0000", 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041246370999999997 2024-03-24T12:05:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711483+0000", 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068668600000000005 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711540+0000", 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7515000000000001e-05 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711848+0000", 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030785799999999999 2024-03-24T12:05:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.845781+0000", 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13393276200000001 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.845858+0000", 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7063000000000004e-05 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846286+0000", 2024-03-24T12:05:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042794699999999997 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:144423 8.17 8:ecb5801d:::benchmark_data_smithi064_120708_object22397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:52.711278+0000", 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.11808747500001, 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13569683299999999, 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 144423 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711278+0000", 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711278+0000", 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711279+0000", 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.265e-06 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711284+0000", 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7319999999999998e-06 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711293+0000", 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1030000000000008e-06 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711301+0000", 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2449999999999998e-06 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711897+0000", 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059516500000000004 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.712007+0000", 2024-03-24T12:05:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110769 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.712386+0000", 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037831600000000001 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.716109+0000", 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037229670000000002 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.716136+0000", 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7500000000000001e-05 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.716177+0000", 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0615999999999999e-05 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.716748+0000", 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057089300000000005 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846770+0000", 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130021991 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846833+0000", 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3149000000000001e-05 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846975+0000", 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014212200000000001 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:01.955 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:05:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:01.955 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:01.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:02.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:02.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:02.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:02.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.7 to 1 (10000) 2024-03-24T12:05:02.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:02.234 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:02.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:02.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:02.373 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:02.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:02.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:02.511 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:02.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:02.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[42243]: 9.5 scrub ok 2024-03-24T12:05:02.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[42243]: 4.1d scrub starts 2024-03-24T12:05:02.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[42243]: 4.1d scrub ok 2024-03-24T12:05:02.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[42243]: 9.a scrub ok 2024-03-24T12:05:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3256402748' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:05:02.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[42243]: from='client.35438 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:05:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[30223]: 9.5 scrub ok 2024-03-24T12:05:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[30223]: 4.1d scrub starts 2024-03-24T12:05:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[30223]: 4.1d scrub ok 2024-03-24T12:05:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[30223]: 9.a scrub ok 2024-03-24T12:05:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3256402748' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:05:02.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:02 smithi064 ceph-mon[30223]: from='client.35438 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:05:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:02 smithi193 ceph-mon[33593]: 9.5 scrub ok 2024-03-24T12:05:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:02 smithi193 ceph-mon[33593]: 4.1d scrub starts 2024-03-24T12:05:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:02 smithi193 ceph-mon[33593]: 4.1d scrub ok 2024-03-24T12:05:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:02 smithi193 ceph-mon[33593]: 9.a scrub ok 2024-03-24T12:05:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:02 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3256402748' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:05:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:02 smithi193 ceph-mon[33593]: from='client.35438 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:05:02.613 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:02.645 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:02.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:02.747 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:02.818 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:02.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:02.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:02.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:03.058 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:03.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:03.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:03.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:03.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:03.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:05:03.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:03.368 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:03.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:03.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: pgmap v579: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:05:03.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: 4.c scrub starts 2024-03-24T12:05:03.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: 4.c scrub ok 2024-03-24T12:05:03.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: from='client.35438 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:05:03.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T12:05:03.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: 1.f scrub starts 2024-03-24T12:05:03.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: 1.f scrub ok 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: pgmap v579: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: 4.c scrub starts 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: 4.c scrub ok 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: from='client.35438 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: 1.f scrub starts 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: 1.f scrub ok 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T12:05:03.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:03 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T12:05:03.470 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:03.503 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:03.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: pgmap v579: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:05:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: 4.c scrub starts 2024-03-24T12:05:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: 4.c scrub ok 2024-03-24T12:05:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: from='client.35438 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:05:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T12:05:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: 1.f scrub starts 2024-03-24T12:05:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: 1.f scrub ok 2024-03-24T12:05:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T12:05:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:03 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T12:05:03.606 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:03.639 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:03.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:03.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:03.788 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:03.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:03.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:03.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:03.935 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:03.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:04.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:04.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:04.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:04.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:04.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:04.351 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:04.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:04.356 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-24T12:05:04.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:05:04.440 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:04 smithi064 ceph-mon[30223]: 6.17 scrub starts 2024-03-24T12:05:04.440 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:04 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T12:05:04.440 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:04 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T12:05:04.440 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:04 smithi064 ceph-mon[30223]: 6.17 scrub ok 2024-03-24T12:05:04.440 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:04 smithi064 ceph-mon[42243]: 6.17 scrub starts 2024-03-24T12:05:04.440 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:04 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T12:05:04.440 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:04 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T12:05:04.440 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:04 smithi064 ceph-mon[42243]: 6.17 scrub ok 2024-03-24T12:05:04.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:04.504 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:04.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:04.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:04 smithi193 ceph-mon[33593]: 6.17 scrub starts 2024-03-24T12:05:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:04 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T12:05:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:04 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T12:05:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:04 smithi193 ceph-mon[33593]: 6.17 scrub ok 2024-03-24T12:05:04.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:04.652 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:04.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:04.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:04.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:04.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:04.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:04.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:04.946 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:04.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:05.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:05.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:05.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:05.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:05.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:05.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:05.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:05.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:05.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:05.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:05.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[42243]: pgmap v581: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[30223]: pgmap v581: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T12:05:05.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:05 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T12:05:05.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:05.501 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:05.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:05 smithi193 ceph-mon[33593]: pgmap v581: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:05:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:05 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:05:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:05 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:05:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:05 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T12:05:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:05 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T12:05:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:05 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T12:05:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:05 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T12:05:05.602 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:05.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:05.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:05.738 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:05.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:05.774 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:05.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:05.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:05.912 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:05.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:06.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:06.063 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:06.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:06.165 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:06.188 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:06.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:06.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:06.329 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:06.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:06.430 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:06.466 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:06.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:06.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:05:06.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:05:06.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T12:05:06.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T12:05:06.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T12:05:06.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T12:05:06.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[30223]: 9.c scrub starts 2024-03-24T12:05:06.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:05:06.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:05:06.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T12:05:06.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T12:05:06.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T12:05:06.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T12:05:06.528 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:06 smithi064 ceph-mon[42243]: 9.c scrub starts 2024-03-24T12:05:06.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:06 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:05:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:06 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:05:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:06 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T12:05:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:06 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T12:05:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:06 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T12:05:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:06 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T12:05:06.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:06 smithi193 ceph-mon[33593]: 9.c scrub starts 2024-03-24T12:05:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:06.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:06.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:06.747 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:06.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:06.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:06.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:06.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:06.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:06.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:07.017 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:07.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:07.118 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:07.147 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:07.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:07.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:07.279 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:07.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:07.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:07.412 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:07.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:07.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[30223]: pgmap v582: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:05:07.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T12:05:07.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T12:05:07.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T12:05:07.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T12:05:07.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T12:05:07.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T12:05:07.436 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:05:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:07.437 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[42243]: pgmap v582: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:05:07.437 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T12:05:07.437 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T12:05:07.437 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T12:05:07.437 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T12:05:07.437 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T12:05:07.437 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:07 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T12:05:07.461 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:07 smithi193 ceph-mon[33593]: pgmap v582: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T12:05:07.461 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:07 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T12:05:07.461 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:07 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T12:05:07.461 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:07 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T12:05:07.461 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:07 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T12:05:07.461 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:07 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T12:05:07.461 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:07 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T12:05:07.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:07.560 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:07.640 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:07.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:07.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:07.787 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:07.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:07.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:05:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:05:07.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:07.930 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:07.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:08.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:08.073 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:08.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:08.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:08.187 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:05:08.208 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:08.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:08 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T12:05:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:08 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T12:05:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:08 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2428830578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:08 smithi064 ceph-mon[30223]: from='client.25450 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:08 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T12:05:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:08 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T12:05:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:08 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2428830578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:08 smithi064 ceph-mon[42243]: from='client.25450 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:08.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:08.348 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:08.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:08.450 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:08.483 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:08.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:08.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:08.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:08 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T12:05:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:08 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T12:05:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:08 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2428830578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:08 smithi193 ceph-mon[33593]: from='client.25450 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:09.431 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:09.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:09.532 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: pgmap v583: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: 9.c scrub ok 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: 1.d scrub starts 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: 1.d scrub ok 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: from='client.25450 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T12:05:09.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2985038692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: pgmap v583: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: 9.c scrub ok 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: 1.d scrub starts 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: 1.d scrub ok 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:05:09.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: from='client.25450 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:05:09.539 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T12:05:09.539 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:09 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2985038692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:09.572 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:09.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: pgmap v583: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: 9.c scrub ok 2024-03-24T12:05:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:05:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:05:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: 1.d scrub starts 2024-03-24T12:05:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: 1.d scrub ok 2024-03-24T12:05:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:05:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:05:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: from='client.25450 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:05:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T12:05:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:09 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2985038692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:09.673 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:09.709 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:09.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:09.810 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:09.842 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:09.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:09.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:09.973 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:09.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:10.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:10.107 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:10.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:10.209 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:10.235 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:10.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[30223]: 2.1c deep-scrub starts 2024-03-24T12:05:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[30223]: 2.1c deep-scrub ok 2024-03-24T12:05:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[30223]: 9.f deep-scrub starts 2024-03-24T12:05:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[30223]: pgmap v585: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:05:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[30223]: 4.3 scrub starts 2024-03-24T12:05:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[30223]: 4.3 scrub ok 2024-03-24T12:05:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[42243]: 2.1c deep-scrub starts 2024-03-24T12:05:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[42243]: 2.1c deep-scrub ok 2024-03-24T12:05:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[42243]: 9.f deep-scrub starts 2024-03-24T12:05:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[42243]: pgmap v585: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:05:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[42243]: 4.3 scrub starts 2024-03-24T12:05:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:10 smithi064 ceph-mon[42243]: 4.3 scrub ok 2024-03-24T12:05:10.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:10.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:10.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:05:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:05:10.502 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:05:10.502 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:05:10.502 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:05:10.502 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:05:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:10.514 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:10.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:10 smithi193 ceph-mon[33593]: 2.1c deep-scrub starts 2024-03-24T12:05:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:10 smithi193 ceph-mon[33593]: 2.1c deep-scrub ok 2024-03-24T12:05:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:10 smithi193 ceph-mon[33593]: 9.f deep-scrub starts 2024-03-24T12:05:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:10 smithi193 ceph-mon[33593]: pgmap v585: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:05:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:10 smithi193 ceph-mon[33593]: 4.3 scrub starts 2024-03-24T12:05:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:10 smithi193 ceph-mon[33593]: 4.3 scrub ok 2024-03-24T12:05:10.615 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:10.648 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:10.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:10.750 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:10.785 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:10.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:10.886 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:10.918 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:10.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:11.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:11.058 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:11.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:11.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:11.196 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:11.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:11.297 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:11.329 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:11.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:11.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:11.465 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:11.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T12:05:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T12:05:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[30223]: 6.7 scrub starts 2024-03-24T12:05:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[30223]: 6.7 scrub ok 2024-03-24T12:05:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T12:05:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T12:05:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T12:05:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T12:05:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[42243]: 6.7 scrub starts 2024-03-24T12:05:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[42243]: 6.7 scrub ok 2024-03-24T12:05:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T12:05:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:11 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T12:05:11.566 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:11 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T12:05:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:11 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T12:05:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:11 smithi193 ceph-mon[33593]: 6.7 scrub starts 2024-03-24T12:05:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:11 smithi193 ceph-mon[33593]: 6.7 scrub ok 2024-03-24T12:05:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:11 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T12:05:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:11 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T12:05:11.602 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:11.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:11.703 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:11.735 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:11.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:11.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:11.872 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:11.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:11.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:12.008 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:12.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:12.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:05:12.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:12.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:12.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:12.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:12.286 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:12.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:05:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: 1.a scrub starts 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: 1.a scrub ok 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: pgmap v586: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 553 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: 1.a scrub starts 2024-03-24T12:05:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: 1.a scrub ok 2024-03-24T12:05:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: pgmap v586: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 553 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:05:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T12:05:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T12:05:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:12.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:12.386 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:12.422 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:12.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:12.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:12.558 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:12.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:05:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:05:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: 1.a scrub starts 2024-03-24T12:05:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: 1.a scrub ok 2024-03-24T12:05:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: pgmap v586: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 553 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:05:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T12:05:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T12:05:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:12.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:12.693 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:12.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:12.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:12.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:12.934 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:12.967 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:12.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:13.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:13.106 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:13.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:13.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:13.245 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:13.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:13.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:13.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:13.385 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:13.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:13.480 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:13 smithi064 ceph-mon[30223]: 9.6 scrub starts 2024-03-24T12:05:13.480 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:13 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T12:05:13.480 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:13 smithi064 ceph-mon[42243]: 9.6 scrub starts 2024-03-24T12:05:13.480 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:13 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T12:05:13.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:13.524 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:13.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:13 smithi193 ceph-mon[33593]: 9.6 scrub starts 2024-03-24T12:05:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:13 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T12:05:13.627 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:13.657 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:13.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:13.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:13.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:13.886 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:13.917 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:13.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:14.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:14.057 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:14.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:14.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:14.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:14.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:14.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:14.326 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:14.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:14.428 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:14.464 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:14.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T12:05:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: 5.18 deep-scrub starts 2024-03-24T12:05:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: 5.18 deep-scrub ok 2024-03-24T12:05:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: pgmap v587: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: 9.6 scrub ok 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: 6.0 scrub starts 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[30223]: 6.0 scrub ok 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: 5.18 deep-scrub starts 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: 5.18 deep-scrub ok 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: pgmap v587: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: 9.6 scrub ok 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: 6.0 scrub starts 2024-03-24T12:05:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:14 smithi064 ceph-mon[42243]: 6.0 scrub ok 2024-03-24T12:05:14.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T12:05:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: 5.18 deep-scrub starts 2024-03-24T12:05:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: 5.18 deep-scrub ok 2024-03-24T12:05:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:05:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:05:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: pgmap v587: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-24T12:05:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: 9.6 scrub ok 2024-03-24T12:05:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: 6.0 scrub starts 2024-03-24T12:05:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:14 smithi193 ceph-mon[33593]: 6.0 scrub ok 2024-03-24T12:05:14.602 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:14.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:14.703 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:14.733 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:14.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:14.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:14.870 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:14.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:14.972 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:15.008 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:15.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:15.110 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:15.147 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:15.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:15.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:15.285 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:15.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:15.387 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:15.421 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:15.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:15.524 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:15.548 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:15.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T12:05:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[30223]: 4.a deep-scrub starts 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[30223]: 4.a deep-scrub ok 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[42243]: 4.a deep-scrub starts 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[42243]: 4.a deep-scrub ok 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T12:05:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:15 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T12:05:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:15 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T12:05:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:15 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T12:05:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:15 smithi193 ceph-mon[33593]: 4.a deep-scrub starts 2024-03-24T12:05:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:15 smithi193 ceph-mon[33593]: 4.a deep-scrub ok 2024-03-24T12:05:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:15 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T12:05:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:15 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T12:05:15.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:15.676 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:15.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:15.778 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:15.813 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:15.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:15.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:15.951 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:15.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:16.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:16.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:16.205 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:16.244 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:16.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:16.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:16 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T12:05:16.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:16 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T12:05:16.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:16 smithi064 ceph-mon[30223]: 9.4 scrub starts 2024-03-24T12:05:16.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:16 smithi064 ceph-mon[30223]: pgmap v588: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-24T12:05:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:16 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T12:05:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:16 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T12:05:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:16 smithi064 ceph-mon[42243]: 9.4 scrub starts 2024-03-24T12:05:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:16 smithi064 ceph-mon[42243]: pgmap v588: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-24T12:05:16.346 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:16.377 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:16.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:16.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:16.515 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:16.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:16 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T12:05:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:16 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T12:05:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:16 smithi193 ceph-mon[33593]: 9.4 scrub starts 2024-03-24T12:05:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:16 smithi193 ceph-mon[33593]: pgmap v588: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-24T12:05:16.617 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:16.652 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:16.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:16.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:16.788 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:16.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:16.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:16.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:16.937 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:16.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:17.038 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:17.071 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:17.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:17.172 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:17.205 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:17.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:17.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:17.338 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:17.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:17.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T12:05:17.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T12:05:17.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 9.4 scrub ok 2024-03-24T12:05:17.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 1.9 deep-scrub starts 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 1.9 deep-scrub ok 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 5.1e scrub starts 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[30223]: 5.1e scrub ok 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 9.4 scrub ok 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 1.9 deep-scrub starts 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 1.9 deep-scrub ok 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 5.1e scrub starts 2024-03-24T12:05:17.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-mon[42243]: 5.1e scrub ok 2024-03-24T12:05:17.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 9.4 scrub ok 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 1.9 deep-scrub starts 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 1.9 deep-scrub ok 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 5.1e scrub starts 2024-03-24T12:05:17.472 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-mon[33593]: 5.1e scrub ok 2024-03-24T12:05:17.473 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:17.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:17.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:17.617 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:17.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:05:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:05:17.636 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:05:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:05:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:17.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:17.747 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:17.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:17.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:05:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:05:17.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:17.881 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:17.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:17.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:18.026 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:18.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:18.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:18.161 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:18.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:18.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:18.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:18.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:18.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:18.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:18.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:18.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: 6.15 deep-scrub starts 2024-03-24T12:05:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: 6.15 deep-scrub ok 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: 4.8 scrub starts 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: 4.8 scrub ok 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: pgmap v589: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: 6.15 deep-scrub starts 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:05:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: 6.15 deep-scrub ok 2024-03-24T12:05:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: 4.8 scrub starts 2024-03-24T12:05:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: 4.8 scrub ok 2024-03-24T12:05:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: pgmap v589: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-24T12:05:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T12:05:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:18 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T12:05:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:18.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:18.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:18.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: 6.15 deep-scrub starts 2024-03-24T12:05:18.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:05:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:05:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: 6.15 deep-scrub ok 2024-03-24T12:05:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: 4.8 scrub starts 2024-03-24T12:05:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: 4.8 scrub ok 2024-03-24T12:05:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: pgmap v589: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-24T12:05:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T12:05:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:18 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T12:05:18.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:05:18.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:18.710 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:18.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:18.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:18.852 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:18.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:18.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:18.998 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:18.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:19.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:19.135 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:19.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:05:19.225 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:05:19.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:19.311 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:19.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:19.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:19.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:19.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:19.546 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T12:05:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T12:05:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[30223]: from='client.35453 v1:172.21.15.64:0/680314057' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:05:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T12:05:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[42243]: from='client.35453 v1:172.21.15.64:0/680314057' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T12:05:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:19 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T12:05:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:19.579 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:19.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:19 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T12:05:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:19 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T12:05:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:19 smithi193 ceph-mon[33593]: from='client.35453 v1:172.21.15.64:0/680314057' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:05:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:19 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T12:05:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:19 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T12:05:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:19 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T12:05:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:19 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T12:05:19.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:19.709 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:19.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:19.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:19.846 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:19.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:19.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:19.979 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:19.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:20.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:20.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:20.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:20.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:20.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:05:20.274 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:20.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:20.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:20.409 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:20.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:20.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:20.545 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:20.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:20 smithi064 ceph-mon[30223]: 1.8 scrub starts 2024-03-24T12:05:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:20 smithi064 ceph-mon[30223]: 1.8 scrub ok 2024-03-24T12:05:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:20 smithi064 ceph-mon[30223]: pgmap v590: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 913 B/s rd, 0 B/s wr, 996 op/s 2024-03-24T12:05:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:20 smithi064 ceph-mon[42243]: 1.8 scrub starts 2024-03-24T12:05:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:20 smithi064 ceph-mon[42243]: 1.8 scrub ok 2024-03-24T12:05:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:20 smithi064 ceph-mon[42243]: pgmap v590: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 913 B/s rd, 0 B/s wr, 996 op/s 2024-03-24T12:05:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:20 smithi193 ceph-mon[33593]: 1.8 scrub starts 2024-03-24T12:05:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:20 smithi193 ceph-mon[33593]: 1.8 scrub ok 2024-03-24T12:05:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:20 smithi193 ceph-mon[33593]: pgmap v590: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 913 B/s rd, 0 B/s wr, 996 op/s 2024-03-24T12:05:20.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:20.686 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:20.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:20.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:20.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:20.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:20.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:20.957 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:20.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:21.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:21.091 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:21.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:21.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:21.231 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:21.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:21.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:21.381 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2024-03-24T12:05:17.383737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4308015478,"num_objects":65925,"num_object_clones":0,"num_object_copies":197775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65925,"num_whiteouts":0,"num_read":1508,"num_read_kb":1260,"num_write":186617,"num_write_kb":8051886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77342,"ondisk_log_size":77342,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16440488,"kb_used_data":12596520,"kb_used_omap":12,"kb_used_meta":3843635,"kb_avail":733520728,"statfs":{"total":767960285184,"available":751125225472,"internally_reserved":0,"allocated":12898836480,"data_stored":12895371962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3935882800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-768606208,"num_objects":-11728,"num_object_clones":0,"num_object_copies":-35184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11728,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11728,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003250"},"pg_stats":[{"pgid":"1.d","version":"83'130","reported_seq":392,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178615+0000","last_change":"2024-03-24T12:05:08.153739+0000","last_active":"2024-03-24T12:05:08.178615+0000","last_peered":"2024-03-24T12:05:08.178615+0000","last_clean":"2024-03-24T12:05:08.178615+0000","last_became_active":"2024-03-24T12:03:19.822426+0000","last_became_peered":"2024-03-24T12:03:19.822426+0000","last_unstale":"2024-03-24T12:05:08.178615+0000","last_undegraded":"2024-03-24T12:05:08.178615+0000","last_fullsized":"2024-03-24T12:05:08.178615+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:08.153554+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:08.153554+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:36.332030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"296'11536","reported_seq":11778,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.916546+0000","last_change":"2024-03-24T12:05:01.328489+0000","last_active":"2024-03-24T12:05:12.916546+0000","last_peered":"2024-03-24T12:05:12.916546+0000","last_clean":"2024-03-24T12:05:12.916546+0000","last_became_active":"2024-03-24T12:02:46.085796+0000","last_became_peered":"2024-03-24T12:02:46.085796+0000","last_unstale":"2024-03-24T12:05:12.916546+0000","last_undegraded":"2024-03-24T12:05:12.916546+0000","last_fullsized":"2024-03-24T12:05:12.916546+0000","mapping_epoch":246,"log_start":"275'6200","ondisk_log_start":"275'6200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'10673","last_scrub_stamp":"2024-03-24T12:05:01.328295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.328295+0000","objects_scrubbed":5181,"log_size":5336,"log_dups_size":0,"ondisk_log_size":5336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:02.812687+0000","scrub_duration":1847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278265856,"num_objects":4246,"num_object_clones":0,"num_object_copies":12738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11536,"num_write_kb":505024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":687,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.352622+0000","last_change":"2024-03-24T12:05:10.185964+0000","last_active":"2024-03-24T12:05:10.352622+0000","last_peered":"2024-03-24T12:05:10.352622+0000","last_clean":"2024-03-24T12:05:10.352622+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:05:10.352622+0000","last_undegraded":"2024-03-24T12:05:10.352622+0000","last_fullsized":"2024-03-24T12:05:10.352622+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:05:10.185824+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T11:59:08.941456+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.185824+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:19.277117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":551,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182054+0000","last_change":"2024-03-24T12:03:08.065053+0000","last_active":"2024-03-24T12:05:08.182054+0000","last_peered":"2024-03-24T12:05:08.182054+0000","last_clean":"2024-03-24T12:05:08.182054+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T12:05:08.182054+0000","last_undegraded":"2024-03-24T12:05:08.182054+0000","last_fullsized":"2024-03-24T12:05:08.182054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:03:08.064862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:08.064862+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:11.506057+0000","last_change":"2024-03-24T12:05:11.506093+0000","last_active":"2024-03-24T12:05:11.506057+0000","last_peered":"2024-03-24T12:05:11.506057+0000","last_clean":"2024-03-24T12:05:11.506057+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T12:05:11.506057+0000","last_undegraded":"2024-03-24T12:05:11.506057+0000","last_fullsized":"2024-03-24T12:05:11.506057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:11.505956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.505956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:12.205758+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":500,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182748+0000","last_change":"2024-03-24T12:03:58.095718+0000","last_active":"2024-03-24T12:05:08.182748+0000","last_peered":"2024-03-24T12:05:08.182748+0000","last_clean":"2024-03-24T12:05:08.182748+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T12:05:08.182748+0000","last_undegraded":"2024-03-24T12:05:08.182748+0000","last_fullsized":"2024-03-24T12:05:08.182748+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.095464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.095464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'130","reported_seq":383,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186592+0000","last_change":"2024-03-24T12:03:19.942347+0000","last_active":"2024-03-24T12:05:08.186592+0000","last_peered":"2024-03-24T12:05:08.186592+0000","last_clean":"2024-03-24T12:05:08.186592+0000","last_became_active":"2024-03-24T12:03:19.822938+0000","last_became_peered":"2024-03-24T12:03:19.822938+0000","last_unstale":"2024-03-24T12:05:08.186592+0000","last_undegraded":"2024-03-24T12:05:08.186592+0000","last_fullsized":"2024-03-24T12:05:08.186592+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:19.942241+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.942241+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"296'11987","reported_seq":12150,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.659803+0000","last_change":"2024-03-24T12:05:13.460893+0000","last_active":"2024-03-24T12:05:16.659803+0000","last_peered":"2024-03-24T12:05:16.659803+0000","last_clean":"2024-03-24T12:05:16.659803+0000","last_became_active":"2024-03-24T12:02:46.437833+0000","last_became_peered":"2024-03-24T12:02:46.437833+0000","last_unstale":"2024-03-24T12:05:16.659803+0000","last_undegraded":"2024-03-24T12:05:16.659803+0000","last_fullsized":"2024-03-24T12:05:16.659803+0000","mapping_epoch":246,"log_start":"281'7500","ondisk_log_start":"281'7500","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'11770","last_scrub_stamp":"2024-03-24T12:05:13.460650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.460650+0000","objects_scrubbed":4284,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:13.546677+0000","scrub_duration":1179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264699904,"num_objects":4039,"num_object_clones":0,"num_object_copies":12117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11987,"num_write_kb":512832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":534,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.174407+0000","last_change":"2024-03-24T12:03:49.868525+0000","last_active":"2024-03-24T12:05:08.174407+0000","last_peered":"2024-03-24T12:05:08.174407+0000","last_clean":"2024-03-24T12:05:08.174407+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:05:08.174407+0000","last_undegraded":"2024-03-24T12:05:08.174407+0000","last_fullsized":"2024-03-24T12:05:08.174407+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:03:49.868385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:49.868385+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":529,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.191352+0000","last_change":"2024-03-24T12:04:09.866901+0000","last_active":"2024-03-24T12:05:08.191352+0000","last_peered":"2024-03-24T12:05:08.191352+0000","last_clean":"2024-03-24T12:05:08.191352+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:05:08.191352+0000","last_undegraded":"2024-03-24T12:05:08.191352+0000","last_fullsized":"2024-03-24T12:05:08.191352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:09.866736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T12:04:09.866736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:27.550845+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":522,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179802+0000","last_change":"2024-03-24T12:04:52.456633+0000","last_active":"2024-03-24T12:05:08.179802+0000","last_peered":"2024-03-24T12:05:08.179802+0000","last_clean":"2024-03-24T12:05:08.179802+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T12:05:08.179802+0000","last_undegraded":"2024-03-24T12:05:08.179802+0000","last_fullsized":"2024-03-24T12:05:08.179802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:52.456151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:52.456151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:09.892431+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":493,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.173853+0000","last_change":"2024-03-24T12:03:17.844077+0000","last_active":"2024-03-24T12:05:08.173853+0000","last_peered":"2024-03-24T12:05:08.173853+0000","last_clean":"2024-03-24T12:05:08.173853+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T12:05:08.173853+0000","last_undegraded":"2024-03-24T12:05:08.173853+0000","last_fullsized":"2024-03-24T12:05:08.173853+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:17.843838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:17.843838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'130","reported_seq":577,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190168+0000","last_change":"2024-03-24T12:05:02.370703+0000","last_active":"2024-03-24T12:05:08.190168+0000","last_peered":"2024-03-24T12:05:08.190168+0000","last_clean":"2024-03-24T12:05:08.190168+0000","last_became_active":"2024-03-24T12:03:19.825134+0000","last_became_peered":"2024-03-24T12:03:19.825134+0000","last_unstale":"2024-03-24T12:05:08.190168+0000","last_undegraded":"2024-03-24T12:05:08.190168+0000","last_fullsized":"2024-03-24T12:05:08.190168+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:02.370579+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.370579+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.301086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"296'11913","reported_seq":12087,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.598966+0000","last_change":"2024-03-24T12:04:58.062065+0000","last_active":"2024-03-24T12:05:16.598966+0000","last_peered":"2024-03-24T12:05:16.598966+0000","last_clean":"2024-03-24T12:05:16.598966+0000","last_became_active":"2024-03-24T12:02:46.088943+0000","last_became_peered":"2024-03-24T12:02:46.088943+0000","last_unstale":"2024-03-24T12:05:16.598966+0000","last_undegraded":"2024-03-24T12:05:16.598966+0000","last_fullsized":"2024-03-24T12:05:16.598966+0000","mapping_epoch":246,"log_start":"280'7400","ondisk_log_start":"280'7400","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'10426","last_scrub_stamp":"2024-03-24T12:04:58.061867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:58.061867+0000","objects_scrubbed":5636,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:15.860491+0000","scrub_duration":1823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267190272,"num_objects":4077,"num_object_clones":0,"num_object_copies":12231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11913,"num_write_kb":511680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"296'44","reported_seq":587,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:14.441187+0000","last_change":"2024-03-24T12:05:14.441279+0000","last_active":"2024-03-24T12:05:14.441187+0000","last_peered":"2024-03-24T12:05:14.441187+0000","last_clean":"2024-03-24T12:05:14.441187+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:05:14.441187+0000","last_undegraded":"2024-03-24T12:05:14.441187+0000","last_fullsized":"2024-03-24T12:05:14.441187+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.441098+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:31.476169+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180551+0000","last_change":"2024-03-24T12:05:00.489674+0000","last_active":"2024-03-24T12:05:08.180551+0000","last_peered":"2024-03-24T12:05:08.180551+0000","last_clean":"2024-03-24T12:05:08.180551+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T12:05:08.180551+0000","last_undegraded":"2024-03-24T12:05:08.180551+0000","last_fullsized":"2024-03-24T12:05:08.180551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:00.489543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:05:00.489543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:14.786599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":522,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181163+0000","last_change":"2024-03-24T12:04:48.399451+0000","last_active":"2024-03-24T12:05:08.181163+0000","last_peered":"2024-03-24T12:05:08.181163+0000","last_clean":"2024-03-24T12:05:08.181163+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T12:05:08.181163+0000","last_undegraded":"2024-03-24T12:05:08.181163+0000","last_fullsized":"2024-03-24T12:05:08.181163+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:48.399346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:48.399346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:48.716347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":517,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:14.128779+0000","last_change":"2024-03-24T12:05:14.128826+0000","last_active":"2024-03-24T12:05:14.128779+0000","last_peered":"2024-03-24T12:05:14.128779+0000","last_clean":"2024-03-24T12:05:14.128779+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T12:05:14.128779+0000","last_undegraded":"2024-03-24T12:05:14.128779+0000","last_fullsized":"2024-03-24T12:05:14.128779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:14.128698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.128698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.394262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"296'11711","reported_seq":11899,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.332717+0000","last_change":"2024-03-24T12:04:09.253965+0000","last_active":"2024-03-24T12:05:16.332717+0000","last_peered":"2024-03-24T12:05:16.332717+0000","last_clean":"2024-03-24T12:05:16.332717+0000","last_became_active":"2024-03-24T12:02:46.436511+0000","last_became_peered":"2024-03-24T12:02:46.436511+0000","last_unstale":"2024-03-24T12:05:16.332717+0000","last_undegraded":"2024-03-24T12:05:16.332717+0000","last_fullsized":"2024-03-24T12:05:16.332717+0000","mapping_epoch":246,"log_start":"275'6300","ondisk_log_start":"275'6300","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'6898","last_scrub_stamp":"2024-03-24T12:04:09.253706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:09.253706+0000","objects_scrubbed":6832,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:19.559939+0000","scrub_duration":1893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259719168,"num_objects":3963,"num_object_clones":0,"num_object_copies":11889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":501568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":573,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.187676+0000","last_change":"2024-03-24T12:03:52.852751+0000","last_active":"2024-03-24T12:05:08.187676+0000","last_peered":"2024-03-24T12:05:08.187676+0000","last_clean":"2024-03-24T12:05:08.187676+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T12:05:08.187676+0000","last_undegraded":"2024-03-24T12:05:08.187676+0000","last_fullsized":"2024-03-24T12:05:08.187676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:03:52.852641+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.852641+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179807+0000","last_change":"2024-03-24T12:05:05.492513+0000","last_active":"2024-03-24T12:05:08.179807+0000","last_peered":"2024-03-24T12:05:08.179807+0000","last_clean":"2024-03-24T12:05:08.179807+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T12:05:08.179807+0000","last_undegraded":"2024-03-24T12:05:08.179807+0000","last_fullsized":"2024-03-24T12:05:08.179807+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:05:05.492318+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.492318+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:27.030210+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186377+0000","last_change":"2024-03-24T12:03:56.825579+0000","last_active":"2024-03-24T12:05:08.186377+0000","last_peered":"2024-03-24T12:05:08.186377+0000","last_clean":"2024-03-24T12:05:08.186377+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:05:08.186377+0000","last_undegraded":"2024-03-24T12:05:08.186377+0000","last_fullsized":"2024-03-24T12:05:08.186377+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.825474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.825474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:13.727177+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.727086+0000","last_change":"2024-03-24T12:05:12.727225+0000","last_active":"2024-03-24T12:05:12.727086+0000","last_peered":"2024-03-24T12:05:12.727086+0000","last_clean":"2024-03-24T12:05:12.727086+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T12:05:12.727086+0000","last_undegraded":"2024-03-24T12:05:12.727086+0000","last_fullsized":"2024-03-24T12:05:12.727086+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:12.726872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:12.726872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:37.941465+0000","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":543,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182348+0000","last_change":"2024-03-24T12:03:15.017062+0000","last_active":"2024-03-24T12:05:08.182348+0000","last_peered":"2024-03-24T12:05:08.182348+0000","last_clean":"2024-03-24T12:05:08.182348+0000","last_became_active":"2024-03-24T11:59:53.779445+0000","last_became_peered":"2024-03-24T11:59:53.779445+0000","last_unstale":"2024-03-24T12:05:08.182348+0000","last_undegraded":"2024-03-24T12:05:08.182348+0000","last_fullsized":"2024-03-24T12:05:08.182348+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:15.016829+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:15.016829+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"296'11290","reported_seq":11401,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.452059+0000","last_change":"2024-03-24T12:02:46.091030+0000","last_active":"2024-03-24T12:05:12.452059+0000","last_peered":"2024-03-24T12:05:12.452059+0000","last_clean":"2024-03-24T12:05:12.452059+0000","last_became_active":"2024-03-24T12:02:46.090706+0000","last_became_peered":"2024-03-24T12:02:46.090706+0000","last_unstale":"2024-03-24T12:05:12.452059+0000","last_undegraded":"2024-03-24T12:05:12.452059+0000","last_fullsized":"2024-03-24T12:05:12.452059+0000","mapping_epoch":246,"log_start":"279'7100","ondisk_log_start":"279'7100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274595840,"num_objects":4190,"num_object_clones":0,"num_object_copies":12570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11290,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":551,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185081+0000","last_change":"2024-03-24T12:05:01.942196+0000","last_active":"2024-03-24T12:05:08.185081+0000","last_peered":"2024-03-24T12:05:08.185081+0000","last_clean":"2024-03-24T12:05:08.185081+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:05:08.185081+0000","last_undegraded":"2024-03-24T12:05:08.185081+0000","last_fullsized":"2024-03-24T12:05:08.185081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:01.942039+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.942039+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:24.554745+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181937+0000","last_change":"2024-03-24T12:04:00.667345+0000","last_active":"2024-03-24T12:05:08.181937+0000","last_peered":"2024-03-24T12:05:08.181937+0000","last_clean":"2024-03-24T12:05:08.181937+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:05:08.181937+0000","last_undegraded":"2024-03-24T12:05:08.181937+0000","last_fullsized":"2024-03-24T12:05:08.181937+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:04:00.667221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:09.226148+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":687,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.352468+0000","last_change":"2024-03-24T12:05:06.479237+0000","last_active":"2024-03-24T12:05:10.352468+0000","last_peered":"2024-03-24T12:05:10.352468+0000","last_clean":"2024-03-24T12:05:10.352468+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T12:05:10.352468+0000","last_undegraded":"2024-03-24T12:05:10.352468+0000","last_fullsized":"2024-03-24T12:05:10.352468+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:06.478884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.478884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:34.542656+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":509,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186985+0000","last_change":"2024-03-24T12:03:47.910225+0000","last_active":"2024-03-24T12:05:08.186985+0000","last_peered":"2024-03-24T12:05:08.186985+0000","last_clean":"2024-03-24T12:05:08.186985+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T12:05:08.186985+0000","last_undegraded":"2024-03-24T12:05:08.186985+0000","last_fullsized":"2024-03-24T12:05:08.186985+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:03:47.909970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:47.909970+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":804,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.142691+0000","last_change":"2024-03-24T12:05:16.142814+0000","last_active":"2024-03-24T12:05:16.142691+0000","last_peered":"2024-03-24T12:05:16.142691+0000","last_clean":"2024-03-24T12:05:16.142691+0000","last_became_active":"2024-03-24T11:59:53.771452+0000","last_became_peered":"2024-03-24T11:59:53.771452+0000","last_unstale":"2024-03-24T12:05:16.142691+0000","last_undegraded":"2024-03-24T12:05:16.142691+0000","last_fullsized":"2024-03-24T12:05:16.142691+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.142552+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:17.240353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"296'11442","reported_seq":11553,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.415977+0000","last_change":"2024-03-24T12:02:46.437086+0000","last_active":"2024-03-24T12:05:12.415977+0000","last_peered":"2024-03-24T12:05:12.415977+0000","last_clean":"2024-03-24T12:05:12.415977+0000","last_became_active":"2024-03-24T12:02:46.436446+0000","last_became_peered":"2024-03-24T12:02:46.436446+0000","last_unstale":"2024-03-24T12:05:12.415977+0000","last_undegraded":"2024-03-24T12:05:12.415977+0000","last_fullsized":"2024-03-24T12:05:12.415977+0000","mapping_epoch":246,"log_start":"279'7200","ondisk_log_start":"279'7200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277479424,"num_objects":4234,"num_object_clones":0,"num_object_copies":12702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":567,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.194143+0000","last_change":"2024-03-24T12:03:48.579115+0000","last_active":"2024-03-24T12:05:08.194143+0000","last_peered":"2024-03-24T12:05:08.194143+0000","last_clean":"2024-03-24T12:05:08.194143+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:05:08.194143+0000","last_undegraded":"2024-03-24T12:05:08.194143+0000","last_fullsized":"2024-03-24T12:05:08.194143+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:03:48.578981+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.578981+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":529,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189790+0000","last_change":"2024-03-24T12:05:04.036530+0000","last_active":"2024-03-24T12:05:08.189790+0000","last_peered":"2024-03-24T12:05:08.189790+0000","last_clean":"2024-03-24T12:05:08.189790+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:05:08.189790+0000","last_undegraded":"2024-03-24T12:05:08.189790+0000","last_fullsized":"2024-03-24T12:05:08.189790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:04.036311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:05:04.036311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:14.102415+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":680,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.352400+0000","last_change":"2024-03-24T12:05:05.860928+0000","last_active":"2024-03-24T12:05:10.352400+0000","last_peered":"2024-03-24T12:05:10.352400+0000","last_clean":"2024-03-24T12:05:10.352400+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T12:05:10.352400+0000","last_undegraded":"2024-03-24T12:05:10.352400+0000","last_fullsized":"2024-03-24T12:05:10.352400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:05.860814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.860814+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:12.200839+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189204+0000","last_change":"2024-03-24T12:04:02.219664+0000","last_active":"2024-03-24T12:05:08.189204+0000","last_peered":"2024-03-24T12:05:08.189204+0000","last_clean":"2024-03-24T12:05:08.189204+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T12:05:08.189204+0000","last_undegraded":"2024-03-24T12:05:08.189204+0000","last_fullsized":"2024-03-24T12:05:08.189204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.219541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.219541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:11.682190+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":551,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:11.242652+0000","last_change":"2024-03-24T12:05:11.242695+0000","last_active":"2024-03-24T12:05:11.242652+0000","last_peered":"2024-03-24T12:05:11.242652+0000","last_clean":"2024-03-24T12:05:11.242652+0000","last_became_active":"2024-03-24T11:59:53.771384+0000","last_became_peered":"2024-03-24T11:59:53.771384+0000","last_unstale":"2024-03-24T12:05:11.242652+0000","last_undegraded":"2024-03-24T12:05:11.242652+0000","last_fullsized":"2024-03-24T12:05:11.242652+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:11.242568+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.242568+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.583012+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'130","reported_seq":385,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.173979+0000","last_change":"2024-03-24T12:03:20.776671+0000","last_active":"2024-03-24T12:05:08.173979+0000","last_peered":"2024-03-24T12:05:08.173979+0000","last_clean":"2024-03-24T12:05:08.173979+0000","last_became_active":"2024-03-24T12:03:19.826030+0000","last_became_peered":"2024-03-24T12:03:19.826030+0000","last_unstale":"2024-03-24T12:05:08.173979+0000","last_undegraded":"2024-03-24T12:05:08.173979+0000","last_fullsized":"2024-03-24T12:05:08.173979+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:20.776532+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:20.776532+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3","version":"296'11602","reported_seq":11790,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.483448+0000","last_change":"2024-03-24T12:04:10.081523+0000","last_active":"2024-03-24T12:05:16.483448+0000","last_peered":"2024-03-24T12:05:16.483448+0000","last_clean":"2024-03-24T12:05:16.483448+0000","last_became_active":"2024-03-24T12:02:46.436507+0000","last_became_peered":"2024-03-24T12:02:46.436507+0000","last_unstale":"2024-03-24T12:05:16.483448+0000","last_undegraded":"2024-03-24T12:05:16.483448+0000","last_fullsized":"2024-03-24T12:05:16.483448+0000","mapping_epoch":246,"log_start":"282'7400","ondisk_log_start":"282'7400","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'6826","last_scrub_stamp":"2024-03-24T12:04:10.081256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:10.081256+0000","objects_scrubbed":6794,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254148608,"num_objects":3878,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":544,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186702+0000","last_change":"2024-03-24T12:03:09.967801+0000","last_active":"2024-03-24T12:05:08.186702+0000","last_peered":"2024-03-24T12:05:08.186702+0000","last_clean":"2024-03-24T12:05:08.186702+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:05:08.186702+0000","last_undegraded":"2024-03-24T12:05:08.186702+0000","last_fullsized":"2024-03-24T12:05:08.186702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:03:09.967712+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:03:09.967712+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178918+0000","last_change":"2024-03-24T12:04:54.129547+0000","last_active":"2024-03-24T12:05:08.178918+0000","last_peered":"2024-03-24T12:05:08.178918+0000","last_clean":"2024-03-24T12:05:08.178918+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:05:08.178918+0000","last_undegraded":"2024-03-24T12:05:08.178918+0000","last_fullsized":"2024-03-24T12:05:08.178918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.129436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.129436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.954691+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":504,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184181+0000","last_change":"2024-03-24T12:03:12.111480+0000","last_active":"2024-03-24T12:05:08.184181+0000","last_peered":"2024-03-24T12:05:08.184181+0000","last_clean":"2024-03-24T12:05:08.184181+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T12:05:08.184181+0000","last_undegraded":"2024-03-24T12:05:08.184181+0000","last_fullsized":"2024-03-24T12:05:08.184181+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:12.111323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:12.111323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188936+0000","last_change":"2024-03-24T12:04:48.633752+0000","last_active":"2024-03-24T12:05:08.188936+0000","last_peered":"2024-03-24T12:05:08.188936+0000","last_clean":"2024-03-24T12:05:08.188936+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T12:05:08.188936+0000","last_undegraded":"2024-03-24T12:05:08.188936+0000","last_fullsized":"2024-03-24T12:05:08.188936+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:48.633639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:48.633639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:00.099164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"296'11507","reported_seq":11673,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.506691+0000","last_change":"2024-03-24T12:05:07.843040+0000","last_active":"2024-03-24T12:05:12.506691+0000","last_peered":"2024-03-24T12:05:12.506691+0000","last_clean":"2024-03-24T12:05:12.506691+0000","last_became_active":"2024-03-24T12:02:46.087133+0000","last_became_peered":"2024-03-24T12:02:46.087133+0000","last_unstale":"2024-03-24T12:05:12.506691+0000","last_undegraded":"2024-03-24T12:05:12.506691+0000","last_fullsized":"2024-03-24T12:05:12.506691+0000","mapping_epoch":246,"log_start":"280'7300","ondisk_log_start":"280'7300","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'11157","last_scrub_stamp":"2024-03-24T12:05:07.842597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:07.842597+0000","objects_scrubbed":4609,"log_size":4207,"log_dups_size":0,"ondisk_log_size":4207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:36.787715+0000","scrub_duration":2120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273874944,"num_objects":4179,"num_object_clones":0,"num_object_copies":12537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":501952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":671,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178494+0000","last_change":"2024-03-24T12:04:13.607127+0000","last_active":"2024-03-24T12:05:08.178494+0000","last_peered":"2024-03-24T12:05:08.178494+0000","last_clean":"2024-03-24T12:05:08.178494+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:05:08.178494+0000","last_undegraded":"2024-03-24T12:05:08.178494+0000","last_fullsized":"2024-03-24T12:05:08.178494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:04:13.606837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:13.606837+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:33.283948+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":558,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183499+0000","last_change":"2024-03-24T12:02:54.860022+0000","last_active":"2024-03-24T12:05:08.183499+0000","last_peered":"2024-03-24T12:05:08.183499+0000","last_clean":"2024-03-24T12:05:08.183499+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T12:05:08.183499+0000","last_undegraded":"2024-03-24T12:05:08.183499+0000","last_fullsized":"2024-03-24T12:05:08.183499+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:02:54.859867+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:02:54.859867+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":528,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190480+0000","last_change":"2024-03-24T12:05:04.423091+0000","last_active":"2024-03-24T12:05:08.190480+0000","last_peered":"2024-03-24T12:05:08.190480+0000","last_clean":"2024-03-24T12:05:08.190480+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:05:08.190480+0000","last_undegraded":"2024-03-24T12:05:08.190480+0000","last_fullsized":"2024-03-24T12:05:08.190480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:04.422989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:05:04.422989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.438929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180240+0000","last_change":"2024-03-24T12:04:59.065642+0000","last_active":"2024-03-24T12:05:08.180240+0000","last_peered":"2024-03-24T12:05:08.180240+0000","last_clean":"2024-03-24T12:05:08.180240+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T12:05:08.180240+0000","last_undegraded":"2024-03-24T12:05:08.180240+0000","last_fullsized":"2024-03-24T12:05:08.180240+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.065539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.065539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:07.880065+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":548,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190159+0000","last_change":"2024-03-24T12:03:15.627195+0000","last_active":"2024-03-24T12:05:08.190159+0000","last_peered":"2024-03-24T12:05:08.190159+0000","last_clean":"2024-03-24T12:05:08.190159+0000","last_became_active":"2024-03-24T12:03:15.302146+0000","last_became_peered":"2024-03-24T12:03:15.302146+0000","last_unstale":"2024-03-24T12:05:08.190159+0000","last_undegraded":"2024-03-24T12:05:08.190159+0000","last_fullsized":"2024-03-24T12:05:08.190159+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:15.627055+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:15.627055+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"296'11770","reported_seq":11960,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.418159+0000","last_change":"2024-03-24T12:04:12.206158+0000","last_active":"2024-03-24T12:05:16.418159+0000","last_peered":"2024-03-24T12:05:16.418159+0000","last_clean":"2024-03-24T12:05:16.418159+0000","last_became_active":"2024-03-24T12:02:46.436562+0000","last_became_peered":"2024-03-24T12:02:46.436562+0000","last_unstale":"2024-03-24T12:05:16.418159+0000","last_undegraded":"2024-03-24T12:05:16.418159+0000","last_fullsized":"2024-03-24T12:05:16.418159+0000","mapping_epoch":246,"log_start":"282'7600","ondisk_log_start":"282'7600","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7141","last_scrub_stamp":"2024-03-24T12:04:12.205805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.205805+0000","objects_scrubbed":7081,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:30.960946+0000","scrub_duration":2066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265093144,"num_objects":4046,"num_object_clones":0,"num_object_copies":12138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11770,"num_write_kb":506049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":505,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183708+0000","last_change":"2024-03-24T12:03:12.939439+0000","last_active":"2024-03-24T12:05:08.183708+0000","last_peered":"2024-03-24T12:05:08.183708+0000","last_clean":"2024-03-24T12:05:08.183708+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:05:08.183708+0000","last_undegraded":"2024-03-24T12:05:08.183708+0000","last_fullsized":"2024-03-24T12:05:08.183708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:12.939336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:03:12.939336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":569,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178331+0000","last_change":"2024-03-24T12:03:52.775624+0000","last_active":"2024-03-24T12:05:08.178331+0000","last_peered":"2024-03-24T12:05:08.178331+0000","last_clean":"2024-03-24T12:05:08.178331+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:05:08.178331+0000","last_undegraded":"2024-03-24T12:05:08.178331+0000","last_fullsized":"2024-03-24T12:05:08.178331+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:03:52.775356+0000","last_deep_scrub":"195'26","last_deep_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.775356+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":272,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189606+0000","last_change":"2024-03-24T12:04:03.362973+0000","last_active":"2024-03-24T12:05:08.189606+0000","last_peered":"2024-03-24T12:05:08.189606+0000","last_clean":"2024-03-24T12:05:08.189606+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:05:08.189606+0000","last_undegraded":"2024-03-24T12:05:08.189606+0000","last_fullsized":"2024-03-24T12:05:08.189606+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:04:03.362834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:03.362834+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":509,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185438+0000","last_change":"2024-03-24T12:03:14.944540+0000","last_active":"2024-03-24T12:05:08.185438+0000","last_peered":"2024-03-24T12:05:08.185438+0000","last_clean":"2024-03-24T12:05:08.185438+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T12:05:08.185438+0000","last_undegraded":"2024-03-24T12:05:08.185438+0000","last_fullsized":"2024-03-24T12:05:08.185438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:14.944367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:14.944367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":548,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.187038+0000","last_change":"2024-03-24T12:03:16.889700+0000","last_active":"2024-03-24T12:05:08.187038+0000","last_peered":"2024-03-24T12:05:08.187038+0000","last_clean":"2024-03-24T12:05:08.187038+0000","last_became_active":"2024-03-24T12:03:15.292363+0000","last_became_peered":"2024-03-24T12:03:15.292363+0000","last_unstale":"2024-03-24T12:05:08.187038+0000","last_undegraded":"2024-03-24T12:05:08.187038+0000","last_fullsized":"2024-03-24T12:05:08.187038+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:16.889509+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:16.889509+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"296'11860","reported_seq":12050,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.486163+0000","last_change":"2024-03-24T12:04:11.225904+0000","last_active":"2024-03-24T12:05:16.486163+0000","last_peered":"2024-03-24T12:05:16.486163+0000","last_clean":"2024-03-24T12:05:16.486163+0000","last_became_active":"2024-03-24T12:02:46.438121+0000","last_became_peered":"2024-03-24T12:02:46.438121+0000","last_unstale":"2024-03-24T12:05:16.486163+0000","last_undegraded":"2024-03-24T12:05:16.486163+0000","last_fullsized":"2024-03-24T12:05:16.486163+0000","mapping_epoch":246,"log_start":"282'7700","ondisk_log_start":"282'7700","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'7114","last_scrub_stamp":"2024-03-24T12:04:11.225516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:11.225516+0000","objects_scrubbed":7067,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259915776,"num_objects":3966,"num_object_clones":0,"num_object_copies":11898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11860,"num_write_kb":506432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":512,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186265+0000","last_change":"2024-03-24T12:03:49.287663+0000","last_active":"2024-03-24T12:05:08.186265+0000","last_peered":"2024-03-24T12:05:08.186265+0000","last_clean":"2024-03-24T12:05:08.186265+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T12:05:08.186265+0000","last_undegraded":"2024-03-24T12:05:08.186265+0000","last_fullsized":"2024-03-24T12:05:08.186265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:49.287554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:03:49.287554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"241'29","reported_seq":561,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:09.620392+0000","last_change":"2024-03-24T12:05:09.620452+0000","last_active":"2024-03-24T12:05:09.620392+0000","last_peered":"2024-03-24T12:05:09.620392+0000","last_clean":"2024-03-24T12:05:09.620392+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T12:05:09.620392+0000","last_undegraded":"2024-03-24T12:05:09.620392+0000","last_fullsized":"2024-03-24T12:05:09.620392+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-24T12:05:09.620298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.620298+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:27.220062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179954+0000","last_change":"2024-03-24T12:03:58.207492+0000","last_active":"2024-03-24T12:05:08.179954+0000","last_peered":"2024-03-24T12:05:08.179954+0000","last_clean":"2024-03-24T12:05:08.179954+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:05:08.179954+0000","last_undegraded":"2024-03-24T12:05:08.179954+0000","last_fullsized":"2024-03-24T12:05:08.179954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.207386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.207386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:25.672325+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188403+0000","last_change":"2024-03-24T12:03:52.366880+0000","last_active":"2024-03-24T12:05:08.188403+0000","last_peered":"2024-03-24T12:05:08.188403+0000","last_clean":"2024-03-24T12:05:08.188403+0000","last_became_active":"2024-03-24T11:59:19.096322+0000","last_became_peered":"2024-03-24T11:59:19.096322+0000","last_unstale":"2024-03-24T12:05:08.188403+0000","last_undegraded":"2024-03-24T12:05:08.188403+0000","last_fullsized":"2024-03-24T12:05:08.188403+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.366551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.366551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":570,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190760+0000","last_change":"2024-03-24T12:04:56.364590+0000","last_active":"2024-03-24T12:05:08.190760+0000","last_peered":"2024-03-24T12:05:08.190760+0000","last_clean":"2024-03-24T12:05:08.190760+0000","last_became_active":"2024-03-24T12:03:15.295100+0000","last_became_peered":"2024-03-24T12:03:15.295100+0000","last_unstale":"2024-03-24T12:05:08.190760+0000","last_undegraded":"2024-03-24T12:05:08.190760+0000","last_fullsized":"2024-03-24T12:05:08.190760+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:56.364427+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:56.364427+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:03.226918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"296'15813","reported_seq":16257,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.902314+0000","last_change":"2024-03-24T12:04:53.459598+0000","last_active":"2024-03-24T12:05:12.902314+0000","last_peered":"2024-03-24T12:05:12.902314+0000","last_clean":"2024-03-24T12:05:12.902314+0000","last_became_active":"2024-03-24T12:02:46.087237+0000","last_became_peered":"2024-03-24T12:02:46.087237+0000","last_unstale":"2024-03-24T12:05:12.902314+0000","last_undegraded":"2024-03-24T12:05:12.902314+0000","last_fullsized":"2024-03-24T12:05:12.902314+0000","mapping_epoch":246,"log_start":"282'10501","ondisk_log_start":"282'10501","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14298","last_scrub_stamp":"2024-03-24T12:04:53.458513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:53.458513+0000","objects_scrubbed":5863,"log_size":5312,"log_dups_size":0,"ondisk_log_size":5312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:23.033128+0000","scrub_duration":1994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279445504,"num_objects":4264,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11344,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":505,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184314+0000","last_change":"2024-03-24T12:02:58.920843+0000","last_active":"2024-03-24T12:05:08.184314+0000","last_peered":"2024-03-24T12:05:08.184314+0000","last_clean":"2024-03-24T12:05:08.184314+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T12:05:08.184314+0000","last_undegraded":"2024-03-24T12:05:08.184314+0000","last_fullsized":"2024-03-24T12:05:08.184314+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:58.920712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:58.920712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":551,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172397+0000","last_change":"2024-03-24T12:03:51.871366+0000","last_active":"2024-03-24T12:05:08.172397+0000","last_peered":"2024-03-24T12:05:08.172397+0000","last_clean":"2024-03-24T12:05:08.172397+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T12:05:08.172397+0000","last_undegraded":"2024-03-24T12:05:08.172397+0000","last_fullsized":"2024-03-24T12:05:08.172397+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:03:51.871217+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.871217+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:19.921684+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":358,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.174950+0000","last_change":"2024-03-24T12:05:04.552941+0000","last_active":"2024-03-24T12:05:08.174950+0000","last_peered":"2024-03-24T12:05:08.174950+0000","last_clean":"2024-03-24T12:05:08.174950+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:05:08.174950+0000","last_undegraded":"2024-03-24T12:05:08.174950+0000","last_fullsized":"2024-03-24T12:05:08.174950+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:04.552647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:04.552647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:23.348306+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.173520+0000","last_change":"2024-03-24T12:04:02.023510+0000","last_active":"2024-03-24T12:05:08.173520+0000","last_peered":"2024-03-24T12:05:08.173520+0000","last_clean":"2024-03-24T12:05:08.173520+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:05:08.173520+0000","last_undegraded":"2024-03-24T12:05:08.173520+0000","last_fullsized":"2024-03-24T12:05:08.173520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.023298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.023298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190075+0000","last_change":"2024-03-24T12:05:03.017119+0000","last_active":"2024-03-24T12:05:08.190075+0000","last_peered":"2024-03-24T12:05:08.190075+0000","last_clean":"2024-03-24T12:05:08.190075+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T12:05:08.190075+0000","last_undegraded":"2024-03-24T12:05:08.190075+0000","last_fullsized":"2024-03-24T12:05:08.190075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.016987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.016987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:31.930987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":570,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185703+0000","last_change":"2024-03-24T12:04:59.345528+0000","last_active":"2024-03-24T12:05:08.185703+0000","last_peered":"2024-03-24T12:05:08.185703+0000","last_clean":"2024-03-24T12:05:08.185703+0000","last_became_active":"2024-03-24T12:03:15.297422+0000","last_became_peered":"2024-03-24T12:03:15.297422+0000","last_unstale":"2024-03-24T12:05:08.185703+0000","last_undegraded":"2024-03-24T12:05:08.185703+0000","last_fullsized":"2024-03-24T12:05:08.185703+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:59.345388+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.345388+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:06.312683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"296'11449","reported_seq":11560,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.466748+0000","last_change":"2024-03-24T12:02:46.086845+0000","last_active":"2024-03-24T12:05:12.466748+0000","last_peered":"2024-03-24T12:05:12.466748+0000","last_clean":"2024-03-24T12:05:12.466748+0000","last_became_active":"2024-03-24T12:02:46.086523+0000","last_became_peered":"2024-03-24T12:02:46.086523+0000","last_unstale":"2024-03-24T12:05:12.466748+0000","last_undegraded":"2024-03-24T12:05:12.466748+0000","last_fullsized":"2024-03-24T12:05:12.466748+0000","mapping_epoch":246,"log_start":"278'7200","ondisk_log_start":"278'7200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282656768,"num_objects":4313,"num_object_clones":0,"num_object_copies":12939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11449,"num_write_kb":504384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":522,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180298+0000","last_change":"2024-03-24T12:05:06.101252+0000","last_active":"2024-03-24T12:05:08.180298+0000","last_peered":"2024-03-24T12:05:08.180298+0000","last_clean":"2024-03-24T12:05:08.180298+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T12:05:08.180298+0000","last_undegraded":"2024-03-24T12:05:08.180298+0000","last_fullsized":"2024-03-24T12:05:08.180298+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:06.100942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.100942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:34.955762+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":729,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.352370+0000","last_change":"2024-03-24T12:04:47.544969+0000","last_active":"2024-03-24T12:05:10.352370+0000","last_peered":"2024-03-24T12:05:10.352370+0000","last_clean":"2024-03-24T12:05:10.352370+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:05:10.352370+0000","last_undegraded":"2024-03-24T12:05:10.352370+0000","last_fullsized":"2024-03-24T12:05:10.352370+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:04:47.544810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:47.544810+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:01.237310+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":802,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178778+0000","last_change":"2024-03-24T12:04:58.016031+0000","last_active":"2024-03-24T12:05:08.178778+0000","last_peered":"2024-03-24T12:05:08.178778+0000","last_clean":"2024-03-24T12:05:08.178778+0000","last_became_active":"2024-03-24T12:03:15.298475+0000","last_became_peered":"2024-03-24T12:03:15.298475+0000","last_unstale":"2024-03-24T12:05:08.178778+0000","last_undegraded":"2024-03-24T12:05:08.178778+0000","last_fullsized":"2024-03-24T12:05:08.178778+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:58.015904+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:03:16.845832+0000","last_clean_scrub_stamp":"2024-03-24T12:04:58.015904+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.816330+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183757+0000","last_change":"2024-03-24T12:03:13.102194+0000","last_active":"2024-03-24T12:05:08.183757+0000","last_peered":"2024-03-24T12:05:08.183757+0000","last_clean":"2024-03-24T12:05:08.183757+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T12:05:08.183757+0000","last_undegraded":"2024-03-24T12:05:08.183757+0000","last_fullsized":"2024-03-24T12:05:08.183757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:13.102066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:13.102066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":524,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181838+0000","last_change":"2024-03-24T12:04:08.680127+0000","last_active":"2024-03-24T12:05:08.181838+0000","last_peered":"2024-03-24T12:05:08.181838+0000","last_clean":"2024-03-24T12:05:08.181838+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T12:05:08.181838+0000","last_undegraded":"2024-03-24T12:05:08.181838+0000","last_fullsized":"2024-03-24T12:05:08.181838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:04:08.679904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:08.679904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:14.442813+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.463323+0000","last_change":"2024-03-24T12:05:10.463358+0000","last_active":"2024-03-24T12:05:10.463323+0000","last_peered":"2024-03-24T12:05:10.463323+0000","last_clean":"2024-03-24T12:05:10.463323+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T12:05:10.463323+0000","last_undegraded":"2024-03-24T12:05:10.463323+0000","last_fullsized":"2024-03-24T12:05:10.463323+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.463242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.463242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.094315+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"296'11571","reported_seq":11743,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.655891+0000","last_change":"2024-03-24T12:05:01.808405+0000","last_active":"2024-03-24T12:05:16.655891+0000","last_peered":"2024-03-24T12:05:16.655891+0000","last_clean":"2024-03-24T12:05:16.655891+0000","last_became_active":"2024-03-24T12:02:46.437687+0000","last_became_peered":"2024-03-24T12:02:46.437687+0000","last_unstale":"2024-03-24T12:05:16.655891+0000","last_undegraded":"2024-03-24T12:05:16.655891+0000","last_fullsized":"2024-03-24T12:05:16.655891+0000","mapping_epoch":246,"log_start":"277'7000","ondisk_log_start":"277'7000","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'10522","last_scrub_stamp":"2024-03-24T12:05:01.808221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.808221+0000","objects_scrubbed":5128,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.285085+0000","scrub_duration":1543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":498944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":523,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183916+0000","last_change":"2024-03-24T12:03:52.368433+0000","last_active":"2024-03-24T12:05:08.183916+0000","last_peered":"2024-03-24T12:05:08.183916+0000","last_clean":"2024-03-24T12:05:08.183916+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T12:05:08.183916+0000","last_undegraded":"2024-03-24T12:05:08.183916+0000","last_fullsized":"2024-03-24T12:05:08.183916+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.367743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:13.600133+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.367743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":559,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.194091+0000","last_change":"2024-03-24T12:05:08.084142+0000","last_active":"2024-03-24T12:05:08.194091+0000","last_peered":"2024-03-24T12:05:08.194091+0000","last_clean":"2024-03-24T12:05:08.194091+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T12:05:08.194091+0000","last_undegraded":"2024-03-24T12:05:08.194091+0000","last_fullsized":"2024-03-24T12:05:08.194091+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:05:08.083956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:08.083956+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:34.045721+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":531,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179586+0000","last_change":"2024-03-24T12:03:50.315583+0000","last_active":"2024-03-24T12:05:08.179586+0000","last_peered":"2024-03-24T12:05:08.179586+0000","last_clean":"2024-03-24T12:05:08.179586+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T12:05:08.179586+0000","last_undegraded":"2024-03-24T12:05:08.179586+0000","last_fullsized":"2024-03-24T12:05:08.179586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:50.315462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:50.315462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":671,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:15.951525+0000","last_change":"2024-03-24T12:04:54.359217+0000","last_active":"2024-03-24T12:05:15.951525+0000","last_peered":"2024-03-24T12:05:15.951525+0000","last_clean":"2024-03-24T12:05:15.951525+0000","last_became_active":"2024-03-24T11:59:19.095838+0000","last_became_peered":"2024-03-24T11:59:19.095838+0000","last_unstale":"2024-03-24T12:05:15.951525+0000","last_undegraded":"2024-03-24T12:05:15.951525+0000","last_fullsized":"2024-03-24T12:05:15.951525+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:04:54.358848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.358848+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:16.589772+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":364,"num_read_kb":359,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:11.223467+0000","last_change":"2024-03-24T12:05:11.223509+0000","last_active":"2024-03-24T12:05:11.223467+0000","last_peered":"2024-03-24T12:05:11.223467+0000","last_clean":"2024-03-24T12:05:11.223467+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:05:11.223467+0000","last_undegraded":"2024-03-24T12:05:11.223467+0000","last_fullsized":"2024-03-24T12:05:11.223467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:11.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:28.595270+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":809,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179727+0000","last_change":"2024-03-24T12:04:55.074407+0000","last_active":"2024-03-24T12:05:08.179727+0000","last_peered":"2024-03-24T12:05:08.179727+0000","last_clean":"2024-03-24T12:05:08.179727+0000","last_became_active":"2024-03-24T12:03:15.298007+0000","last_became_peered":"2024-03-24T12:03:15.298007+0000","last_unstale":"2024-03-24T12:05:08.179727+0000","last_undegraded":"2024-03-24T12:05:08.179727+0000","last_fullsized":"2024-03-24T12:05:08.179727+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:55.074332+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:55.074332+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:23.720636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"296'14556","reported_seq":14914,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.921106+0000","last_change":"2024-03-24T12:04:56.353651+0000","last_active":"2024-03-24T12:05:12.921106+0000","last_peered":"2024-03-24T12:05:12.921106+0000","last_clean":"2024-03-24T12:05:12.921106+0000","last_became_active":"2024-03-24T12:02:46.090175+0000","last_became_peered":"2024-03-24T12:02:46.090175+0000","last_unstale":"2024-03-24T12:05:12.921106+0000","last_undegraded":"2024-03-24T12:05:12.921106+0000","last_fullsized":"2024-03-24T12:05:12.921106+0000","mapping_epoch":246,"log_start":"282'9201","ondisk_log_start":"282'9201","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'13292","last_scrub_stamp":"2024-03-24T12:04:56.353492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:56.353492+0000","objects_scrubbed":5572,"log_size":5355,"log_dups_size":0,"ondisk_log_size":5355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:00.814462+0000","scrub_duration":1967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277676032,"num_objects":4237,"num_object_clones":0,"num_object_copies":12711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":498368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":506,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.176674+0000","last_change":"2024-03-24T12:03:58.906510+0000","last_active":"2024-03-24T12:05:08.176674+0000","last_peered":"2024-03-24T12:05:08.176674+0000","last_clean":"2024-03-24T12:05:08.176674+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:05:08.176674+0000","last_undegraded":"2024-03-24T12:05:08.176674+0000","last_fullsized":"2024-03-24T12:05:08.176674+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.906274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:16.217543+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"241'41","reported_seq":684,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:11.644621+0000","last_change":"2024-03-24T12:04:07.131702+0000","last_active":"2024-03-24T12:05:11.644621+0000","last_peered":"2024-03-24T12:05:11.644621+0000","last_clean":"2024-03-24T12:05:11.644621+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T12:05:11.644621+0000","last_undegraded":"2024-03-24T12:05:11.644621+0000","last_fullsized":"2024-03-24T12:05:11.644621+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-24T12:04:07.131550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:07.131550+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182860+0000","last_change":"2024-03-24T12:04:54.196862+0000","last_active":"2024-03-24T12:05:08.182860+0000","last_peered":"2024-03-24T12:05:08.182860+0000","last_clean":"2024-03-24T12:05:08.182860+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T12:05:08.182860+0000","last_undegraded":"2024-03-24T12:05:08.182860+0000","last_fullsized":"2024-03-24T12:05:08.182860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.196749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.196749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:20.902352+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181296+0000","last_change":"2024-03-24T12:04:02.704225+0000","last_active":"2024-03-24T12:05:08.181296+0000","last_peered":"2024-03-24T12:05:08.181296+0000","last_clean":"2024-03-24T12:05:08.181296+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:05:08.181296+0000","last_undegraded":"2024-03-24T12:05:08.181296+0000","last_fullsized":"2024-03-24T12:05:08.181296+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.704086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.704086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:26.546480+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":564,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184184+0000","last_change":"2024-03-24T12:04:47.152479+0000","last_active":"2024-03-24T12:05:08.184184+0000","last_peered":"2024-03-24T12:05:08.184184+0000","last_clean":"2024-03-24T12:05:08.184184+0000","last_became_active":"2024-03-24T12:03:15.310321+0000","last_became_peered":"2024-03-24T12:03:15.310321+0000","last_unstale":"2024-03-24T12:05:08.184184+0000","last_undegraded":"2024-03-24T12:05:08.184184+0000","last_fullsized":"2024-03-24T12:05:08.184184+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:47.152371+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:47.152371+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:06.868017+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"296'13144","reported_seq":13286,"reported_epoch":296,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T12:05:16.678159+0000","last_change":"2024-03-24T12:05:09.242319+0000","last_active":"2024-03-24T12:05:16.678159+0000","last_peered":"2024-03-24T12:05:16.678159+0000","last_clean":"2024-03-24T12:05:16.678159+0000","last_became_active":"2024-03-24T12:02:46.436594+0000","last_became_peered":"2024-03-24T12:02:46.436594+0000","last_unstale":"2024-03-24T12:05:16.678159+0000","last_undegraded":"2024-03-24T12:05:16.678159+0000","last_fullsized":"2024-03-24T12:05:16.678159+0000","mapping_epoch":246,"log_start":"288'8600","ondisk_log_start":"288'8600","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":1367,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 7s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266534912,"num_objects":4067,"num_object_clones":0,"num_object_copies":12201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":507008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":678,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.175289+0000","last_change":"2024-03-24T12:05:02.500334+0000","last_active":"2024-03-24T12:05:08.175289+0000","last_peered":"2024-03-24T12:05:08.175289+0000","last_clean":"2024-03-24T12:05:08.175289+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T12:05:08.175289+0000","last_undegraded":"2024-03-24T12:05:08.175289+0000","last_fullsized":"2024-03-24T12:05:08.175289+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:02.500062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.500062+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:10.180018+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":561,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.191057+0000","last_change":"2024-03-24T12:05:05.062227+0000","last_active":"2024-03-24T12:05:08.191057+0000","last_peered":"2024-03-24T12:05:08.191057+0000","last_clean":"2024-03-24T12:05:08.191057+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T12:05:08.191057+0000","last_undegraded":"2024-03-24T12:05:08.191057+0000","last_fullsized":"2024-03-24T12:05:08.191057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:05.061844+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.061844+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.609038+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.174319+0000","last_change":"2024-03-24T12:04:03.014426+0000","last_active":"2024-03-24T12:05:08.174319+0000","last_peered":"2024-03-24T12:05:08.174319+0000","last_clean":"2024-03-24T12:05:08.174319+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:05:08.174319+0000","last_undegraded":"2024-03-24T12:05:08.174319+0000","last_fullsized":"2024-03-24T12:05:08.174319+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:03.014306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:04:03.014306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:31.228036+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":494,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.173979+0000","last_change":"2024-03-24T12:03:02.711368+0000","last_active":"2024-03-24T12:05:08.173979+0000","last_peered":"2024-03-24T12:05:08.173979+0000","last_clean":"2024-03-24T12:05:08.173979+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T12:05:08.173979+0000","last_undegraded":"2024-03-24T12:05:08.173979+0000","last_fullsized":"2024-03-24T12:05:08.173979+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:02.711167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:03:02.711167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":562,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190277+0000","last_change":"2024-03-24T12:03:17.558368+0000","last_active":"2024-03-24T12:05:08.190277+0000","last_peered":"2024-03-24T12:05:08.190277+0000","last_clean":"2024-03-24T12:05:08.190277+0000","last_became_active":"2024-03-24T12:03:15.298983+0000","last_became_peered":"2024-03-24T12:03:15.298983+0000","last_unstale":"2024-03-24T12:05:08.190277+0000","last_undegraded":"2024-03-24T12:05:08.190277+0000","last_fullsized":"2024-03-24T12:05:08.190277+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:17.558199+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:03:17.558199+0000","last_clean_scrub_stamp":"2024-03-24T12:03:17.558199+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"83'130","reported_seq":561,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181646+0000","last_change":"2024-03-24T12:03:43.603483+0000","last_active":"2024-03-24T12:05:08.181646+0000","last_peered":"2024-03-24T12:05:08.181646+0000","last_clean":"2024-03-24T12:05:08.181646+0000","last_became_active":"2024-03-24T12:03:19.827283+0000","last_became_peered":"2024-03-24T12:03:19.827283+0000","last_unstale":"2024-03-24T12:05:08.181646+0000","last_undegraded":"2024-03-24T12:05:08.181646+0000","last_fullsized":"2024-03-24T12:05:08.181646+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:43.603335+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:43.603335+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"296'11799","reported_seq":11958,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.287453+0000","last_change":"2024-03-24T12:05:15.798010+0000","last_active":"2024-03-24T12:05:16.287453+0000","last_peered":"2024-03-24T12:05:16.287453+0000","last_clean":"2024-03-24T12:05:16.287453+0000","last_became_active":"2024-03-24T12:02:46.438409+0000","last_became_peered":"2024-03-24T12:02:46.438409+0000","last_unstale":"2024-03-24T12:05:16.287453+0000","last_undegraded":"2024-03-24T12:05:16.287453+0000","last_fullsized":"2024-03-24T12:05:16.287453+0000","mapping_epoch":246,"log_start":"277'7000","ondisk_log_start":"277'7000","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'11761","last_scrub_stamp":"2024-03-24T12:05:15.797693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:15.797693+0000","objects_scrubbed":4070,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:27.948738+0000","scrub_duration":1009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262602752,"num_objects":4007,"num_object_clones":0,"num_object_copies":12021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11799,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":514,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184889+0000","last_change":"2024-03-24T12:03:49.293374+0000","last_active":"2024-03-24T12:05:08.184889+0000","last_peered":"2024-03-24T12:05:08.184889+0000","last_clean":"2024-03-24T12:05:08.184889+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:05:08.184889+0000","last_undegraded":"2024-03-24T12:05:08.184889+0000","last_fullsized":"2024-03-24T12:05:08.184889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:49.292942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:49.292942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:11.359075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":566,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183611+0000","last_change":"2024-03-24T12:02:57.909944+0000","last_active":"2024-03-24T12:05:08.183611+0000","last_peered":"2024-03-24T12:05:08.183611+0000","last_clean":"2024-03-24T12:05:08.183611+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:05:08.183611+0000","last_undegraded":"2024-03-24T12:05:08.183611+0000","last_fullsized":"2024-03-24T12:05:08.183611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:02:57.909767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:57.909767+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":532,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186851+0000","last_change":"2024-03-24T12:03:53.808708+0000","last_active":"2024-03-24T12:05:08.186851+0000","last_peered":"2024-03-24T12:05:08.186851+0000","last_clean":"2024-03-24T12:05:08.186851+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:05:08.186851+0000","last_undegraded":"2024-03-24T12:05:08.186851+0000","last_fullsized":"2024-03-24T12:05:08.186851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:03:53.808558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:53.808558+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190535+0000","last_change":"2024-03-24T12:04:55.858996+0000","last_active":"2024-03-24T12:05:08.190535+0000","last_peered":"2024-03-24T12:05:08.190535+0000","last_clean":"2024-03-24T12:05:08.190535+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:05:08.190535+0000","last_undegraded":"2024-03-24T12:05:08.190535+0000","last_fullsized":"2024-03-24T12:05:08.190535+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:55.858730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:04:55.858730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:19.264966+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"83'130","reported_seq":386,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181465+0000","last_change":"2024-03-24T12:03:20.685816+0000","last_active":"2024-03-24T12:05:08.181465+0000","last_peered":"2024-03-24T12:05:08.181465+0000","last_clean":"2024-03-24T12:05:08.181465+0000","last_became_active":"2024-03-24T12:03:19.828385+0000","last_became_peered":"2024-03-24T12:03:19.828385+0000","last_unstale":"2024-03-24T12:05:08.181465+0000","last_undegraded":"2024-03-24T12:05:08.181465+0000","last_fullsized":"2024-03-24T12:05:08.181465+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:20.685628+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:20.685628+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":538,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172511+0000","last_change":"2024-03-24T12:03:56.951891+0000","last_active":"2024-03-24T12:05:08.172511+0000","last_peered":"2024-03-24T12:05:08.172511+0000","last_clean":"2024-03-24T12:05:08.172511+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:05:08.172511+0000","last_undegraded":"2024-03-24T12:05:08.172511+0000","last_fullsized":"2024-03-24T12:05:08.172511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:03:56.951745+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.951745+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:10.859861+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:15.567692+0000","last_change":"2024-03-24T12:05:15.567838+0000","last_active":"2024-03-24T12:05:15.567692+0000","last_peered":"2024-03-24T12:05:15.567692+0000","last_clean":"2024-03-24T12:05:15.567692+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T12:05:15.567692+0000","last_undegraded":"2024-03-24T12:05:15.567692+0000","last_fullsized":"2024-03-24T12:05:15.567692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:15.567364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:15.567364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:20.030418+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":498,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188402+0000","last_change":"2024-03-24T12:03:11.626954+0000","last_active":"2024-03-24T12:05:08.188402+0000","last_peered":"2024-03-24T12:05:08.188402+0000","last_clean":"2024-03-24T12:05:08.188402+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:05:08.188402+0000","last_undegraded":"2024-03-24T12:05:08.188402+0000","last_fullsized":"2024-03-24T12:05:08.188402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:11.626651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:11.626651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":510,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180278+0000","last_change":"2024-03-24T12:04:49.349330+0000","last_active":"2024-03-24T12:05:08.180278+0000","last_peered":"2024-03-24T12:05:08.180278+0000","last_clean":"2024-03-24T12:05:08.180278+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T12:05:08.180278+0000","last_undegraded":"2024-03-24T12:05:08.180278+0000","last_fullsized":"2024-03-24T12:05:08.180278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:49.349208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:49.349208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:01.447429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"83'130","reported_seq":382,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182535+0000","last_change":"2024-03-24T12:03:48.080327+0000","last_active":"2024-03-24T12:05:08.182535+0000","last_peered":"2024-03-24T12:05:08.182535+0000","last_clean":"2024-03-24T12:05:08.182535+0000","last_became_active":"2024-03-24T12:03:19.828310+0000","last_became_peered":"2024-03-24T12:03:19.828310+0000","last_unstale":"2024-03-24T12:05:08.182535+0000","last_undegraded":"2024-03-24T12:05:08.182535+0000","last_fullsized":"2024-03-24T12:05:08.182535+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:48.080209+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.080209+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":528,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188208+0000","last_change":"2024-03-24T12:03:14.587051+0000","last_active":"2024-03-24T12:05:08.188208+0000","last_peered":"2024-03-24T12:05:08.188208+0000","last_clean":"2024-03-24T12:05:08.188208+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:05:08.188208+0000","last_undegraded":"2024-03-24T12:05:08.188208+0000","last_fullsized":"2024-03-24T12:05:08.188208+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:03:14.586799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:14.586799+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":527,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188187+0000","last_change":"2024-03-24T12:04:55.414653+0000","last_active":"2024-03-24T12:05:08.188187+0000","last_peered":"2024-03-24T12:05:08.188187+0000","last_clean":"2024-03-24T12:05:08.188187+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:05:08.188187+0000","last_undegraded":"2024-03-24T12:05:08.188187+0000","last_fullsized":"2024-03-24T12:05:08.188187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:55.414526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:04:55.414526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:59.086401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":504,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183203+0000","last_change":"2024-03-24T12:04:11.242963+0000","last_active":"2024-03-24T12:05:08.183203+0000","last_peered":"2024-03-24T12:05:08.183203+0000","last_clean":"2024-03-24T12:05:08.183203+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T12:05:08.183203+0000","last_undegraded":"2024-03-24T12:05:08.183203+0000","last_fullsized":"2024-03-24T12:05:08.183203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:11.242845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:11.242845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:18.573926+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":512,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179063+0000","last_change":"2024-03-24T12:03:44.319205+0000","last_active":"2024-03-24T12:05:08.179063+0000","last_peered":"2024-03-24T12:05:08.179063+0000","last_clean":"2024-03-24T12:05:08.179063+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:05:08.179063+0000","last_undegraded":"2024-03-24T12:05:08.179063+0000","last_fullsized":"2024-03-24T12:05:08.179063+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:44.318488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:44.318488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"83'130","reported_seq":397,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172349+0000","last_change":"2024-03-24T12:03:47.814740+0000","last_active":"2024-03-24T12:05:08.172349+0000","last_peered":"2024-03-24T12:05:08.172349+0000","last_clean":"2024-03-24T12:05:08.172349+0000","last_became_active":"2024-03-24T12:03:19.827207+0000","last_became_peered":"2024-03-24T12:03:19.827207+0000","last_unstale":"2024-03-24T12:05:08.172349+0000","last_undegraded":"2024-03-24T12:05:08.172349+0000","last_fullsized":"2024-03-24T12:05:08.172349+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:47.814624+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:47.814624+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":545,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189953+0000","last_change":"2024-03-24T12:03:46.711296+0000","last_active":"2024-03-24T12:05:08.189953+0000","last_peered":"2024-03-24T12:05:08.189953+0000","last_clean":"2024-03-24T12:05:08.189953+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:05:08.189953+0000","last_undegraded":"2024-03-24T12:05:08.189953+0000","last_fullsized":"2024-03-24T12:05:08.189953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:03:46.710805+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:03:46.710805+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182828+0000","last_change":"2024-03-24T12:04:59.284655+0000","last_active":"2024-03-24T12:05:08.182828+0000","last_peered":"2024-03-24T12:05:08.182828+0000","last_clean":"2024-03-24T12:05:08.182828+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T12:05:08.182828+0000","last_undegraded":"2024-03-24T12:05:08.182828+0000","last_fullsized":"2024-03-24T12:05:08.182828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.284540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.284540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:03.580492+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":677,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.352407+0000","last_change":"2024-03-24T12:03:51.156004+0000","last_active":"2024-03-24T12:05:10.352407+0000","last_peered":"2024-03-24T12:05:10.352407+0000","last_clean":"2024-03-24T12:05:10.352407+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T12:05:10.352407+0000","last_undegraded":"2024-03-24T12:05:10.352407+0000","last_fullsized":"2024-03-24T12:05:10.352407+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:03:51.155727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.155727+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":501,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172351+0000","last_change":"2024-03-24T12:03:57.971802+0000","last_active":"2024-03-24T12:05:08.172351+0000","last_peered":"2024-03-24T12:05:08.172351+0000","last_clean":"2024-03-24T12:05:08.172351+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:05:08.172351+0000","last_undegraded":"2024-03-24T12:05:08.172351+0000","last_fullsized":"2024-03-24T12:05:08.172351+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:57.971678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:57.971678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:09.634713+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"83'130","reported_seq":382,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181334+0000","last_change":"2024-03-24T12:03:19.958725+0000","last_active":"2024-03-24T12:05:08.181334+0000","last_peered":"2024-03-24T12:05:08.181334+0000","last_clean":"2024-03-24T12:05:08.181334+0000","last_became_active":"2024-03-24T12:03:19.823862+0000","last_became_peered":"2024-03-24T12:03:19.823862+0000","last_unstale":"2024-03-24T12:05:08.181334+0000","last_undegraded":"2024-03-24T12:05:08.181334+0000","last_fullsized":"2024-03-24T12:05:08.181334+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:19.958576+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.958576+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":541,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.245873+0000","last_change":"2024-03-24T12:05:16.245975+0000","last_active":"2024-03-24T12:05:16.245873+0000","last_peered":"2024-03-24T12:05:16.245873+0000","last_clean":"2024-03-24T12:05:16.245873+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T12:05:16.245873+0000","last_undegraded":"2024-03-24T12:05:16.245873+0000","last_fullsized":"2024-03-24T12:05:16.245873+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:32.914881+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":513,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.648374+0000","last_change":"2024-03-24T12:05:10.648412+0000","last_active":"2024-03-24T12:05:10.648374+0000","last_peered":"2024-03-24T12:05:10.648374+0000","last_clean":"2024-03-24T12:05:10.648374+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T12:05:10.648374+0000","last_undegraded":"2024-03-24T12:05:10.648374+0000","last_fullsized":"2024-03-24T12:05:10.648374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.648144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.648144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:40.482552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":670,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190420+0000","last_change":"2024-03-24T12:03:08.608291+0000","last_active":"2024-03-24T12:05:08.190420+0000","last_peered":"2024-03-24T12:05:08.190420+0000","last_clean":"2024-03-24T12:05:08.190420+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T12:05:08.190420+0000","last_undegraded":"2024-03-24T12:05:08.190420+0000","last_fullsized":"2024-03-24T12:05:08.190420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:03:08.607673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:08.607673+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.187400+0000","last_change":"2024-03-24T12:05:03.041683+0000","last_active":"2024-03-24T12:05:08.187400+0000","last_peered":"2024-03-24T12:05:08.187400+0000","last_clean":"2024-03-24T12:05:08.187400+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T12:05:08.187400+0000","last_undegraded":"2024-03-24T12:05:08.187400+0000","last_fullsized":"2024-03-24T12:05:08.187400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.041566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.041566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:05.581449+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":308,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185933+0000","last_change":"2024-03-24T12:03:48.299639+0000","last_active":"2024-03-24T12:05:08.185933+0000","last_peered":"2024-03-24T12:05:08.185933+0000","last_clean":"2024-03-24T12:05:08.185933+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:05:08.185933+0000","last_undegraded":"2024-03-24T12:05:08.185933+0000","last_fullsized":"2024-03-24T12:05:08.185933+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:03:48.299495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.299495+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":527,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189611+0000","last_change":"2024-03-24T12:04:05.334498+0000","last_active":"2024-03-24T12:05:08.189611+0000","last_peered":"2024-03-24T12:05:08.189611+0000","last_clean":"2024-03-24T12:05:08.189611+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T12:05:08.189611+0000","last_undegraded":"2024-03-24T12:05:08.189611+0000","last_fullsized":"2024-03-24T12:05:08.189611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:05.334379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:04:05.334379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:15.371303+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":506,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190401+0000","last_change":"2024-03-24T12:03:58.669666+0000","last_active":"2024-03-24T12:05:08.190401+0000","last_peered":"2024-03-24T12:05:08.190401+0000","last_clean":"2024-03-24T12:05:08.190401+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T12:05:08.190401+0000","last_undegraded":"2024-03-24T12:05:08.190401+0000","last_fullsized":"2024-03-24T12:05:08.190401+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.669210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.669210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:25.766145+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":501,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.174405+0000","last_change":"2024-03-24T12:03:52.887118+0000","last_active":"2024-03-24T12:05:08.174405+0000","last_peered":"2024-03-24T12:05:08.174405+0000","last_clean":"2024-03-24T12:05:08.174405+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T12:05:08.174405+0000","last_undegraded":"2024-03-24T12:05:08.174405+0000","last_fullsized":"2024-03-24T12:05:08.174405+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.887008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.887008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":539,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184731+0000","last_change":"2024-03-24T12:03:17.860265+0000","last_active":"2024-03-24T12:05:08.184731+0000","last_peered":"2024-03-24T12:05:08.184731+0000","last_clean":"2024-03-24T12:05:08.184731+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:05:08.184731+0000","last_undegraded":"2024-03-24T12:05:08.184731+0000","last_fullsized":"2024-03-24T12:05:08.184731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:03:17.860114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:17.860114+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":537,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:13.110283+0000","last_change":"2024-03-24T12:05:13.110324+0000","last_active":"2024-03-24T12:05:13.110283+0000","last_peered":"2024-03-24T12:05:13.110283+0000","last_clean":"2024-03-24T12:05:13.110283+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T12:05:13.110283+0000","last_undegraded":"2024-03-24T12:05:13.110283+0000","last_fullsized":"2024-03-24T12:05:13.110283+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:13.110170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.110170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:29.861447+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182976+0000","last_change":"2024-03-24T12:03:10.080967+0000","last_active":"2024-03-24T12:05:08.182976+0000","last_peered":"2024-03-24T12:05:08.182976+0000","last_clean":"2024-03-24T12:05:08.182976+0000","last_became_active":"2024-03-24T11:59:19.099779+0000","last_became_peered":"2024-03-24T11:59:19.099779+0000","last_unstale":"2024-03-24T12:05:08.182976+0000","last_undegraded":"2024-03-24T12:05:08.182976+0000","last_fullsized":"2024-03-24T12:05:08.182976+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:10.080832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:03:10.080832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":494,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184966+0000","last_change":"2024-03-24T12:03:04.639223+0000","last_active":"2024-03-24T12:05:08.184966+0000","last_peered":"2024-03-24T12:05:08.184966+0000","last_clean":"2024-03-24T12:05:08.184966+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T12:05:08.184966+0000","last_undegraded":"2024-03-24T12:05:08.184966+0000","last_fullsized":"2024-03-24T12:05:08.184966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:04.638975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:04.638975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":524,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178141+0000","last_change":"2024-03-24T12:02:58.593788+0000","last_active":"2024-03-24T12:05:08.178141+0000","last_peered":"2024-03-24T12:05:08.178141+0000","last_clean":"2024-03-24T12:05:08.178141+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:05:08.178141+0000","last_undegraded":"2024-03-24T12:05:08.178141+0000","last_fullsized":"2024-03-24T12:05:08.178141+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:02:58.593653+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:02:58.593653+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":512,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183399+0000","last_change":"2024-03-24T12:03:18.028601+0000","last_active":"2024-03-24T12:05:08.183399+0000","last_peered":"2024-03-24T12:05:08.183399+0000","last_clean":"2024-03-24T12:05:08.183399+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T12:05:08.183399+0000","last_undegraded":"2024-03-24T12:05:08.183399+0000","last_fullsized":"2024-03-24T12:05:08.183399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:18.028433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:18.028433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":519,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188106+0000","last_change":"2024-03-24T12:04:12.475747+0000","last_active":"2024-03-24T12:05:08.188106+0000","last_peered":"2024-03-24T12:05:08.188106+0000","last_clean":"2024-03-24T12:05:08.188106+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:05:08.188106+0000","last_undegraded":"2024-03-24T12:05:08.188106+0000","last_fullsized":"2024-03-24T12:05:08.188106+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.475631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.475631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:23.221135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":509,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179964+0000","last_change":"2024-03-24T12:04:50.160877+0000","last_active":"2024-03-24T12:05:08.179964+0000","last_peered":"2024-03-24T12:05:08.179964+0000","last_clean":"2024-03-24T12:05:08.179964+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:05:08.179964+0000","last_undegraded":"2024-03-24T12:05:08.179964+0000","last_fullsized":"2024-03-24T12:05:08.179964+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:50.160745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:04:50.160745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.457289+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"83'130","reported_seq":558,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180103+0000","last_change":"2024-03-24T12:04:51.411706+0000","last_active":"2024-03-24T12:05:08.180103+0000","last_peered":"2024-03-24T12:05:08.180103+0000","last_clean":"2024-03-24T12:05:08.180103+0000","last_became_active":"2024-03-24T12:03:19.819346+0000","last_became_peered":"2024-03-24T12:03:19.819346+0000","last_unstale":"2024-03-24T12:05:08.180103+0000","last_undegraded":"2024-03-24T12:05:08.180103+0000","last_fullsized":"2024-03-24T12:05:08.180103+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:51.411487+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:51.411487+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:11.653420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":533,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185492+0000","last_change":"2024-03-24T12:03:00.601558+0000","last_active":"2024-03-24T12:05:08.185492+0000","last_peered":"2024-03-24T12:05:08.185492+0000","last_clean":"2024-03-24T12:05:08.185492+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:05:08.185492+0000","last_undegraded":"2024-03-24T12:05:08.185492+0000","last_fullsized":"2024-03-24T12:05:08.185492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:03:00.601428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:00.601428+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":519,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190374+0000","last_change":"2024-03-24T12:04:02.372915+0000","last_active":"2024-03-24T12:05:08.190374+0000","last_peered":"2024-03-24T12:05:08.190374+0000","last_clean":"2024-03-24T12:05:08.190374+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T12:05:08.190374+0000","last_undegraded":"2024-03-24T12:05:08.190374+0000","last_fullsized":"2024-03-24T12:05:08.190374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.372798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.372798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:28.536106+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":513,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179124+0000","last_change":"2024-03-24T12:04:12.188145+0000","last_active":"2024-03-24T12:05:08.179124+0000","last_peered":"2024-03-24T12:05:08.179124+0000","last_clean":"2024-03-24T12:05:08.179124+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:05:08.179124+0000","last_undegraded":"2024-03-24T12:05:08.179124+0000","last_fullsized":"2024-03-24T12:05:08.179124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.188028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.188028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:33.103624+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":494,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190657+0000","last_change":"2024-03-24T12:03:05.719881+0000","last_active":"2024-03-24T12:05:08.190657+0000","last_peered":"2024-03-24T12:05:08.190657+0000","last_clean":"2024-03-24T12:05:08.190657+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:05:08.190657+0000","last_undegraded":"2024-03-24T12:05:08.190657+0000","last_fullsized":"2024-03-24T12:05:08.190657+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:05.719621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:05.719621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":545,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172709+0000","last_change":"2024-03-24T12:03:54.900129+0000","last_active":"2024-03-24T12:05:08.172709+0000","last_peered":"2024-03-24T12:05:08.172709+0000","last_clean":"2024-03-24T12:05:08.172709+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:05:08.172709+0000","last_undegraded":"2024-03-24T12:05:08.172709+0000","last_fullsized":"2024-03-24T12:05:08.172709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:03:54.899985+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:03:54.899985+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182258+0000","last_change":"2024-03-24T12:04:12.161106+0000","last_active":"2024-03-24T12:05:08.182258+0000","last_peered":"2024-03-24T12:05:08.182258+0000","last_clean":"2024-03-24T12:05:08.182258+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T12:05:08.182258+0000","last_undegraded":"2024-03-24T12:05:08.182258+0000","last_fullsized":"2024-03-24T12:05:08.182258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.160982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.160982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:18.654795+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":681,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.352339+0000","last_change":"2024-03-24T12:03:56.395856+0000","last_active":"2024-03-24T12:05:10.352339+0000","last_peered":"2024-03-24T12:05:10.352339+0000","last_clean":"2024-03-24T12:05:10.352339+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:05:10.352339+0000","last_undegraded":"2024-03-24T12:05:10.352339+0000","last_fullsized":"2024-03-24T12:05:10.352339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.395650+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:21.518834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":519,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180488+0000","last_change":"2024-03-24T12:03:53.366274+0000","last_active":"2024-03-24T12:05:08.180488+0000","last_peered":"2024-03-24T12:05:08.180488+0000","last_clean":"2024-03-24T12:05:08.180488+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T12:05:08.180488+0000","last_undegraded":"2024-03-24T12:05:08.180488+0000","last_fullsized":"2024-03-24T12:05:08.180488+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:03:53.366154+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:53.366154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:09.034022+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":555,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190607+0000","last_change":"2024-03-24T12:03:46.337757+0000","last_active":"2024-03-24T12:05:08.190607+0000","last_peered":"2024-03-24T12:05:08.190607+0000","last_clean":"2024-03-24T12:05:08.190607+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T12:05:08.190607+0000","last_undegraded":"2024-03-24T12:05:08.190607+0000","last_fullsized":"2024-03-24T12:05:08.190607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:03:46.337589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:46.337589+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":507,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185136+0000","last_change":"2024-03-24T12:03:09.599073+0000","last_active":"2024-03-24T12:05:08.185136+0000","last_peered":"2024-03-24T12:05:08.185136+0000","last_clean":"2024-03-24T12:05:08.185136+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:05:08.185136+0000","last_undegraded":"2024-03-24T12:05:08.185136+0000","last_fullsized":"2024-03-24T12:05:08.185136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:03:09.598786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:09.598786+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":506,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172790+0000","last_change":"2024-03-24T12:03:13.743176+0000","last_active":"2024-03-24T12:05:08.172790+0000","last_peered":"2024-03-24T12:05:08.172790+0000","last_clean":"2024-03-24T12:05:08.172790+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T12:05:08.172790+0000","last_undegraded":"2024-03-24T12:05:08.172790+0000","last_fullsized":"2024-03-24T12:05:08.172790+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:13.743060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:03:13.743060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189208+0000","last_change":"2024-03-24T12:04:14.188719+0000","last_active":"2024-03-24T12:05:08.189208+0000","last_peered":"2024-03-24T12:05:08.189208+0000","last_clean":"2024-03-24T12:05:08.189208+0000","last_became_active":"2024-03-24T11:59:19.098557+0000","last_became_peered":"2024-03-24T11:59:19.098557+0000","last_unstale":"2024-03-24T12:05:08.189208+0000","last_undegraded":"2024-03-24T12:05:08.189208+0000","last_fullsized":"2024-03-24T12:05:08.189208+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:04:14.188600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:33.242073+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":576,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:14.526556+0000","last_change":"2024-03-24T12:05:14.526595+0000","last_active":"2024-03-24T12:05:14.526556+0000","last_peered":"2024-03-24T12:05:14.526556+0000","last_clean":"2024-03-24T12:05:14.526556+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T12:05:14.526556+0000","last_undegraded":"2024-03-24T12:05:14.526556+0000","last_fullsized":"2024-03-24T12:05:14.526556+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:05:14.526488+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.526488+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:39.548612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":519,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186181+0000","last_change":"2024-03-24T12:03:00.369508+0000","last_active":"2024-03-24T12:05:08.186181+0000","last_peered":"2024-03-24T12:05:08.186181+0000","last_clean":"2024-03-24T12:05:08.186181+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:05:08.186181+0000","last_undegraded":"2024-03-24T12:05:08.186181+0000","last_fullsized":"2024-03-24T12:05:08.186181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:00.369376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:20.890106+0000","last_clean_scrub_stamp":"2024-03-24T12:03:00.369376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":273,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.187896+0000","last_change":"2024-03-24T12:04:57.823173+0000","last_active":"2024-03-24T12:05:08.187896+0000","last_peered":"2024-03-24T12:05:08.187896+0000","last_clean":"2024-03-24T12:05:08.187896+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:05:08.187896+0000","last_undegraded":"2024-03-24T12:05:08.187896+0000","last_fullsized":"2024-03-24T12:05:08.187896+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:57.823045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:57.823045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.912865+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":512,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186872+0000","last_change":"2024-03-24T12:04:53.809154+0000","last_active":"2024-03-24T12:05:08.186872+0000","last_peered":"2024-03-24T12:05:08.186872+0000","last_clean":"2024-03-24T12:05:08.186872+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T12:05:08.186872+0000","last_undegraded":"2024-03-24T12:05:08.186872+0000","last_fullsized":"2024-03-24T12:05:08.186872+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:53.809021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:53.809021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:09.097308+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":565,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178475+0000","last_change":"2024-03-24T12:05:01.677576+0000","last_active":"2024-03-24T12:05:08.178475+0000","last_peered":"2024-03-24T12:05:08.178475+0000","last_clean":"2024-03-24T12:05:08.178475+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:05:08.178475+0000","last_undegraded":"2024-03-24T12:05:08.178475+0000","last_fullsized":"2024-03-24T12:05:08.178475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:05:01.675875+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.675875+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:03.534752+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188844+0000","last_change":"2024-03-24T12:03:51.776115+0000","last_active":"2024-03-24T12:05:08.188844+0000","last_peered":"2024-03-24T12:05:08.188844+0000","last_clean":"2024-03-24T12:05:08.188844+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:05:08.188844+0000","last_undegraded":"2024-03-24T12:05:08.188844+0000","last_fullsized":"2024-03-24T12:05:08.188844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:03:51.775978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.775978+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:15.993859+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":504,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186318+0000","last_change":"2024-03-24T12:03:07.915966+0000","last_active":"2024-03-24T12:05:08.186318+0000","last_peered":"2024-03-24T12:05:08.186318+0000","last_clean":"2024-03-24T12:05:08.186318+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T12:05:08.186318+0000","last_undegraded":"2024-03-24T12:05:08.186318+0000","last_fullsized":"2024-03-24T12:05:08.186318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:07.915894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:07.915894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":503,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188104+0000","last_change":"2024-03-24T12:03:19.844533+0000","last_active":"2024-03-24T12:05:08.188104+0000","last_peered":"2024-03-24T12:05:08.188104+0000","last_clean":"2024-03-24T12:05:08.188104+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T12:05:08.188104+0000","last_undegraded":"2024-03-24T12:05:08.188104+0000","last_fullsized":"2024-03-24T12:05:08.188104+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:19.844413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.844413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":541,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184015+0000","last_change":"2024-03-24T12:03:44.894651+0000","last_active":"2024-03-24T12:05:08.184015+0000","last_peered":"2024-03-24T12:05:08.184015+0000","last_clean":"2024-03-24T12:05:08.184015+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:05:08.184015+0000","last_undegraded":"2024-03-24T12:05:08.184015+0000","last_fullsized":"2024-03-24T12:05:08.184015+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:03:44.894523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:44.894523+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":532,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:09.177593+0000","last_change":"2024-03-24T12:05:09.177628+0000","last_active":"2024-03-24T12:05:09.177593+0000","last_peered":"2024-03-24T12:05:09.177593+0000","last_clean":"2024-03-24T12:05:09.177593+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T12:05:09.177593+0000","last_undegraded":"2024-03-24T12:05:09.177593+0000","last_fullsized":"2024-03-24T12:05:09.177593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.177509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:30.995931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":504,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183677+0000","last_change":"2024-03-24T12:03:52.099674+0000","last_active":"2024-03-24T12:05:08.183677+0000","last_peered":"2024-03-24T12:05:08.183677+0000","last_clean":"2024-03-24T12:05:08.183677+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T12:05:08.183677+0000","last_undegraded":"2024-03-24T12:05:08.183677+0000","last_fullsized":"2024-03-24T12:05:08.183677+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.099551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.099551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":493,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.175042+0000","last_change":"2024-03-24T12:03:19.828077+0000","last_active":"2024-03-24T12:05:08.175042+0000","last_peered":"2024-03-24T12:05:08.175042+0000","last_clean":"2024-03-24T12:05:08.175042+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:05:08.175042+0000","last_undegraded":"2024-03-24T12:05:08.175042+0000","last_fullsized":"2024-03-24T12:05:08.175042+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.827956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":531,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184891+0000","last_change":"2024-03-24T12:03:49.858697+0000","last_active":"2024-03-24T12:05:08.184891+0000","last_peered":"2024-03-24T12:05:08.184891+0000","last_clean":"2024-03-24T12:05:08.184891+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:05:08.184891+0000","last_undegraded":"2024-03-24T12:05:08.184891+0000","last_fullsized":"2024-03-24T12:05:08.184891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:03:49.858566+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:03:49.858566+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":529,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179432+0000","last_change":"2024-03-24T12:04:52.100350+0000","last_active":"2024-03-24T12:05:08.179432+0000","last_peered":"2024-03-24T12:05:08.179432+0000","last_clean":"2024-03-24T12:05:08.179432+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:05:08.179432+0000","last_undegraded":"2024-03-24T12:05:08.179432+0000","last_fullsized":"2024-03-24T12:05:08.179432+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:52.100251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:04:52.100251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:06.420776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179498+0000","last_change":"2024-03-24T12:05:05.120365+0000","last_active":"2024-03-24T12:05:08.179498+0000","last_peered":"2024-03-24T12:05:08.179498+0000","last_clean":"2024-03-24T12:05:08.179498+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:05:08.179498+0000","last_undegraded":"2024-03-24T12:05:08.179498+0000","last_fullsized":"2024-03-24T12:05:08.179498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:05.120099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.120099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:09.215137+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184342+0000","last_change":"2024-03-24T12:03:56.118050+0000","last_active":"2024-03-24T12:05:08.184342+0000","last_peered":"2024-03-24T12:05:08.184342+0000","last_clean":"2024-03-24T12:05:08.184342+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T12:05:08.184342+0000","last_undegraded":"2024-03-24T12:05:08.184342+0000","last_fullsized":"2024-03-24T12:05:08.184342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.117875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.117875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:15.713771+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":545,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182960+0000","last_change":"2024-03-24T12:05:06.830698+0000","last_active":"2024-03-24T12:05:08.182960+0000","last_peered":"2024-03-24T12:05:08.182960+0000","last_clean":"2024-03-24T12:05:08.182960+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:05:08.182960+0000","last_undegraded":"2024-03-24T12:05:08.182960+0000","last_fullsized":"2024-03-24T12:05:08.182960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:05:06.830559+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.830559+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.321532+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":515,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181199+0000","last_change":"2024-03-24T12:03:55.732013+0000","last_active":"2024-03-24T12:05:08.181199+0000","last_peered":"2024-03-24T12:05:08.181199+0000","last_clean":"2024-03-24T12:05:08.181199+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T12:05:08.181199+0000","last_undegraded":"2024-03-24T12:05:08.181199+0000","last_fullsized":"2024-03-24T12:05:08.181199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:55.731711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:03:55.731711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:12.824844+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182101+0000","last_change":"2024-03-24T12:04:10.294033+0000","last_active":"2024-03-24T12:05:08.182101+0000","last_peered":"2024-03-24T12:05:08.182101+0000","last_clean":"2024-03-24T12:05:08.182101+0000","last_became_active":"2024-03-24T11:59:19.117244+0000","last_became_peered":"2024-03-24T11:59:19.117244+0000","last_unstale":"2024-03-24T12:05:08.182101+0000","last_undegraded":"2024-03-24T12:05:08.182101+0000","last_fullsized":"2024-03-24T12:05:08.182101+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:10.293805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:10.293805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:39.610672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183928+0000","last_change":"2024-03-24T12:04:12.242117+0000","last_active":"2024-03-24T12:05:08.183928+0000","last_peered":"2024-03-24T12:05:08.183928+0000","last_clean":"2024-03-24T12:05:08.183928+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T12:05:08.183928+0000","last_undegraded":"2024-03-24T12:05:08.183928+0000","last_fullsized":"2024-03-24T12:05:08.183928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.241824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.241824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:21.596862+0000","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":560,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185592+0000","last_change":"2024-03-24T12:04:50.623302+0000","last_active":"2024-03-24T12:05:08.185592+0000","last_peered":"2024-03-24T12:05:08.185592+0000","last_clean":"2024-03-24T12:05:08.185592+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T12:05:08.185592+0000","last_undegraded":"2024-03-24T12:05:08.185592+0000","last_fullsized":"2024-03-24T12:05:08.185592+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:04:50.623172+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:04:50.623172+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:15.353509+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":506,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172924+0000","last_change":"2024-03-24T12:03:07.691595+0000","last_active":"2024-03-24T12:05:08.172924+0000","last_peered":"2024-03-24T12:05:08.172924+0000","last_clean":"2024-03-24T12:05:08.172924+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:05:08.172924+0000","last_undegraded":"2024-03-24T12:05:08.172924+0000","last_fullsized":"2024-03-24T12:05:08.172924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:07.691468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:03:07.691468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.732900+0000","last_change":"2024-03-24T12:05:12.732940+0000","last_active":"2024-03-24T12:05:12.732900+0000","last_peered":"2024-03-24T12:05:12.732900+0000","last_clean":"2024-03-24T12:05:12.732900+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T12:05:12.732900+0000","last_undegraded":"2024-03-24T12:05:12.732900+0000","last_fullsized":"2024-03-24T12:05:12.732900+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:05:12.732801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:30.917678+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190796+0000","last_change":"2024-03-24T12:04:54.805094+0000","last_active":"2024-03-24T12:05:08.190796+0000","last_peered":"2024-03-24T12:05:08.190796+0000","last_clean":"2024-03-24T12:05:08.190796+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:05:08.190796+0000","last_undegraded":"2024-03-24T12:05:08.190796+0000","last_fullsized":"2024-03-24T12:05:08.190796+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.804969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.804969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.941068+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4307419160,"num_objects":65727,"num_object_clones":0,"num_object_copies":197181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65727,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185895,"num_write_kb":8051841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12780646400,"data_stored":12780634184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73748,"ondisk_log_size":73748,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":364,"num_read_kb":359,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":781,"ondisk_log_size":781,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005216,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2801264,"kb_used_data":2314728,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":90943888,"statfs":{"total":95995035648,"available":93126541312,"internally_reserved":0,"allocated":2370281472,"data_stored":2369853698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332263,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151656,"kb_used_data":777504,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92593496,"statfs":{"total":95995035648,"available":94815739904,"internally_reserved":0,"allocated":796164096,"data_stored":795744308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2875192,"kb_used_data":2320560,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":90869960,"statfs":{"total":95995035648,"available":93050839040,"internally_reserved":0,"allocated":2376253440,"data_stored":2375816871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738564,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1587500,"kb_used_data":1096804,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92157652,"statfs":{"total":95995035648,"available":94369435648,"internally_reserved":0,"allocated":1123127296,"data_stored":1122702461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411515,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215476,"kb_used_data":769196,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92529676,"statfs":{"total":95995035648,"available":94750388224,"internally_reserved":0,"allocated":787656704,"data_stored":787225171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117172,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2778504,"kb_used_data":2201728,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":90966648,"statfs":{"total":95995035648,"available":93149847552,"internally_reserved":0,"allocated":2254569472,"data_stored":2254117889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":590608839},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920944,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145784,"kb_used_data":771696,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92599368,"statfs":{"total":95995035648,"available":94821752832,"internally_reserved":0,"allocated":790216704,"data_stored":789780146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659307,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2885112,"kb_used_data":2344304,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":90860040,"statfs":{"total":95995035648,"available":93040680960,"internally_reserved":0,"allocated":2400567296,"data_stored":2400131418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1116209152,"data_stored":1116209152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2354319360,"data_stored":2354315288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":780795904,"data_stored":780795904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2229075968,"data_stored":2229075968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":780533760,"data_stored":780533760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2382499840,"data_stored":2382495768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":784072704,"data_stored":784072704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2353139712,"data_stored":2353135640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:05:21.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:05:21.394 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:21.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:21.446 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:21.467 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:21.478 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:21.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[30223]: 5.6 deep-scrub starts 2024-03-24T12:05:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[30223]: 5.6 deep-scrub ok 2024-03-24T12:05:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T12:05:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T12:05:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[30223]: 1.b deep-scrub starts 2024-03-24T12:05:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[30223]: 1.b deep-scrub ok 2024-03-24T12:05:21.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[42243]: 5.6 deep-scrub starts 2024-03-24T12:05:21.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[42243]: 5.6 deep-scrub ok 2024-03-24T12:05:21.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T12:05:21.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T12:05:21.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[42243]: 1.b deep-scrub starts 2024-03-24T12:05:21.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:21 smithi064 ceph-mon[42243]: 1.b deep-scrub ok 2024-03-24T12:05:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:21.570 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:21.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:21 smithi193 ceph-mon[33593]: 5.6 deep-scrub starts 2024-03-24T12:05:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:21 smithi193 ceph-mon[33593]: 5.6 deep-scrub ok 2024-03-24T12:05:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:21 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T12:05:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:21 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T12:05:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:21 smithi193 ceph-mon[33593]: 1.b deep-scrub starts 2024-03-24T12:05:21.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:21 smithi193 ceph-mon[33593]: 1.b deep-scrub ok 2024-03-24T12:05:21.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:21.699 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:21.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:21.800 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:21.828 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:21.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:21.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:21.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:21.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:22.073 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:22.107 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:22.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:22.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:22.292 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:22.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:22.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:22.428 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:22.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[30223]: pgmap v591: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[30223]: 1.11 scrub starts 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[30223]: 1.11 scrub ok 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[30223]: 9.2 scrub starts 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[42243]: pgmap v591: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[42243]: 1.11 scrub starts 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[42243]: 1.11 scrub ok 2024-03-24T12:05:22.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:22 smithi064 ceph-mon[42243]: 9.2 scrub starts 2024-03-24T12:05:22.530 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:22.558 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:22.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:22 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:05:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:22 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:05:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:22 smithi193 ceph-mon[33593]: pgmap v591: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-24T12:05:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:22 smithi193 ceph-mon[33593]: 1.11 scrub starts 2024-03-24T12:05:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:22 smithi193 ceph-mon[33593]: 1.11 scrub ok 2024-03-24T12:05:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:22 smithi193 ceph-mon[33593]: 9.2 scrub starts 2024-03-24T12:05:22.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:22.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:22.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:22.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:22.846 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:22.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:22.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:23.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:23.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:23.137 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:23.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:23.240 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:23.280 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:23.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:23.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:23.415 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:23.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:23.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:23.546 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:23.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:23.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[42243]: 2.1a deep-scrub starts 2024-03-24T12:05:23.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[42243]: 2.1a deep-scrub ok 2024-03-24T12:05:23.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[42243]: 6.10 scrub starts 2024-03-24T12:05:23.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[42243]: 6.10 scrub ok 2024-03-24T12:05:23.547 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[42243]: 9.8 scrub starts 2024-03-24T12:05:23.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[30223]: 2.1a deep-scrub starts 2024-03-24T12:05:23.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[30223]: 2.1a deep-scrub ok 2024-03-24T12:05:23.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[30223]: 6.10 scrub starts 2024-03-24T12:05:23.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[30223]: 6.10 scrub ok 2024-03-24T12:05:23.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:23 smithi064 ceph-mon[30223]: 9.8 scrub starts 2024-03-24T12:05:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:23 smithi193 ceph-mon[33593]: 2.1a deep-scrub starts 2024-03-24T12:05:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:23 smithi193 ceph-mon[33593]: 2.1a deep-scrub ok 2024-03-24T12:05:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:23 smithi193 ceph-mon[33593]: 6.10 scrub starts 2024-03-24T12:05:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:23 smithi193 ceph-mon[33593]: 6.10 scrub ok 2024-03-24T12:05:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:23 smithi193 ceph-mon[33593]: 9.8 scrub starts 2024-03-24T12:05:23.648 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:23.680 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:23.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:23.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:23.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:23.823 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:23.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:23.925 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:23.957 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:23.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:23.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:24.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:24.088 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:24.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:24.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:24.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:24.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:24.329 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:24.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:24.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:24.464 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:24.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:24.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[30223]: 6.2 deep-scrub starts 2024-03-24T12:05:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[30223]: 6.2 deep-scrub ok 2024-03-24T12:05:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[30223]: 9.2 scrub ok 2024-03-24T12:05:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[30223]: 9.3 deep-scrub starts 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[30223]: pgmap v592: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[30223]: 4.1e scrub starts 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[30223]: 4.1e scrub ok 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[42243]: 6.2 deep-scrub starts 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[42243]: 6.2 deep-scrub ok 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[42243]: 9.2 scrub ok 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[42243]: 9.3 deep-scrub starts 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[42243]: pgmap v592: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[42243]: 4.1e scrub starts 2024-03-24T12:05:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:24 smithi064 ceph-mon[42243]: 4.1e scrub ok 2024-03-24T12:05:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:24 smithi193 ceph-mon[33593]: 6.2 deep-scrub starts 2024-03-24T12:05:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:24 smithi193 ceph-mon[33593]: 6.2 deep-scrub ok 2024-03-24T12:05:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:24 smithi193 ceph-mon[33593]: 9.2 scrub ok 2024-03-24T12:05:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:24 smithi193 ceph-mon[33593]: 9.3 deep-scrub starts 2024-03-24T12:05:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:24 smithi193 ceph-mon[33593]: pgmap v592: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:05:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:24 smithi193 ceph-mon[33593]: 4.1e scrub starts 2024-03-24T12:05:24.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:24 smithi193 ceph-mon[33593]: 4.1e scrub ok 2024-03-24T12:05:24.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:24.634 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:24.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:24.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:24.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:24.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:24.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:24.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:24.911 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:24.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:25.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:25.052 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:25.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:25.134 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:25.154 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:25.182 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:25.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:25.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:25.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:25 smithi064 ceph-mon[30223]: 1.7 deep-scrub starts 2024-03-24T12:05:25.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:25 smithi064 ceph-mon[30223]: 1.7 deep-scrub ok 2024-03-24T12:05:25.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:25 smithi064 ceph-mon[30223]: 9.8 scrub ok 2024-03-24T12:05:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:25 smithi064 ceph-mon[42243]: 1.7 deep-scrub starts 2024-03-24T12:05:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:25 smithi064 ceph-mon[42243]: 1.7 deep-scrub ok 2024-03-24T12:05:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:25 smithi064 ceph-mon[42243]: 9.8 scrub ok 2024-03-24T12:05:25.326 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:25.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:25.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:25.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:25.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:25.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:25 smithi193 ceph-mon[33593]: 1.7 deep-scrub starts 2024-03-24T12:05:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:25 smithi193 ceph-mon[33593]: 1.7 deep-scrub ok 2024-03-24T12:05:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:25 smithi193 ceph-mon[33593]: 9.8 scrub ok 2024-03-24T12:05:25.615 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:25.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:25.716 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:25.751 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:25.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:25.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:25.896 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:25.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:25.998 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:26.026 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:26.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:26.127 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:26.160 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:26.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:26.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:26.292 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:26.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:26.394 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:26.425 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:26.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:26.429 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:26 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T12:05:26.429 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:26 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T12:05:26.429 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:26 smithi064 ceph-mon[30223]: pgmap v593: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 867 op/s 2024-03-24T12:05:26.429 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:26 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T12:05:26.429 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:26 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T12:05:26.429 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:26 smithi064 ceph-mon[42243]: pgmap v593: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 867 op/s 2024-03-24T12:05:26.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:26.535 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:05:26.535 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:05:26.564 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:26.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:26 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T12:05:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:26 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T12:05:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:26 smithi193 ceph-mon[33593]: pgmap v593: 168 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 867 op/s 2024-03-24T12:05:26.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:26.702 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:26.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:26.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:26.848 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:26.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:26.949 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:26.989 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:26.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:27.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:27.129 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:27.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:27.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:27.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:27.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:27.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T12:05:27.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T12:05:27.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[30223]: 6.11 scrub starts 2024-03-24T12:05:27.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[30223]: 6.11 scrub ok 2024-03-24T12:05:27.290 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:27.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:05:27.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:27.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:27.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:27.538 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:27.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:27.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:05:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T12:05:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T12:05:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[42243]: 6.11 scrub starts 2024-03-24T12:05:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[42243]: 6.11 scrub ok 2024-03-24T12:05:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:27 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T12:05:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:27 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T12:05:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:27 smithi193 ceph-mon[33593]: 6.11 scrub starts 2024-03-24T12:05:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:27 smithi193 ceph-mon[33593]: 6.11 scrub ok 2024-03-24T12:05:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:27.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:05:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:05:27.640 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:27.676 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:27.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:27.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:27.812 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:27.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:27.914 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:27.952 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:27.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:28.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:28.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:28.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:28.251 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:28.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:28.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:28.403 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:28.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:28.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:28.542 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:28.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:28.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[30223]: 9.e scrub starts 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[30223]: pgmap v594: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[30223]: 3.1 deep-scrub starts 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[30223]: 3.1 deep-scrub ok 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[42243]: 9.e scrub starts 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[42243]: pgmap v594: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[42243]: 3.1 deep-scrub starts 2024-03-24T12:05:28.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:28 smithi064 ceph-mon[42243]: 3.1 deep-scrub ok 2024-03-24T12:05:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:28 smithi193 ceph-mon[33593]: 9.e scrub starts 2024-03-24T12:05:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:28 smithi193 ceph-mon[33593]: pgmap v594: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:05:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:28 smithi193 ceph-mon[33593]: 3.1 deep-scrub starts 2024-03-24T12:05:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:28 smithi193 ceph-mon[33593]: 3.1 deep-scrub ok 2024-03-24T12:05:28.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:28.676 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:28.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:28.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:28.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:28.809 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:28.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:28.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:28.940 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:28.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:28.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:29.041 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:29.074 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:29.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:29.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:29.211 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:29.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:29.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:29.357 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:29.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:29.365 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[30223]: 2.d deep-scrub starts 2024-03-24T12:05:29.365 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[30223]: 2.d deep-scrub ok 2024-03-24T12:05:29.365 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[30223]: 9.e scrub ok 2024-03-24T12:05:29.365 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[30223]: 5.7 scrub starts 2024-03-24T12:05:29.366 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[30223]: 5.7 scrub ok 2024-03-24T12:05:29.366 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[42243]: 2.d deep-scrub starts 2024-03-24T12:05:29.366 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[42243]: 2.d deep-scrub ok 2024-03-24T12:05:29.366 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[42243]: 9.e scrub ok 2024-03-24T12:05:29.366 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[42243]: 5.7 scrub starts 2024-03-24T12:05:29.366 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:29 smithi064 ceph-mon[42243]: 5.7 scrub ok 2024-03-24T12:05:29.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:29.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:29.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:29.506 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:05:29.562 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:05:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:29 smithi193 ceph-mon[33593]: 2.d deep-scrub starts 2024-03-24T12:05:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:29 smithi193 ceph-mon[33593]: 2.d deep-scrub ok 2024-03-24T12:05:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:29 smithi193 ceph-mon[33593]: 9.e scrub ok 2024-03-24T12:05:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:29 smithi193 ceph-mon[33593]: 5.7 scrub starts 2024-03-24T12:05:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:29 smithi193 ceph-mon[33593]: 5.7 scrub ok 2024-03-24T12:05:29.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:29.607 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:05:29.637 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:29.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:29.670 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:05:29.715 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:05:29.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:29.755 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:05:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:29.784 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:29.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:29.801 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:05:29.848 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:05:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:05:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:05:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:05:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:05:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:05:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:05:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:05:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:05:29.885 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:29.914 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:29.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:05:29.990 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:05:30.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:30.075 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:30.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:30.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:30.209 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:30.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:30.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:30.337 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:30.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:30.438 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:30.472 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:30.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:30.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T12:05:30.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T12:05:30.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[30223]: pgmap v595: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 994 op/s 2024-03-24T12:05:30.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1810709411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:30.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[30223]: from='client.25465 v1:172.21.15.64:0/2194693085' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:05:30.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T12:05:30.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T12:05:30.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[42243]: pgmap v595: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 994 op/s 2024-03-24T12:05:30.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1810709411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:30.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:30 smithi064 ceph-mon[42243]: from='client.25465 v1:172.21.15.64:0/2194693085' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:05:30.573 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:30 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T12:05:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:30 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T12:05:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:30 smithi193 ceph-mon[33593]: pgmap v595: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 994 op/s 2024-03-24T12:05:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:30 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1810709411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:30.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:30 smithi193 ceph-mon[33593]: from='client.25465 v1:172.21.15.64:0/2194693085' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:05:30.609 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:30.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:30.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:30.711 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:30.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:30.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:30.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:30.879 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:30.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:30.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:31.023 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:31.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:31.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:31.162 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:31.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:31.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:31.299 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:31.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:31.321 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:05:31.400 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:31.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:31.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:31.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[30223]: 4.1 scrub starts 2024-03-24T12:05:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[30223]: 4.1 scrub ok 2024-03-24T12:05:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[30223]: 9.f deep-scrub ok 2024-03-24T12:05:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2287953221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:05:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[30223]: from='client.25471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:05:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[42243]: 4.1 scrub starts 2024-03-24T12:05:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[42243]: 4.1 scrub ok 2024-03-24T12:05:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[42243]: 9.f deep-scrub ok 2024-03-24T12:05:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2287953221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:05:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:31 smithi064 ceph-mon[42243]: from='client.25471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:05:31.575 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:31.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:31 smithi193 ceph-mon[33593]: 4.1 scrub starts 2024-03-24T12:05:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:31 smithi193 ceph-mon[33593]: 4.1 scrub ok 2024-03-24T12:05:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:31 smithi193 ceph-mon[33593]: 9.f deep-scrub ok 2024-03-24T12:05:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2287953221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:05:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:31 smithi193 ceph-mon[33593]: from='client.25471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:05:31.676 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:31.715 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:31.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:31.764 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2024-03-24T12:05:29.389045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3526367606,"num_objects":53998,"num_object_clones":0,"num_object_copies":161994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53998,"num_whiteouts":0,"num_read":1518,"num_read_kb":1270,"num_write":198544,"num_write_kb":8051886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77310,"ondisk_log_size":77310,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14326056,"kb_used_data":10456808,"kb_used_omap":12,"kb_used_meta":3868595,"kb_avail":735635160,"statfs":{"total":767960285184,"available":753290403840,"internally_reserved":0,"allocated":10707771392,"data_stored":10704306874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3961441840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-781647872,"num_objects":-11927,"num_object_clones":0,"num_object_copies":-35781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11927,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11927,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005308"},"pg_stats":[{"pgid":"1.d","version":"83'130","reported_seq":392,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178615+0000","last_change":"2024-03-24T12:05:08.153739+0000","last_active":"2024-03-24T12:05:08.178615+0000","last_peered":"2024-03-24T12:05:08.178615+0000","last_clean":"2024-03-24T12:05:08.178615+0000","last_became_active":"2024-03-24T12:03:19.822426+0000","last_became_peered":"2024-03-24T12:03:19.822426+0000","last_unstale":"2024-03-24T12:05:08.178615+0000","last_undegraded":"2024-03-24T12:05:08.178615+0000","last_fullsized":"2024-03-24T12:05:08.178615+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:08.153554+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:08.153554+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:36.332030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"296'12472","reported_seq":12713,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:27.962786+0000","last_change":"2024-03-24T12:05:01.328489+0000","last_active":"2024-03-24T12:05:27.962786+0000","last_peered":"2024-03-24T12:05:27.962786+0000","last_clean":"2024-03-24T12:05:27.962786+0000","last_became_active":"2024-03-24T12:02:46.085796+0000","last_became_peered":"2024-03-24T12:02:46.085796+0000","last_unstale":"2024-03-24T12:05:27.962786+0000","last_undegraded":"2024-03-24T12:05:27.962786+0000","last_fullsized":"2024-03-24T12:05:27.962786+0000","mapping_epoch":246,"log_start":"278'7200","ondisk_log_start":"278'7200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'10673","last_scrub_stamp":"2024-03-24T12:05:01.328295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.328295+0000","objects_scrubbed":5181,"log_size":5272,"log_dups_size":0,"ondisk_log_size":5272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:02.812687+0000","scrub_duration":1847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216924160,"num_objects":3310,"num_object_clones":0,"num_object_copies":9930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12472,"num_write_kb":505024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":690,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:25.355718+0000","last_change":"2024-03-24T12:05:10.185964+0000","last_active":"2024-03-24T12:05:25.355718+0000","last_peered":"2024-03-24T12:05:25.355718+0000","last_clean":"2024-03-24T12:05:25.355718+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:05:25.355718+0000","last_undegraded":"2024-03-24T12:05:25.355718+0000","last_fullsized":"2024-03-24T12:05:25.355718+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:05:10.185824+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T11:59:08.941456+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.185824+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:19.277117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":560,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:17.253802+0000","last_change":"2024-03-24T12:05:17.253802+0000","last_active":"2024-03-24T12:05:17.253802+0000","last_peered":"2024-03-24T12:05:17.253802+0000","last_clean":"2024-03-24T12:05:17.253802+0000","last_became_active":"2024-03-24T11:52:17.423531+0000","last_became_peered":"2024-03-24T11:52:17.423531+0000","last_unstale":"2024-03-24T12:05:17.253802+0000","last_undegraded":"2024-03-24T12:05:17.253802+0000","last_fullsized":"2024-03-24T12:05:17.253802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:05:17.253630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.253630+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:41.333041+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:11.506057+0000","last_change":"2024-03-24T12:05:11.506093+0000","last_active":"2024-03-24T12:05:11.506057+0000","last_peered":"2024-03-24T12:05:11.506057+0000","last_clean":"2024-03-24T12:05:11.506057+0000","last_became_active":"2024-03-24T11:52:15.143512+0000","last_became_peered":"2024-03-24T11:52:15.143512+0000","last_unstale":"2024-03-24T12:05:11.506057+0000","last_undegraded":"2024-03-24T12:05:11.506057+0000","last_fullsized":"2024-03-24T12:05:11.506057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:11.505956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.505956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:12.205758+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":501,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182805+0000","last_change":"2024-03-24T12:03:58.095718+0000","last_active":"2024-03-24T12:05:08.182805+0000","last_peered":"2024-03-24T12:05:08.182805+0000","last_clean":"2024-03-24T12:05:08.182805+0000","last_became_active":"2024-03-24T11:52:21.852195+0000","last_became_peered":"2024-03-24T11:52:21.852195+0000","last_unstale":"2024-03-24T12:05:08.182805+0000","last_undegraded":"2024-03-24T12:05:08.182805+0000","last_fullsized":"2024-03-24T12:05:08.182805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.095464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.095464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'130","reported_seq":383,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186592+0000","last_change":"2024-03-24T12:03:19.942347+0000","last_active":"2024-03-24T12:05:08.186592+0000","last_peered":"2024-03-24T12:05:08.186592+0000","last_clean":"2024-03-24T12:05:08.186592+0000","last_became_active":"2024-03-24T12:03:19.822938+0000","last_became_peered":"2024-03-24T12:03:19.822938+0000","last_unstale":"2024-03-24T12:05:08.186592+0000","last_undegraded":"2024-03-24T12:05:08.186592+0000","last_fullsized":"2024-03-24T12:05:08.186592+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:19.942241+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.942241+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"296'12646","reported_seq":12809,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:26.671322+0000","last_change":"2024-03-24T12:05:13.460893+0000","last_active":"2024-03-24T12:05:26.671322+0000","last_peered":"2024-03-24T12:05:26.671322+0000","last_clean":"2024-03-24T12:05:26.671322+0000","last_became_active":"2024-03-24T12:02:46.437833+0000","last_became_peered":"2024-03-24T12:02:46.437833+0000","last_unstale":"2024-03-24T12:05:26.671322+0000","last_undegraded":"2024-03-24T12:05:26.671322+0000","last_fullsized":"2024-03-24T12:05:26.671322+0000","mapping_epoch":246,"log_start":"285'8100","ondisk_log_start":"285'8100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'11770","last_scrub_stamp":"2024-03-24T12:05:13.460650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.460650+0000","objects_scrubbed":4284,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:13.546677+0000","scrub_duration":1179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221511680,"num_objects":3380,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":512832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":534,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.174407+0000","last_change":"2024-03-24T12:03:49.868525+0000","last_active":"2024-03-24T12:05:08.174407+0000","last_peered":"2024-03-24T12:05:08.174407+0000","last_clean":"2024-03-24T12:05:08.174407+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:05:08.174407+0000","last_undegraded":"2024-03-24T12:05:08.174407+0000","last_fullsized":"2024-03-24T12:05:08.174407+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:03:49.868385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:49.868385+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":529,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.191352+0000","last_change":"2024-03-24T12:04:09.866901+0000","last_active":"2024-03-24T12:05:08.191352+0000","last_peered":"2024-03-24T12:05:08.191352+0000","last_clean":"2024-03-24T12:05:08.191352+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:05:08.191352+0000","last_undegraded":"2024-03-24T12:05:08.191352+0000","last_fullsized":"2024-03-24T12:05:08.191352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:09.866736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:37.995261+0000","last_clean_scrub_stamp":"2024-03-24T12:04:09.866736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:27.550845+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":523,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179850+0000","last_change":"2024-03-24T12:04:52.456633+0000","last_active":"2024-03-24T12:05:08.179850+0000","last_peered":"2024-03-24T12:05:08.179850+0000","last_clean":"2024-03-24T12:05:08.179850+0000","last_became_active":"2024-03-24T11:52:19.383819+0000","last_became_peered":"2024-03-24T11:52:19.383819+0000","last_unstale":"2024-03-24T12:05:08.179850+0000","last_undegraded":"2024-03-24T12:05:08.179850+0000","last_fullsized":"2024-03-24T12:05:08.179850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:52.456151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:52.456151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:09.892431+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:14.719255+0000","last_change":"2024-03-24T12:05:14.719255+0000","last_active":"2024-03-24T12:05:14.719255+0000","last_peered":"2024-03-24T12:05:14.719255+0000","last_clean":"2024-03-24T12:05:14.719255+0000","last_became_active":"2024-03-24T11:52:21.386844+0000","last_became_peered":"2024-03-24T11:52:21.386844+0000","last_unstale":"2024-03-24T12:05:14.719255+0000","last_undegraded":"2024-03-24T12:05:14.719255+0000","last_fullsized":"2024-03-24T12:05:14.719255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:14.719129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.719129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.565197+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'130","reported_seq":577,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190168+0000","last_change":"2024-03-24T12:05:02.370703+0000","last_active":"2024-03-24T12:05:08.190168+0000","last_peered":"2024-03-24T12:05:08.190168+0000","last_clean":"2024-03-24T12:05:08.190168+0000","last_became_active":"2024-03-24T12:03:19.825134+0000","last_became_peered":"2024-03-24T12:03:19.825134+0000","last_unstale":"2024-03-24T12:05:08.190168+0000","last_undegraded":"2024-03-24T12:05:08.190168+0000","last_fullsized":"2024-03-24T12:05:08.190168+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:02.370579+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.370579+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.301086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"296'12535","reported_seq":12708,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:26.586149+0000","last_change":"2024-03-24T12:04:58.062065+0000","last_active":"2024-03-24T12:05:26.586149+0000","last_peered":"2024-03-24T12:05:26.586149+0000","last_clean":"2024-03-24T12:05:26.586149+0000","last_became_active":"2024-03-24T12:02:46.088943+0000","last_became_peered":"2024-03-24T12:02:46.088943+0000","last_unstale":"2024-03-24T12:05:26.586149+0000","last_undegraded":"2024-03-24T12:05:26.586149+0000","last_fullsized":"2024-03-24T12:05:26.586149+0000","mapping_epoch":246,"log_start":"285'8000","ondisk_log_start":"285'8000","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'10426","last_scrub_stamp":"2024-03-24T12:04:58.061867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:58.061867+0000","objects_scrubbed":5636,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:15.860491+0000","scrub_duration":1823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226426880,"num_objects":3455,"num_object_clones":0,"num_object_copies":10365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12535,"num_write_kb":511680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"296'44","reported_seq":588,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:14.441279+0000","last_change":"2024-03-24T12:05:14.441279+0000","last_active":"2024-03-24T12:05:14.441279+0000","last_peered":"2024-03-24T12:05:14.441279+0000","last_clean":"2024-03-24T12:05:14.441279+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:05:14.441279+0000","last_undegraded":"2024-03-24T12:05:14.441279+0000","last_fullsized":"2024-03-24T12:05:14.441279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.441098+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:31.476169+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180551+0000","last_change":"2024-03-24T12:05:00.489674+0000","last_active":"2024-03-24T12:05:08.180551+0000","last_peered":"2024-03-24T12:05:08.180551+0000","last_clean":"2024-03-24T12:05:08.180551+0000","last_became_active":"2024-03-24T11:52:15.146739+0000","last_became_peered":"2024-03-24T11:52:15.146739+0000","last_unstale":"2024-03-24T12:05:08.180551+0000","last_undegraded":"2024-03-24T12:05:08.180551+0000","last_fullsized":"2024-03-24T12:05:08.180551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:00.489543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:05:00.489543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:14.786599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":522,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181163+0000","last_change":"2024-03-24T12:04:48.399451+0000","last_active":"2024-03-24T12:05:08.181163+0000","last_peered":"2024-03-24T12:05:08.181163+0000","last_clean":"2024-03-24T12:05:08.181163+0000","last_became_active":"2024-03-24T11:52:19.379085+0000","last_became_peered":"2024-03-24T11:52:19.379085+0000","last_unstale":"2024-03-24T12:05:08.181163+0000","last_undegraded":"2024-03-24T12:05:08.181163+0000","last_fullsized":"2024-03-24T12:05:08.181163+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:48.399346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:48.399346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:48.716347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:14.128826+0000","last_change":"2024-03-24T12:05:14.128826+0000","last_active":"2024-03-24T12:05:14.128826+0000","last_peered":"2024-03-24T12:05:14.128826+0000","last_clean":"2024-03-24T12:05:14.128826+0000","last_became_active":"2024-03-24T11:52:21.391333+0000","last_became_peered":"2024-03-24T11:52:21.391333+0000","last_unstale":"2024-03-24T12:05:14.128826+0000","last_undegraded":"2024-03-24T12:05:14.128826+0000","last_fullsized":"2024-03-24T12:05:14.128826+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:14.128698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.128698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.394262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"296'12295","reported_seq":12483,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:26.309031+0000","last_change":"2024-03-24T12:04:09.253965+0000","last_active":"2024-03-24T12:05:26.309031+0000","last_peered":"2024-03-24T12:05:26.309031+0000","last_clean":"2024-03-24T12:05:26.309031+0000","last_became_active":"2024-03-24T12:02:46.436511+0000","last_became_peered":"2024-03-24T12:02:46.436511+0000","last_unstale":"2024-03-24T12:05:26.309031+0000","last_undegraded":"2024-03-24T12:05:26.309031+0000","last_fullsized":"2024-03-24T12:05:26.309031+0000","mapping_epoch":246,"log_start":"277'6900","ondisk_log_start":"277'6900","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'6898","last_scrub_stamp":"2024-03-24T12:04:09.253706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:09.253706+0000","objects_scrubbed":6832,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221446144,"num_objects":3379,"num_object_clones":0,"num_object_copies":10137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12295,"num_write_kb":501568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":573,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.187676+0000","last_change":"2024-03-24T12:03:52.852751+0000","last_active":"2024-03-24T12:05:08.187676+0000","last_peered":"2024-03-24T12:05:08.187676+0000","last_clean":"2024-03-24T12:05:08.187676+0000","last_became_active":"2024-03-24T11:52:17.354608+0000","last_became_peered":"2024-03-24T11:52:17.354608+0000","last_unstale":"2024-03-24T12:05:08.187676+0000","last_undegraded":"2024-03-24T12:05:08.187676+0000","last_fullsized":"2024-03-24T12:05:08.187676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:03:52.852641+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.852641+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179807+0000","last_change":"2024-03-24T12:05:05.492513+0000","last_active":"2024-03-24T12:05:08.179807+0000","last_peered":"2024-03-24T12:05:08.179807+0000","last_clean":"2024-03-24T12:05:08.179807+0000","last_became_active":"2024-03-24T11:52:37.789743+0000","last_became_peered":"2024-03-24T11:52:37.789743+0000","last_unstale":"2024-03-24T12:05:08.179807+0000","last_undegraded":"2024-03-24T12:05:08.179807+0000","last_fullsized":"2024-03-24T12:05:08.179807+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:05:05.492318+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.492318+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:27.030210+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186377+0000","last_change":"2024-03-24T12:03:56.825579+0000","last_active":"2024-03-24T12:05:08.186377+0000","last_peered":"2024-03-24T12:05:08.186377+0000","last_clean":"2024-03-24T12:05:08.186377+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:05:08.186377+0000","last_undegraded":"2024-03-24T12:05:08.186377+0000","last_fullsized":"2024-03-24T12:05:08.186377+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.825474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.825474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:13.727177+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.727086+0000","last_change":"2024-03-24T12:05:12.727225+0000","last_active":"2024-03-24T12:05:12.727086+0000","last_peered":"2024-03-24T12:05:12.727086+0000","last_clean":"2024-03-24T12:05:12.727086+0000","last_became_active":"2024-03-24T11:52:21.388247+0000","last_became_peered":"2024-03-24T11:52:21.388247+0000","last_unstale":"2024-03-24T12:05:12.727086+0000","last_undegraded":"2024-03-24T12:05:12.727086+0000","last_fullsized":"2024-03-24T12:05:12.727086+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:12.726872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:12.726872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:37.941465+0000","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":551,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:19.243328+0000","last_change":"2024-03-24T12:05:19.243370+0000","last_active":"2024-03-24T12:05:19.243328+0000","last_peered":"2024-03-24T12:05:19.243328+0000","last_clean":"2024-03-24T12:05:19.243328+0000","last_became_active":"2024-03-24T11:59:53.779445+0000","last_became_peered":"2024-03-24T11:59:53.779445+0000","last_unstale":"2024-03-24T12:05:19.243328+0000","last_undegraded":"2024-03-24T12:05:19.243328+0000","last_fullsized":"2024-03-24T12:05:19.243328+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:19.243242+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.243242+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:33.424504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"296'12195","reported_seq":12306,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:27.548383+0000","last_change":"2024-03-24T12:02:46.091030+0000","last_active":"2024-03-24T12:05:27.548383+0000","last_peered":"2024-03-24T12:05:27.548383+0000","last_clean":"2024-03-24T12:05:27.548383+0000","last_became_active":"2024-03-24T12:02:46.090706+0000","last_became_peered":"2024-03-24T12:02:46.090706+0000","last_unstale":"2024-03-24T12:05:27.548383+0000","last_undegraded":"2024-03-24T12:05:27.548383+0000","last_fullsized":"2024-03-24T12:05:27.548383+0000","mapping_epoch":246,"log_start":"286'8000","ondisk_log_start":"286'8000","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215285760,"num_objects":3285,"num_object_clones":0,"num_object_copies":9855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12195,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":551,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185081+0000","last_change":"2024-03-24T12:05:01.942196+0000","last_active":"2024-03-24T12:05:08.185081+0000","last_peered":"2024-03-24T12:05:08.185081+0000","last_clean":"2024-03-24T12:05:08.185081+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:05:08.185081+0000","last_undegraded":"2024-03-24T12:05:08.185081+0000","last_fullsized":"2024-03-24T12:05:08.185081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:01.942039+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.942039+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:24.554745+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181937+0000","last_change":"2024-03-24T12:04:00.667345+0000","last_active":"2024-03-24T12:05:08.181937+0000","last_peered":"2024-03-24T12:05:08.181937+0000","last_clean":"2024-03-24T12:05:08.181937+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:05:08.181937+0000","last_undegraded":"2024-03-24T12:05:08.181937+0000","last_fullsized":"2024-03-24T12:05:08.181937+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:04:00.667221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:09.226148+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":689,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.354266+0000","last_change":"2024-03-24T12:05:06.479237+0000","last_active":"2024-03-24T12:05:20.354266+0000","last_peered":"2024-03-24T12:05:20.354266+0000","last_clean":"2024-03-24T12:05:20.354266+0000","last_became_active":"2024-03-24T11:52:19.383062+0000","last_became_peered":"2024-03-24T11:52:19.383062+0000","last_unstale":"2024-03-24T12:05:20.354266+0000","last_undegraded":"2024-03-24T12:05:20.354266+0000","last_fullsized":"2024-03-24T12:05:20.354266+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:06.478884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.478884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:34.542656+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":509,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186985+0000","last_change":"2024-03-24T12:03:47.910225+0000","last_active":"2024-03-24T12:05:08.186985+0000","last_peered":"2024-03-24T12:05:08.186985+0000","last_clean":"2024-03-24T12:05:08.186985+0000","last_became_active":"2024-03-24T11:52:21.392758+0000","last_became_peered":"2024-03-24T11:52:21.392758+0000","last_unstale":"2024-03-24T12:05:08.186985+0000","last_undegraded":"2024-03-24T12:05:08.186985+0000","last_fullsized":"2024-03-24T12:05:08.186985+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:03:47.909970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:47.909970+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":804,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.142691+0000","last_change":"2024-03-24T12:05:16.142814+0000","last_active":"2024-03-24T12:05:16.142691+0000","last_peered":"2024-03-24T12:05:16.142691+0000","last_clean":"2024-03-24T12:05:16.142691+0000","last_became_active":"2024-03-24T11:59:53.771452+0000","last_became_peered":"2024-03-24T11:59:53.771452+0000","last_unstale":"2024-03-24T12:05:16.142691+0000","last_undegraded":"2024-03-24T12:05:16.142691+0000","last_fullsized":"2024-03-24T12:05:16.142691+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.142552+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:17.240353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"296'12321","reported_seq":12479,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:27.516413+0000","last_change":"2024-03-24T12:05:23.128313+0000","last_active":"2024-03-24T12:05:27.516413+0000","last_peered":"2024-03-24T12:05:27.516413+0000","last_clean":"2024-03-24T12:05:27.516413+0000","last_became_active":"2024-03-24T12:02:46.436446+0000","last_became_peered":"2024-03-24T12:02:46.436446+0000","last_unstale":"2024-03-24T12:05:27.516413+0000","last_undegraded":"2024-03-24T12:05:27.516413+0000","last_fullsized":"2024-03-24T12:05:27.516413+0000","mapping_epoch":246,"log_start":"286'8100","ondisk_log_start":"286'8100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'12074","last_scrub_stamp":"2024-03-24T12:05:23.128108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:23.128108+0000","objects_scrubbed":3636,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:28.048490+0000","scrub_duration":1310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219873280,"num_objects":3355,"num_object_clones":0,"num_object_copies":10065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12321,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":575,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:17.147921+0000","last_change":"2024-03-24T12:05:17.147975+0000","last_active":"2024-03-24T12:05:17.147921+0000","last_peered":"2024-03-24T12:05:17.147921+0000","last_clean":"2024-03-24T12:05:17.147921+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:05:17.147921+0000","last_undegraded":"2024-03-24T12:05:17.147921+0000","last_fullsized":"2024-03-24T12:05:17.147921+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:05:17.147838+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.147838+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:46.502046+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189835+0000","last_change":"2024-03-24T12:05:04.036530+0000","last_active":"2024-03-24T12:05:08.189835+0000","last_peered":"2024-03-24T12:05:08.189835+0000","last_clean":"2024-03-24T12:05:08.189835+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:05:08.189835+0000","last_undegraded":"2024-03-24T12:05:08.189835+0000","last_fullsized":"2024-03-24T12:05:08.189835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:04.036311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:05:04.036311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:14.102415+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":682,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.354308+0000","last_change":"2024-03-24T12:05:05.860928+0000","last_active":"2024-03-24T12:05:20.354308+0000","last_peered":"2024-03-24T12:05:20.354308+0000","last_clean":"2024-03-24T12:05:20.354308+0000","last_became_active":"2024-03-24T11:52:19.383226+0000","last_became_peered":"2024-03-24T11:52:19.383226+0000","last_unstale":"2024-03-24T12:05:20.354308+0000","last_undegraded":"2024-03-24T12:05:20.354308+0000","last_fullsized":"2024-03-24T12:05:20.354308+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:05.860814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.860814+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:12.200839+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189204+0000","last_change":"2024-03-24T12:04:02.219664+0000","last_active":"2024-03-24T12:05:08.189204+0000","last_peered":"2024-03-24T12:05:08.189204+0000","last_clean":"2024-03-24T12:05:08.189204+0000","last_became_active":"2024-03-24T11:52:21.388631+0000","last_became_peered":"2024-03-24T11:52:21.388631+0000","last_unstale":"2024-03-24T12:05:08.189204+0000","last_undegraded":"2024-03-24T12:05:08.189204+0000","last_fullsized":"2024-03-24T12:05:08.189204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.219541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.219541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:11.682190+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":551,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:11.242652+0000","last_change":"2024-03-24T12:05:11.242695+0000","last_active":"2024-03-24T12:05:11.242652+0000","last_peered":"2024-03-24T12:05:11.242652+0000","last_clean":"2024-03-24T12:05:11.242652+0000","last_became_active":"2024-03-24T11:59:53.771384+0000","last_became_peered":"2024-03-24T11:59:53.771384+0000","last_unstale":"2024-03-24T12:05:11.242652+0000","last_undegraded":"2024-03-24T12:05:11.242652+0000","last_fullsized":"2024-03-24T12:05:11.242652+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:11.242568+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.242568+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.583012+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'130","reported_seq":393,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.649205+0000","last_change":"2024-03-24T12:05:20.649244+0000","last_active":"2024-03-24T12:05:20.649205+0000","last_peered":"2024-03-24T12:05:20.649205+0000","last_clean":"2024-03-24T12:05:20.649205+0000","last_became_active":"2024-03-24T12:03:19.826030+0000","last_became_peered":"2024-03-24T12:03:19.826030+0000","last_unstale":"2024-03-24T12:05:20.649205+0000","last_undegraded":"2024-03-24T12:05:20.649205+0000","last_fullsized":"2024-03-24T12:05:20.649205+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:20.649114+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:20.649114+0000","last_clean_scrub_stamp":"2024-03-24T12:05:20.649114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:50.501737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3","version":"296'12895","reported_seq":13105,"reported_epoch":296,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T12:05:26.499255+0000","last_change":"2024-03-24T12:05:23.206089+0000","last_active":"2024-03-24T12:05:26.499255+0000","last_peered":"2024-03-24T12:05:26.499255+0000","last_clean":"2024-03-24T12:05:26.499255+0000","last_became_active":"2024-03-24T12:02:46.436507+0000","last_became_peered":"2024-03-24T12:02:46.436507+0000","last_unstale":"2024-03-24T12:05:26.499255+0000","last_undegraded":"2024-03-24T12:05:26.499255+0000","last_fullsized":"2024-03-24T12:05:26.499255+0000","mapping_epoch":246,"log_start":"289'8700","ondisk_log_start":"289'8700","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'6826","last_scrub_stamp":"2024-03-24T12:04:10.081256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:10.081256+0000","objects_scrubbed":711,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":1954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216006656,"num_objects":3296,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":544,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186702+0000","last_change":"2024-03-24T12:03:09.967801+0000","last_active":"2024-03-24T12:05:08.186702+0000","last_peered":"2024-03-24T12:05:08.186702+0000","last_clean":"2024-03-24T12:05:08.186702+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:05:08.186702+0000","last_undegraded":"2024-03-24T12:05:08.186702+0000","last_fullsized":"2024-03-24T12:05:08.186702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:03:09.967712+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:03:09.967712+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178918+0000","last_change":"2024-03-24T12:04:54.129547+0000","last_active":"2024-03-24T12:05:08.178918+0000","last_peered":"2024-03-24T12:05:08.178918+0000","last_clean":"2024-03-24T12:05:08.178918+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:05:08.178918+0000","last_undegraded":"2024-03-24T12:05:08.178918+0000","last_fullsized":"2024-03-24T12:05:08.178918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.129436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.129436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.954691+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":504,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184181+0000","last_change":"2024-03-24T12:03:12.111480+0000","last_active":"2024-03-24T12:05:08.184181+0000","last_peered":"2024-03-24T12:05:08.184181+0000","last_clean":"2024-03-24T12:05:08.184181+0000","last_became_active":"2024-03-24T11:52:19.388223+0000","last_became_peered":"2024-03-24T11:52:19.388223+0000","last_unstale":"2024-03-24T12:05:08.184181+0000","last_undegraded":"2024-03-24T12:05:08.184181+0000","last_fullsized":"2024-03-24T12:05:08.184181+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:12.111323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:12.111323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188936+0000","last_change":"2024-03-24T12:04:48.633752+0000","last_active":"2024-03-24T12:05:08.188936+0000","last_peered":"2024-03-24T12:05:08.188936+0000","last_clean":"2024-03-24T12:05:08.188936+0000","last_became_active":"2024-03-24T11:52:21.851055+0000","last_became_peered":"2024-03-24T11:52:21.851055+0000","last_unstale":"2024-03-24T12:05:08.188936+0000","last_undegraded":"2024-03-24T12:05:08.188936+0000","last_fullsized":"2024-03-24T12:05:08.188936+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:48.633639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:48.633639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:00.099164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"296'12405","reported_seq":12571,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:27.549478+0000","last_change":"2024-03-24T12:05:07.843040+0000","last_active":"2024-03-24T12:05:27.549478+0000","last_peered":"2024-03-24T12:05:27.549478+0000","last_clean":"2024-03-24T12:05:27.549478+0000","last_became_active":"2024-03-24T12:02:46.087133+0000","last_became_peered":"2024-03-24T12:02:46.087133+0000","last_unstale":"2024-03-24T12:05:27.549478+0000","last_undegraded":"2024-03-24T12:05:27.549478+0000","last_fullsized":"2024-03-24T12:05:27.549478+0000","mapping_epoch":246,"log_start":"287'8200","ondisk_log_start":"287'8200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'11157","last_scrub_stamp":"2024-03-24T12:05:07.842597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:07.842597+0000","objects_scrubbed":4609,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:36.787715+0000","scrub_duration":2120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215023616,"num_objects":3281,"num_object_clones":0,"num_object_copies":9843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":501952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":674,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.354293+0000","last_change":"2024-03-24T12:04:13.607127+0000","last_active":"2024-03-24T12:05:20.354293+0000","last_peered":"2024-03-24T12:05:20.354293+0000","last_clean":"2024-03-24T12:05:20.354293+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:05:20.354293+0000","last_undegraded":"2024-03-24T12:05:20.354293+0000","last_fullsized":"2024-03-24T12:05:20.354293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:04:13.606837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:13.606837+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:33.283948+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":558,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183499+0000","last_change":"2024-03-24T12:02:54.860022+0000","last_active":"2024-03-24T12:05:08.183499+0000","last_peered":"2024-03-24T12:05:08.183499+0000","last_clean":"2024-03-24T12:05:08.183499+0000","last_became_active":"2024-03-24T11:52:17.344326+0000","last_became_peered":"2024-03-24T11:52:17.344326+0000","last_unstale":"2024-03-24T12:05:08.183499+0000","last_undegraded":"2024-03-24T12:05:08.183499+0000","last_fullsized":"2024-03-24T12:05:08.183499+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:02:54.859867+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:02:54.859867+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":528,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190480+0000","last_change":"2024-03-24T12:05:04.423091+0000","last_active":"2024-03-24T12:05:08.190480+0000","last_peered":"2024-03-24T12:05:08.190480+0000","last_clean":"2024-03-24T12:05:08.190480+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:05:08.190480+0000","last_undegraded":"2024-03-24T12:05:08.190480+0000","last_fullsized":"2024-03-24T12:05:08.190480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:04.422989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:05:04.422989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.438929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180240+0000","last_change":"2024-03-24T12:04:59.065642+0000","last_active":"2024-03-24T12:05:08.180240+0000","last_peered":"2024-03-24T12:05:08.180240+0000","last_clean":"2024-03-24T12:05:08.180240+0000","last_became_active":"2024-03-24T11:52:21.854623+0000","last_became_peered":"2024-03-24T11:52:21.854623+0000","last_unstale":"2024-03-24T12:05:08.180240+0000","last_undegraded":"2024-03-24T12:05:08.180240+0000","last_fullsized":"2024-03-24T12:05:08.180240+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.065539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.065539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:07.880065+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":548,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190159+0000","last_change":"2024-03-24T12:03:15.627195+0000","last_active":"2024-03-24T12:05:08.190159+0000","last_peered":"2024-03-24T12:05:08.190159+0000","last_clean":"2024-03-24T12:05:08.190159+0000","last_became_active":"2024-03-24T12:03:15.302146+0000","last_became_peered":"2024-03-24T12:03:15.302146+0000","last_unstale":"2024-03-24T12:05:08.190159+0000","last_undegraded":"2024-03-24T12:05:08.190159+0000","last_fullsized":"2024-03-24T12:05:08.190159+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:15.627055+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:15.627055+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"296'12394","reported_seq":12584,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:26.471076+0000","last_change":"2024-03-24T12:04:12.206158+0000","last_active":"2024-03-24T12:05:26.471076+0000","last_peered":"2024-03-24T12:05:26.471076+0000","last_clean":"2024-03-24T12:05:26.471076+0000","last_became_active":"2024-03-24T12:02:46.436562+0000","last_became_peered":"2024-03-24T12:02:46.436562+0000","last_unstale":"2024-03-24T12:05:26.471076+0000","last_undegraded":"2024-03-24T12:05:26.471076+0000","last_fullsized":"2024-03-24T12:05:26.471076+0000","mapping_epoch":246,"log_start":"287'8200","ondisk_log_start":"287'8200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7141","last_scrub_stamp":"2024-03-24T12:04:12.205805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.205805+0000","objects_scrubbed":7081,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:30.960946+0000","scrub_duration":2066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224198680,"num_objects":3422,"num_object_clones":0,"num_object_copies":10266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12394,"num_write_kb":506049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":505,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183708+0000","last_change":"2024-03-24T12:03:12.939439+0000","last_active":"2024-03-24T12:05:08.183708+0000","last_peered":"2024-03-24T12:05:08.183708+0000","last_clean":"2024-03-24T12:05:08.183708+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:05:08.183708+0000","last_undegraded":"2024-03-24T12:05:08.183708+0000","last_fullsized":"2024-03-24T12:05:08.183708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:12.939336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:03:12.939336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":569,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178331+0000","last_change":"2024-03-24T12:03:52.775624+0000","last_active":"2024-03-24T12:05:08.178331+0000","last_peered":"2024-03-24T12:05:08.178331+0000","last_clean":"2024-03-24T12:05:08.178331+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:05:08.178331+0000","last_undegraded":"2024-03-24T12:05:08.178331+0000","last_fullsized":"2024-03-24T12:05:08.178331+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:03:52.775356+0000","last_deep_scrub":"195'26","last_deep_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.775356+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":272,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189606+0000","last_change":"2024-03-24T12:04:03.362973+0000","last_active":"2024-03-24T12:05:08.189606+0000","last_peered":"2024-03-24T12:05:08.189606+0000","last_clean":"2024-03-24T12:05:08.189606+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:05:08.189606+0000","last_undegraded":"2024-03-24T12:05:08.189606+0000","last_fullsized":"2024-03-24T12:05:08.189606+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:04:03.362834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:03.362834+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":517,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:22.782697+0000","last_change":"2024-03-24T12:05:22.782723+0000","last_active":"2024-03-24T12:05:22.782697+0000","last_peered":"2024-03-24T12:05:22.782697+0000","last_clean":"2024-03-24T12:05:22.782697+0000","last_became_active":"2024-03-24T11:52:21.385597+0000","last_became_peered":"2024-03-24T11:52:21.385597+0000","last_unstale":"2024-03-24T12:05:22.782697+0000","last_undegraded":"2024-03-24T12:05:22.782697+0000","last_fullsized":"2024-03-24T12:05:22.782697+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:05:22.782610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:31.980846+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":548,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.187038+0000","last_change":"2024-03-24T12:03:16.889700+0000","last_active":"2024-03-24T12:05:08.187038+0000","last_peered":"2024-03-24T12:05:08.187038+0000","last_clean":"2024-03-24T12:05:08.187038+0000","last_became_active":"2024-03-24T12:03:15.292363+0000","last_became_peered":"2024-03-24T12:03:15.292363+0000","last_unstale":"2024-03-24T12:05:08.187038+0000","last_undegraded":"2024-03-24T12:05:08.187038+0000","last_fullsized":"2024-03-24T12:05:08.187038+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:16.889509+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:16.889509+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"296'12473","reported_seq":12663,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:26.502111+0000","last_change":"2024-03-24T12:04:11.225904+0000","last_active":"2024-03-24T12:05:26.502111+0000","last_peered":"2024-03-24T12:05:26.502111+0000","last_clean":"2024-03-24T12:05:26.502111+0000","last_became_active":"2024-03-24T12:02:46.438121+0000","last_became_peered":"2024-03-24T12:02:46.438121+0000","last_unstale":"2024-03-24T12:05:26.502111+0000","last_undegraded":"2024-03-24T12:05:26.502111+0000","last_fullsized":"2024-03-24T12:05:26.502111+0000","mapping_epoch":246,"log_start":"287'8300","ondisk_log_start":"287'8300","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'7114","last_scrub_stamp":"2024-03-24T12:04:11.225516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:11.225516+0000","objects_scrubbed":7067,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12473,"num_write_kb":506432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:24.576232+0000","last_change":"2024-03-24T12:05:24.576269+0000","last_active":"2024-03-24T12:05:24.576232+0000","last_peered":"2024-03-24T12:05:24.576232+0000","last_clean":"2024-03-24T12:05:24.576232+0000","last_became_active":"2024-03-24T11:52:19.375263+0000","last_became_peered":"2024-03-24T11:52:19.375263+0000","last_unstale":"2024-03-24T12:05:24.576232+0000","last_undegraded":"2024-03-24T12:05:24.576232+0000","last_fullsized":"2024-03-24T12:05:24.576232+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:24.576146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:05:24.576146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:47.369433+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"241'29","reported_seq":561,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:09.620392+0000","last_change":"2024-03-24T12:05:09.620452+0000","last_active":"2024-03-24T12:05:09.620392+0000","last_peered":"2024-03-24T12:05:09.620392+0000","last_clean":"2024-03-24T12:05:09.620392+0000","last_became_active":"2024-03-24T11:52:17.423539+0000","last_became_peered":"2024-03-24T11:52:17.423539+0000","last_unstale":"2024-03-24T12:05:09.620392+0000","last_undegraded":"2024-03-24T12:05:09.620392+0000","last_fullsized":"2024-03-24T12:05:09.620392+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-24T12:05:09.620298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.620298+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:27.220062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179954+0000","last_change":"2024-03-24T12:03:58.207492+0000","last_active":"2024-03-24T12:05:08.179954+0000","last_peered":"2024-03-24T12:05:08.179954+0000","last_clean":"2024-03-24T12:05:08.179954+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:05:08.179954+0000","last_undegraded":"2024-03-24T12:05:08.179954+0000","last_fullsized":"2024-03-24T12:05:08.179954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.207386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.207386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:25.672325+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188403+0000","last_change":"2024-03-24T12:03:52.366880+0000","last_active":"2024-03-24T12:05:08.188403+0000","last_peered":"2024-03-24T12:05:08.188403+0000","last_clean":"2024-03-24T12:05:08.188403+0000","last_became_active":"2024-03-24T11:59:19.096322+0000","last_became_peered":"2024-03-24T11:59:19.096322+0000","last_unstale":"2024-03-24T12:05:08.188403+0000","last_undegraded":"2024-03-24T12:05:08.188403+0000","last_fullsized":"2024-03-24T12:05:08.188403+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.366551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.366551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":570,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190760+0000","last_change":"2024-03-24T12:04:56.364590+0000","last_active":"2024-03-24T12:05:08.190760+0000","last_peered":"2024-03-24T12:05:08.190760+0000","last_clean":"2024-03-24T12:05:08.190760+0000","last_became_active":"2024-03-24T12:03:15.295100+0000","last_became_peered":"2024-03-24T12:03:15.295100+0000","last_unstale":"2024-03-24T12:05:08.190760+0000","last_undegraded":"2024-03-24T12:05:08.190760+0000","last_fullsized":"2024-03-24T12:05:08.190760+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:56.364427+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:56.364427+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:03.226918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"296'16719","reported_seq":17163,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:27.934517+0000","last_change":"2024-03-24T12:04:53.459598+0000","last_active":"2024-03-24T12:05:27.934517+0000","last_peered":"2024-03-24T12:05:27.934517+0000","last_clean":"2024-03-24T12:05:27.934517+0000","last_became_active":"2024-03-24T12:02:46.087237+0000","last_became_peered":"2024-03-24T12:02:46.087237+0000","last_unstale":"2024-03-24T12:05:27.934517+0000","last_undegraded":"2024-03-24T12:05:27.934517+0000","last_fullsized":"2024-03-24T12:05:27.934517+0000","mapping_epoch":246,"log_start":"284'11401","ondisk_log_start":"284'11401","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14298","last_scrub_stamp":"2024-03-24T12:04:53.458513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:53.458513+0000","objects_scrubbed":5863,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:23.033128+0000","scrub_duration":1994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220069888,"num_objects":3358,"num_object_clones":0,"num_object_copies":10074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12250,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":505,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184314+0000","last_change":"2024-03-24T12:02:58.920843+0000","last_active":"2024-03-24T12:05:08.184314+0000","last_peered":"2024-03-24T12:05:08.184314+0000","last_clean":"2024-03-24T12:05:08.184314+0000","last_became_active":"2024-03-24T11:52:19.386922+0000","last_became_peered":"2024-03-24T11:52:19.386922+0000","last_unstale":"2024-03-24T12:05:08.184314+0000","last_undegraded":"2024-03-24T12:05:08.184314+0000","last_fullsized":"2024-03-24T12:05:08.184314+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:58.920712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:02:58.920712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":552,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172446+0000","last_change":"2024-03-24T12:03:51.871366+0000","last_active":"2024-03-24T12:05:08.172446+0000","last_peered":"2024-03-24T12:05:08.172446+0000","last_clean":"2024-03-24T12:05:08.172446+0000","last_became_active":"2024-03-24T11:52:17.353154+0000","last_became_peered":"2024-03-24T11:52:17.353154+0000","last_unstale":"2024-03-24T12:05:08.172446+0000","last_undegraded":"2024-03-24T12:05:08.172446+0000","last_fullsized":"2024-03-24T12:05:08.172446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:03:51.871217+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.871217+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:19.921684+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":358,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.174950+0000","last_change":"2024-03-24T12:05:04.552941+0000","last_active":"2024-03-24T12:05:08.174950+0000","last_peered":"2024-03-24T12:05:08.174950+0000","last_clean":"2024-03-24T12:05:08.174950+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:05:08.174950+0000","last_undegraded":"2024-03-24T12:05:08.174950+0000","last_fullsized":"2024-03-24T12:05:08.174950+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:04.552647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:04.552647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:23.348306+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":529,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:27.651085+0000","last_change":"2024-03-24T12:05:27.651120+0000","last_active":"2024-03-24T12:05:27.651085+0000","last_peered":"2024-03-24T12:05:27.651085+0000","last_clean":"2024-03-24T12:05:27.651085+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:05:27.651085+0000","last_undegraded":"2024-03-24T12:05:27.651085+0000","last_fullsized":"2024-03-24T12:05:27.651085+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:05:27.650987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:27.767455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190075+0000","last_change":"2024-03-24T12:05:03.017119+0000","last_active":"2024-03-24T12:05:08.190075+0000","last_peered":"2024-03-24T12:05:08.190075+0000","last_clean":"2024-03-24T12:05:08.190075+0000","last_became_active":"2024-03-24T11:52:21.389425+0000","last_became_peered":"2024-03-24T11:52:21.389425+0000","last_unstale":"2024-03-24T12:05:08.190075+0000","last_undegraded":"2024-03-24T12:05:08.190075+0000","last_fullsized":"2024-03-24T12:05:08.190075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.016987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.016987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:31.930987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":570,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185703+0000","last_change":"2024-03-24T12:04:59.345528+0000","last_active":"2024-03-24T12:05:08.185703+0000","last_peered":"2024-03-24T12:05:08.185703+0000","last_clean":"2024-03-24T12:05:08.185703+0000","last_became_active":"2024-03-24T12:03:15.297422+0000","last_became_peered":"2024-03-24T12:03:15.297422+0000","last_unstale":"2024-03-24T12:05:08.185703+0000","last_undegraded":"2024-03-24T12:05:08.185703+0000","last_fullsized":"2024-03-24T12:05:08.185703+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:59.345388+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.345388+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:06.312683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"296'12402","reported_seq":12558,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:27.529955+0000","last_change":"2024-03-24T12:05:24.012324+0000","last_active":"2024-03-24T12:05:27.529955+0000","last_peered":"2024-03-24T12:05:27.529955+0000","last_clean":"2024-03-24T12:05:27.529955+0000","last_became_active":"2024-03-24T12:02:46.086523+0000","last_became_peered":"2024-03-24T12:02:46.086523+0000","last_unstale":"2024-03-24T12:05:27.529955+0000","last_undegraded":"2024-03-24T12:05:27.529955+0000","last_fullsized":"2024-03-24T12:05:27.529955+0000","mapping_epoch":246,"log_start":"287'8200","ondisk_log_start":"287'8200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'12174","last_scrub_stamp":"2024-03-24T12:05:24.012132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:24.012132+0000","objects_scrubbed":3616,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:36.040802+0000","scrub_duration":1153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220200960,"num_objects":3360,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12402,"num_write_kb":504384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":522,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180298+0000","last_change":"2024-03-24T12:05:06.101252+0000","last_active":"2024-03-24T12:05:08.180298+0000","last_peered":"2024-03-24T12:05:08.180298+0000","last_clean":"2024-03-24T12:05:08.180298+0000","last_became_active":"2024-03-24T11:52:19.390461+0000","last_became_peered":"2024-03-24T11:52:19.390461+0000","last_unstale":"2024-03-24T12:05:08.180298+0000","last_undegraded":"2024-03-24T12:05:08.180298+0000","last_fullsized":"2024-03-24T12:05:08.180298+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:06.100942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.100942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:34.955762+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":731,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.354268+0000","last_change":"2024-03-24T12:04:47.544969+0000","last_active":"2024-03-24T12:05:20.354268+0000","last_peered":"2024-03-24T12:05:20.354268+0000","last_clean":"2024-03-24T12:05:20.354268+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:05:20.354268+0000","last_undegraded":"2024-03-24T12:05:20.354268+0000","last_fullsized":"2024-03-24T12:05:20.354268+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:04:47.544810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:47.544810+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:01.237310+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":802,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178778+0000","last_change":"2024-03-24T12:04:58.016031+0000","last_active":"2024-03-24T12:05:08.178778+0000","last_peered":"2024-03-24T12:05:08.178778+0000","last_clean":"2024-03-24T12:05:08.178778+0000","last_became_active":"2024-03-24T12:03:15.298475+0000","last_became_peered":"2024-03-24T12:03:15.298475+0000","last_unstale":"2024-03-24T12:05:08.178778+0000","last_undegraded":"2024-03-24T12:05:08.178778+0000","last_fullsized":"2024-03-24T12:05:08.178778+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:58.015904+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:03:16.845832+0000","last_clean_scrub_stamp":"2024-03-24T12:04:58.015904+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.816330+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183757+0000","last_change":"2024-03-24T12:03:13.102194+0000","last_active":"2024-03-24T12:05:08.183757+0000","last_peered":"2024-03-24T12:05:08.183757+0000","last_clean":"2024-03-24T12:05:08.183757+0000","last_became_active":"2024-03-24T11:52:15.141198+0000","last_became_peered":"2024-03-24T11:52:15.141198+0000","last_unstale":"2024-03-24T12:05:08.183757+0000","last_undegraded":"2024-03-24T12:05:08.183757+0000","last_fullsized":"2024-03-24T12:05:08.183757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:13.102066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:13.102066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":524,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181838+0000","last_change":"2024-03-24T12:04:08.680127+0000","last_active":"2024-03-24T12:05:08.181838+0000","last_peered":"2024-03-24T12:05:08.181838+0000","last_clean":"2024-03-24T12:05:08.181838+0000","last_became_active":"2024-03-24T11:52:15.139931+0000","last_became_peered":"2024-03-24T11:52:15.139931+0000","last_unstale":"2024-03-24T12:05:08.181838+0000","last_undegraded":"2024-03-24T12:05:08.181838+0000","last_fullsized":"2024-03-24T12:05:08.181838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:04:08.679904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:08.679904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:14.442813+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":509,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.463358+0000","last_change":"2024-03-24T12:05:10.463358+0000","last_active":"2024-03-24T12:05:10.463358+0000","last_peered":"2024-03-24T12:05:10.463358+0000","last_clean":"2024-03-24T12:05:10.463358+0000","last_became_active":"2024-03-24T11:52:21.383367+0000","last_became_peered":"2024-03-24T11:52:21.383367+0000","last_unstale":"2024-03-24T12:05:10.463358+0000","last_undegraded":"2024-03-24T12:05:10.463358+0000","last_fullsized":"2024-03-24T12:05:10.463358+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.463242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.463242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.094315+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"296'12166","reported_seq":12338,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:26.654922+0000","last_change":"2024-03-24T12:05:01.808405+0000","last_active":"2024-03-24T12:05:26.654922+0000","last_peered":"2024-03-24T12:05:26.654922+0000","last_clean":"2024-03-24T12:05:26.654922+0000","last_became_active":"2024-03-24T12:02:46.437687+0000","last_became_peered":"2024-03-24T12:02:46.437687+0000","last_unstale":"2024-03-24T12:05:26.654922+0000","last_undegraded":"2024-03-24T12:05:26.654922+0000","last_fullsized":"2024-03-24T12:05:26.654922+0000","mapping_epoch":246,"log_start":"283'7600","ondisk_log_start":"283'7600","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'10522","last_scrub_stamp":"2024-03-24T12:05:01.808221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.808221+0000","objects_scrubbed":5128,"log_size":4566,"log_dups_size":0,"ondisk_log_size":4566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.285085+0000","scrub_duration":1543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224526336,"num_objects":3426,"num_object_clones":0,"num_object_copies":10278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":498944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":531,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:19.646274+0000","last_change":"2024-03-24T12:05:19.646312+0000","last_active":"2024-03-24T12:05:19.646274+0000","last_peered":"2024-03-24T12:05:19.646274+0000","last_clean":"2024-03-24T12:05:19.646274+0000","last_became_active":"2024-03-24T11:52:19.382938+0000","last_became_peered":"2024-03-24T11:52:19.382938+0000","last_unstale":"2024-03-24T12:05:19.646274+0000","last_undegraded":"2024-03-24T12:05:19.646274+0000","last_fullsized":"2024-03-24T12:05:19.646274+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.646040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:20.629950+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":559,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.194091+0000","last_change":"2024-03-24T12:05:08.084142+0000","last_active":"2024-03-24T12:05:08.194091+0000","last_peered":"2024-03-24T12:05:08.194091+0000","last_clean":"2024-03-24T12:05:08.194091+0000","last_became_active":"2024-03-24T11:52:17.358493+0000","last_became_peered":"2024-03-24T11:52:17.358493+0000","last_unstale":"2024-03-24T12:05:08.194091+0000","last_undegraded":"2024-03-24T12:05:08.194091+0000","last_fullsized":"2024-03-24T12:05:08.194091+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:05:08.083956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:08.083956+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:34.045721+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":539,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:17.610635+0000","last_change":"2024-03-24T12:05:17.610672+0000","last_active":"2024-03-24T12:05:17.610635+0000","last_peered":"2024-03-24T12:05:17.610635+0000","last_clean":"2024-03-24T12:05:17.610635+0000","last_became_active":"2024-03-24T11:52:15.145100+0000","last_became_peered":"2024-03-24T11:52:15.145100+0000","last_unstale":"2024-03-24T12:05:17.610635+0000","last_undegraded":"2024-03-24T12:05:17.610635+0000","last_fullsized":"2024-03-24T12:05:17.610635+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.610545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.610545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:38.551608+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":682,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:25.977206+0000","last_change":"2024-03-24T12:04:54.359217+0000","last_active":"2024-03-24T12:05:25.977206+0000","last_peered":"2024-03-24T12:05:25.977206+0000","last_clean":"2024-03-24T12:05:25.977206+0000","last_became_active":"2024-03-24T11:59:19.095838+0000","last_became_peered":"2024-03-24T11:59:19.095838+0000","last_unstale":"2024-03-24T12:05:25.977206+0000","last_undegraded":"2024-03-24T12:05:25.977206+0000","last_fullsized":"2024-03-24T12:05:25.977206+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:04:54.358848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.358848+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:16.589772+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":374,"num_read_kb":369,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":518,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:11.223467+0000","last_change":"2024-03-24T12:05:11.223509+0000","last_active":"2024-03-24T12:05:11.223467+0000","last_peered":"2024-03-24T12:05:11.223467+0000","last_clean":"2024-03-24T12:05:11.223467+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:05:11.223467+0000","last_undegraded":"2024-03-24T12:05:11.223467+0000","last_fullsized":"2024-03-24T12:05:11.223467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:11.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:28.595270+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":809,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179727+0000","last_change":"2024-03-24T12:04:55.074407+0000","last_active":"2024-03-24T12:05:08.179727+0000","last_peered":"2024-03-24T12:05:08.179727+0000","last_clean":"2024-03-24T12:05:08.179727+0000","last_became_active":"2024-03-24T12:03:15.298007+0000","last_became_peered":"2024-03-24T12:03:15.298007+0000","last_unstale":"2024-03-24T12:05:08.179727+0000","last_undegraded":"2024-03-24T12:05:08.179727+0000","last_fullsized":"2024-03-24T12:05:08.179727+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:55.074332+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:55.074332+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:23.720636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"296'15454","reported_seq":15812,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:27.950296+0000","last_change":"2024-03-24T12:04:56.353651+0000","last_active":"2024-03-24T12:05:27.950296+0000","last_peered":"2024-03-24T12:05:27.950296+0000","last_clean":"2024-03-24T12:05:27.950296+0000","last_became_active":"2024-03-24T12:02:46.090175+0000","last_became_peered":"2024-03-24T12:02:46.090175+0000","last_unstale":"2024-03-24T12:05:27.950296+0000","last_undegraded":"2024-03-24T12:05:27.950296+0000","last_fullsized":"2024-03-24T12:05:27.950296+0000","mapping_epoch":246,"log_start":"284'10101","ondisk_log_start":"284'10101","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'13292","last_scrub_stamp":"2024-03-24T12:04:56.353492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:04:56.353492+0000","objects_scrubbed":5572,"log_size":5353,"log_dups_size":0,"ondisk_log_size":5353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:00.814462+0000","scrub_duration":1967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218824704,"num_objects":3339,"num_object_clones":0,"num_object_copies":10017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12235,"num_write_kb":498368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":506,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.176674+0000","last_change":"2024-03-24T12:03:58.906510+0000","last_active":"2024-03-24T12:05:08.176674+0000","last_peered":"2024-03-24T12:05:08.176674+0000","last_clean":"2024-03-24T12:05:08.176674+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:05:08.176674+0000","last_undegraded":"2024-03-24T12:05:08.176674+0000","last_fullsized":"2024-03-24T12:05:08.176674+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.906274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:16.217543+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"241'41","reported_seq":686,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.354620+0000","last_change":"2024-03-24T12:04:07.131702+0000","last_active":"2024-03-24T12:05:20.354620+0000","last_peered":"2024-03-24T12:05:20.354620+0000","last_clean":"2024-03-24T12:05:20.354620+0000","last_became_active":"2024-03-24T11:52:37.791317+0000","last_became_peered":"2024-03-24T11:52:37.791317+0000","last_unstale":"2024-03-24T12:05:20.354620+0000","last_undegraded":"2024-03-24T12:05:20.354620+0000","last_fullsized":"2024-03-24T12:05:20.354620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-24T12:04:07.131550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:07.131550+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182860+0000","last_change":"2024-03-24T12:04:54.196862+0000","last_active":"2024-03-24T12:05:08.182860+0000","last_peered":"2024-03-24T12:05:08.182860+0000","last_clean":"2024-03-24T12:05:08.182860+0000","last_became_active":"2024-03-24T11:52:15.144527+0000","last_became_peered":"2024-03-24T11:52:15.144527+0000","last_unstale":"2024-03-24T12:05:08.182860+0000","last_undegraded":"2024-03-24T12:05:08.182860+0000","last_fullsized":"2024-03-24T12:05:08.182860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.196749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.196749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:20.902352+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":503,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181334+0000","last_change":"2024-03-24T12:04:02.704225+0000","last_active":"2024-03-24T12:05:08.181334+0000","last_peered":"2024-03-24T12:05:08.181334+0000","last_clean":"2024-03-24T12:05:08.181334+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:05:08.181334+0000","last_undegraded":"2024-03-24T12:05:08.181334+0000","last_fullsized":"2024-03-24T12:05:08.181334+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.704086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.704086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:26.546480+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":564,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184184+0000","last_change":"2024-03-24T12:04:47.152479+0000","last_active":"2024-03-24T12:05:08.184184+0000","last_peered":"2024-03-24T12:05:08.184184+0000","last_clean":"2024-03-24T12:05:08.184184+0000","last_became_active":"2024-03-24T12:03:15.310321+0000","last_became_peered":"2024-03-24T12:03:15.310321+0000","last_unstale":"2024-03-24T12:05:08.184184+0000","last_undegraded":"2024-03-24T12:05:08.184184+0000","last_fullsized":"2024-03-24T12:05:08.184184+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:47.152371+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:47.152371+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:06.868017+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"296'15618","reported_seq":15852,"reported_epoch":296,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T12:05:26.669364+0000","last_change":"2024-03-24T12:05:09.242319+0000","last_active":"2024-03-24T12:05:26.669364+0000","last_peered":"2024-03-24T12:05:26.669364+0000","last_clean":"2024-03-24T12:05:26.669364+0000","last_became_active":"2024-03-24T12:02:46.436594+0000","last_became_peered":"2024-03-24T12:02:46.436594+0000","last_unstale":"2024-03-24T12:05:26.669364+0000","last_undegraded":"2024-03-24T12:05:26.669364+0000","last_fullsized":"2024-03-24T12:05:26.669364+0000","mapping_epoch":246,"log_start":"295'11100","ondisk_log_start":"295'11100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:02:45.061874+0000","objects_scrubbed":3197,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 17s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224329728,"num_objects":3423,"num_object_clones":0,"num_object_copies":10269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12421,"num_write_kb":507008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":681,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.354585+0000","last_change":"2024-03-24T12:05:02.500334+0000","last_active":"2024-03-24T12:05:20.354585+0000","last_peered":"2024-03-24T12:05:20.354585+0000","last_clean":"2024-03-24T12:05:20.354585+0000","last_became_active":"2024-03-24T11:52:19.377062+0000","last_became_peered":"2024-03-24T11:52:19.377062+0000","last_unstale":"2024-03-24T12:05:20.354585+0000","last_undegraded":"2024-03-24T12:05:20.354585+0000","last_fullsized":"2024-03-24T12:05:20.354585+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:02.500062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.500062+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:10.180018+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":561,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.191057+0000","last_change":"2024-03-24T12:05:05.062227+0000","last_active":"2024-03-24T12:05:08.191057+0000","last_peered":"2024-03-24T12:05:08.191057+0000","last_clean":"2024-03-24T12:05:08.191057+0000","last_became_active":"2024-03-24T11:52:17.358123+0000","last_became_peered":"2024-03-24T11:52:17.358123+0000","last_unstale":"2024-03-24T12:05:08.191057+0000","last_undegraded":"2024-03-24T12:05:08.191057+0000","last_fullsized":"2024-03-24T12:05:08.191057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:05.061844+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.061844+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.609038+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.174319+0000","last_change":"2024-03-24T12:04:03.014426+0000","last_active":"2024-03-24T12:05:08.174319+0000","last_peered":"2024-03-24T12:05:08.174319+0000","last_clean":"2024-03-24T12:05:08.174319+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:05:08.174319+0000","last_undegraded":"2024-03-24T12:05:08.174319+0000","last_fullsized":"2024-03-24T12:05:08.174319+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:03.014306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:04:03.014306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:31.228036+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:13.648743+0000","last_change":"2024-03-24T12:05:13.648786+0000","last_active":"2024-03-24T12:05:13.648743+0000","last_peered":"2024-03-24T12:05:13.648743+0000","last_clean":"2024-03-24T12:05:13.648743+0000","last_became_active":"2024-03-24T11:52:21.385932+0000","last_became_peered":"2024-03-24T11:52:21.385932+0000","last_unstale":"2024-03-24T12:05:13.648743+0000","last_undegraded":"2024-03-24T12:05:13.648743+0000","last_fullsized":"2024-03-24T12:05:13.648743+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:13.648498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.648498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:43.373170+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":570,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:23.690409+0000","last_change":"2024-03-24T12:05:23.690448+0000","last_active":"2024-03-24T12:05:23.690409+0000","last_peered":"2024-03-24T12:05:23.690409+0000","last_clean":"2024-03-24T12:05:23.690409+0000","last_became_active":"2024-03-24T12:03:15.298983+0000","last_became_peered":"2024-03-24T12:03:15.298983+0000","last_unstale":"2024-03-24T12:05:23.690409+0000","last_undegraded":"2024-03-24T12:05:23.690409+0000","last_fullsized":"2024-03-24T12:05:23.690409+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:05:23.690315+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:25.761979+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"83'130","reported_seq":561,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181646+0000","last_change":"2024-03-24T12:03:43.603483+0000","last_active":"2024-03-24T12:05:08.181646+0000","last_peered":"2024-03-24T12:05:08.181646+0000","last_clean":"2024-03-24T12:05:08.181646+0000","last_became_active":"2024-03-24T12:03:19.827283+0000","last_became_peered":"2024-03-24T12:03:19.827283+0000","last_unstale":"2024-03-24T12:05:08.181646+0000","last_undegraded":"2024-03-24T12:05:08.181646+0000","last_fullsized":"2024-03-24T12:05:08.181646+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:43.603335+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:43.603335+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"296'12428","reported_seq":12587,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:26.267710+0000","last_change":"2024-03-24T12:05:15.798010+0000","last_active":"2024-03-24T12:05:26.267710+0000","last_peered":"2024-03-24T12:05:26.267710+0000","last_clean":"2024-03-24T12:05:26.267710+0000","last_became_active":"2024-03-24T12:02:46.438409+0000","last_became_peered":"2024-03-24T12:02:46.438409+0000","last_unstale":"2024-03-24T12:05:26.267710+0000","last_undegraded":"2024-03-24T12:05:26.267710+0000","last_fullsized":"2024-03-24T12:05:26.267710+0000","mapping_epoch":246,"log_start":"282'7600","ondisk_log_start":"282'7600","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'11761","last_scrub_stamp":"2024-03-24T12:05:15.797693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:45.061874+0000","last_clean_scrub_stamp":"2024-03-24T12:05:15.797693+0000","objects_scrubbed":4070,"log_size":4828,"log_dups_size":0,"ondisk_log_size":4828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:27.948738+0000","scrub_duration":1009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221380608,"num_objects":3378,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":523,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.607490+0000","last_change":"2024-03-24T12:05:20.607490+0000","last_active":"2024-03-24T12:05:20.607490+0000","last_peered":"2024-03-24T12:05:20.607490+0000","last_clean":"2024-03-24T12:05:20.607490+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:05:20.607490+0000","last_undegraded":"2024-03-24T12:05:20.607490+0000","last_fullsized":"2024-03-24T12:05:20.607490+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:20.607046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:20.607046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:35.391397+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":567,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183657+0000","last_change":"2024-03-24T12:02:57.909944+0000","last_active":"2024-03-24T12:05:08.183657+0000","last_peered":"2024-03-24T12:05:08.183657+0000","last_clean":"2024-03-24T12:05:08.183657+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:05:08.183657+0000","last_undegraded":"2024-03-24T12:05:08.183657+0000","last_fullsized":"2024-03-24T12:05:08.183657+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:02:57.909767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:02:57.909767+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":532,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186851+0000","last_change":"2024-03-24T12:03:53.808708+0000","last_active":"2024-03-24T12:05:08.186851+0000","last_peered":"2024-03-24T12:05:08.186851+0000","last_clean":"2024-03-24T12:05:08.186851+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:05:08.186851+0000","last_undegraded":"2024-03-24T12:05:08.186851+0000","last_fullsized":"2024-03-24T12:05:08.186851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:03:53.808558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:53.808558+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190535+0000","last_change":"2024-03-24T12:04:55.858996+0000","last_active":"2024-03-24T12:05:08.190535+0000","last_peered":"2024-03-24T12:05:08.190535+0000","last_clean":"2024-03-24T12:05:08.190535+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:05:08.190535+0000","last_undegraded":"2024-03-24T12:05:08.190535+0000","last_fullsized":"2024-03-24T12:05:08.190535+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:55.858730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:04:55.858730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:19.264966+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"83'130","reported_seq":386,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181465+0000","last_change":"2024-03-24T12:03:20.685816+0000","last_active":"2024-03-24T12:05:08.181465+0000","last_peered":"2024-03-24T12:05:08.181465+0000","last_clean":"2024-03-24T12:05:08.181465+0000","last_became_active":"2024-03-24T12:03:19.828385+0000","last_became_peered":"2024-03-24T12:03:19.828385+0000","last_unstale":"2024-03-24T12:05:08.181465+0000","last_undegraded":"2024-03-24T12:05:08.181465+0000","last_fullsized":"2024-03-24T12:05:08.181465+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:20.685628+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:20.685628+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":538,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172511+0000","last_change":"2024-03-24T12:03:56.951891+0000","last_active":"2024-03-24T12:05:08.172511+0000","last_peered":"2024-03-24T12:05:08.172511+0000","last_clean":"2024-03-24T12:05:08.172511+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:05:08.172511+0000","last_undegraded":"2024-03-24T12:05:08.172511+0000","last_fullsized":"2024-03-24T12:05:08.172511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:03:56.951745+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.951745+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:10.859861+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:15.567692+0000","last_change":"2024-03-24T12:05:15.567838+0000","last_active":"2024-03-24T12:05:15.567692+0000","last_peered":"2024-03-24T12:05:15.567692+0000","last_clean":"2024-03-24T12:05:15.567692+0000","last_became_active":"2024-03-24T11:52:15.146272+0000","last_became_peered":"2024-03-24T11:52:15.146272+0000","last_unstale":"2024-03-24T12:05:15.567692+0000","last_undegraded":"2024-03-24T12:05:15.567692+0000","last_fullsized":"2024-03-24T12:05:15.567692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:15.567364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:15.567364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:20.030418+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":498,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188402+0000","last_change":"2024-03-24T12:03:11.626954+0000","last_active":"2024-03-24T12:05:08.188402+0000","last_peered":"2024-03-24T12:05:08.188402+0000","last_clean":"2024-03-24T12:05:08.188402+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:05:08.188402+0000","last_undegraded":"2024-03-24T12:05:08.188402+0000","last_fullsized":"2024-03-24T12:05:08.188402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:11.626651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:11.626651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":510,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180278+0000","last_change":"2024-03-24T12:04:49.349330+0000","last_active":"2024-03-24T12:05:08.180278+0000","last_peered":"2024-03-24T12:05:08.180278+0000","last_clean":"2024-03-24T12:05:08.180278+0000","last_became_active":"2024-03-24T11:52:21.385710+0000","last_became_peered":"2024-03-24T11:52:21.385710+0000","last_unstale":"2024-03-24T12:05:08.180278+0000","last_undegraded":"2024-03-24T12:05:08.180278+0000","last_fullsized":"2024-03-24T12:05:08.180278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:49.349208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:49.349208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:01.447429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"83'130","reported_seq":382,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182535+0000","last_change":"2024-03-24T12:03:48.080327+0000","last_active":"2024-03-24T12:05:08.182535+0000","last_peered":"2024-03-24T12:05:08.182535+0000","last_clean":"2024-03-24T12:05:08.182535+0000","last_became_active":"2024-03-24T12:03:19.828310+0000","last_became_peered":"2024-03-24T12:03:19.828310+0000","last_unstale":"2024-03-24T12:05:08.182535+0000","last_undegraded":"2024-03-24T12:05:08.182535+0000","last_fullsized":"2024-03-24T12:05:08.182535+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:48.080209+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.080209+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":528,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188208+0000","last_change":"2024-03-24T12:03:14.587051+0000","last_active":"2024-03-24T12:05:08.188208+0000","last_peered":"2024-03-24T12:05:08.188208+0000","last_clean":"2024-03-24T12:05:08.188208+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:05:08.188208+0000","last_undegraded":"2024-03-24T12:05:08.188208+0000","last_fullsized":"2024-03-24T12:05:08.188208+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:03:14.586799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:14.586799+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":527,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188187+0000","last_change":"2024-03-24T12:04:55.414653+0000","last_active":"2024-03-24T12:05:08.188187+0000","last_peered":"2024-03-24T12:05:08.188187+0000","last_clean":"2024-03-24T12:05:08.188187+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:05:08.188187+0000","last_undegraded":"2024-03-24T12:05:08.188187+0000","last_fullsized":"2024-03-24T12:05:08.188187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:55.414526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:04:55.414526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:59.086401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":504,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183203+0000","last_change":"2024-03-24T12:04:11.242963+0000","last_active":"2024-03-24T12:05:08.183203+0000","last_peered":"2024-03-24T12:05:08.183203+0000","last_clean":"2024-03-24T12:05:08.183203+0000","last_became_active":"2024-03-24T11:52:19.379431+0000","last_became_peered":"2024-03-24T11:52:19.379431+0000","last_unstale":"2024-03-24T12:05:08.183203+0000","last_undegraded":"2024-03-24T12:05:08.183203+0000","last_fullsized":"2024-03-24T12:05:08.183203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:11.242845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:11.242845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:18.573926+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:17.165139+0000","last_change":"2024-03-24T12:05:17.165172+0000","last_active":"2024-03-24T12:05:17.165139+0000","last_peered":"2024-03-24T12:05:17.165139+0000","last_clean":"2024-03-24T12:05:17.165139+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:05:17.165139+0000","last_undegraded":"2024-03-24T12:05:17.165139+0000","last_fullsized":"2024-03-24T12:05:17.165139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.165061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:33.370529+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"83'130","reported_seq":405,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:21.650978+0000","last_change":"2024-03-24T12:05:21.651027+0000","last_active":"2024-03-24T12:05:21.650978+0000","last_peered":"2024-03-24T12:05:21.650978+0000","last_clean":"2024-03-24T12:05:21.650978+0000","last_became_active":"2024-03-24T12:03:19.827207+0000","last_became_peered":"2024-03-24T12:03:19.827207+0000","last_unstale":"2024-03-24T12:05:21.650978+0000","last_undegraded":"2024-03-24T12:05:21.650978+0000","last_fullsized":"2024-03-24T12:05:21.650978+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:21.650437+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:21.650437+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:43.525952+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":545,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189953+0000","last_change":"2024-03-24T12:03:46.711296+0000","last_active":"2024-03-24T12:05:08.189953+0000","last_peered":"2024-03-24T12:05:08.189953+0000","last_clean":"2024-03-24T12:05:08.189953+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:05:08.189953+0000","last_undegraded":"2024-03-24T12:05:08.189953+0000","last_fullsized":"2024-03-24T12:05:08.189953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:03:46.710805+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:03:46.710805+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182828+0000","last_change":"2024-03-24T12:04:59.284655+0000","last_active":"2024-03-24T12:05:08.182828+0000","last_peered":"2024-03-24T12:05:08.182828+0000","last_clean":"2024-03-24T12:05:08.182828+0000","last_became_active":"2024-03-24T11:52:15.138090+0000","last_became_peered":"2024-03-24T11:52:15.138090+0000","last_unstale":"2024-03-24T12:05:08.182828+0000","last_undegraded":"2024-03-24T12:05:08.182828+0000","last_fullsized":"2024-03-24T12:05:08.182828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.284540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.284540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:03.580492+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":679,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.354614+0000","last_change":"2024-03-24T12:03:51.156004+0000","last_active":"2024-03-24T12:05:20.354614+0000","last_peered":"2024-03-24T12:05:20.354614+0000","last_clean":"2024-03-24T12:05:20.354614+0000","last_became_active":"2024-03-24T11:52:19.389943+0000","last_became_peered":"2024-03-24T11:52:19.389943+0000","last_unstale":"2024-03-24T12:05:20.354614+0000","last_undegraded":"2024-03-24T12:05:20.354614+0000","last_fullsized":"2024-03-24T12:05:20.354614+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:03:51.155727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.155727+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":501,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.172351+0000","last_change":"2024-03-24T12:03:57.971802+0000","last_active":"2024-03-24T12:05:08.172351+0000","last_peered":"2024-03-24T12:05:08.172351+0000","last_clean":"2024-03-24T12:05:08.172351+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:05:08.172351+0000","last_undegraded":"2024-03-24T12:05:08.172351+0000","last_fullsized":"2024-03-24T12:05:08.172351+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:57.971678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:57.971678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:09.634713+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"83'130","reported_seq":382,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181334+0000","last_change":"2024-03-24T12:03:19.958725+0000","last_active":"2024-03-24T12:05:08.181334+0000","last_peered":"2024-03-24T12:05:08.181334+0000","last_clean":"2024-03-24T12:05:08.181334+0000","last_became_active":"2024-03-24T12:03:19.823862+0000","last_became_peered":"2024-03-24T12:03:19.823862+0000","last_unstale":"2024-03-24T12:05:08.181334+0000","last_undegraded":"2024-03-24T12:05:08.181334+0000","last_fullsized":"2024-03-24T12:05:08.181334+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:19.958576+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.958576+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":541,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.245873+0000","last_change":"2024-03-24T12:05:16.245975+0000","last_active":"2024-03-24T12:05:16.245873+0000","last_peered":"2024-03-24T12:05:16.245873+0000","last_clean":"2024-03-24T12:05:16.245873+0000","last_became_active":"2024-03-24T11:52:17.424996+0000","last_became_peered":"2024-03-24T11:52:17.424996+0000","last_unstale":"2024-03-24T12:05:16.245873+0000","last_undegraded":"2024-03-24T12:05:16.245873+0000","last_fullsized":"2024-03-24T12:05:16.245873+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:32.914881+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":513,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:10.648374+0000","last_change":"2024-03-24T12:05:10.648412+0000","last_active":"2024-03-24T12:05:10.648374+0000","last_peered":"2024-03-24T12:05:10.648374+0000","last_clean":"2024-03-24T12:05:10.648374+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T12:05:10.648374+0000","last_undegraded":"2024-03-24T12:05:10.648374+0000","last_fullsized":"2024-03-24T12:05:10.648374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.648144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.648144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:40.482552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":673,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.355007+0000","last_change":"2024-03-24T12:03:08.608291+0000","last_active":"2024-03-24T12:05:20.355007+0000","last_peered":"2024-03-24T12:05:20.355007+0000","last_clean":"2024-03-24T12:05:20.355007+0000","last_became_active":"2024-03-24T11:52:19.380221+0000","last_became_peered":"2024-03-24T11:52:19.380221+0000","last_unstale":"2024-03-24T12:05:20.355007+0000","last_undegraded":"2024-03-24T12:05:20.355007+0000","last_fullsized":"2024-03-24T12:05:20.355007+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:03:08.607673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:08.607673+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.187400+0000","last_change":"2024-03-24T12:05:03.041683+0000","last_active":"2024-03-24T12:05:08.187400+0000","last_peered":"2024-03-24T12:05:08.187400+0000","last_clean":"2024-03-24T12:05:08.187400+0000","last_became_active":"2024-03-24T11:52:21.385452+0000","last_became_peered":"2024-03-24T11:52:21.385452+0000","last_unstale":"2024-03-24T12:05:08.187400+0000","last_undegraded":"2024-03-24T12:05:08.187400+0000","last_fullsized":"2024-03-24T12:05:08.187400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.041566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.041566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:05.581449+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":308,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185933+0000","last_change":"2024-03-24T12:03:48.299639+0000","last_active":"2024-03-24T12:05:08.185933+0000","last_peered":"2024-03-24T12:05:08.185933+0000","last_clean":"2024-03-24T12:05:08.185933+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:05:08.185933+0000","last_undegraded":"2024-03-24T12:05:08.185933+0000","last_fullsized":"2024-03-24T12:05:08.185933+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:03:48.299495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.299495+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":527,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189611+0000","last_change":"2024-03-24T12:04:05.334498+0000","last_active":"2024-03-24T12:05:08.189611+0000","last_peered":"2024-03-24T12:05:08.189611+0000","last_clean":"2024-03-24T12:05:08.189611+0000","last_became_active":"2024-03-24T11:52:15.137116+0000","last_became_peered":"2024-03-24T11:52:15.137116+0000","last_unstale":"2024-03-24T12:05:08.189611+0000","last_undegraded":"2024-03-24T12:05:08.189611+0000","last_fullsized":"2024-03-24T12:05:08.189611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:05.334379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:04:05.334379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:15.371303+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":506,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190401+0000","last_change":"2024-03-24T12:03:58.669666+0000","last_active":"2024-03-24T12:05:08.190401+0000","last_peered":"2024-03-24T12:05:08.190401+0000","last_clean":"2024-03-24T12:05:08.190401+0000","last_became_active":"2024-03-24T11:52:19.377973+0000","last_became_peered":"2024-03-24T11:52:19.377973+0000","last_unstale":"2024-03-24T12:05:08.190401+0000","last_undegraded":"2024-03-24T12:05:08.190401+0000","last_fullsized":"2024-03-24T12:05:08.190401+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.669210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.669210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:25.766145+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":509,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:22.691361+0000","last_change":"2024-03-24T12:05:22.691397+0000","last_active":"2024-03-24T12:05:22.691361+0000","last_peered":"2024-03-24T12:05:22.691361+0000","last_clean":"2024-03-24T12:05:22.691361+0000","last_became_active":"2024-03-24T11:52:21.391019+0000","last_became_peered":"2024-03-24T11:52:21.391019+0000","last_unstale":"2024-03-24T12:05:22.691361+0000","last_undegraded":"2024-03-24T12:05:22.691361+0000","last_fullsized":"2024-03-24T12:05:22.691361+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:22.691279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T12:05:22.691279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:36.222464+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":539,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184731+0000","last_change":"2024-03-24T12:03:17.860265+0000","last_active":"2024-03-24T12:05:08.184731+0000","last_peered":"2024-03-24T12:05:08.184731+0000","last_clean":"2024-03-24T12:05:08.184731+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:05:08.184731+0000","last_undegraded":"2024-03-24T12:05:08.184731+0000","last_fullsized":"2024-03-24T12:05:08.184731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:03:17.860114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:17.860114+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":537,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:13.110283+0000","last_change":"2024-03-24T12:05:13.110324+0000","last_active":"2024-03-24T12:05:13.110283+0000","last_peered":"2024-03-24T12:05:13.110283+0000","last_clean":"2024-03-24T12:05:13.110283+0000","last_became_active":"2024-03-24T11:52:15.138271+0000","last_became_peered":"2024-03-24T11:52:15.138271+0000","last_unstale":"2024-03-24T12:05:13.110283+0000","last_undegraded":"2024-03-24T12:05:13.110283+0000","last_fullsized":"2024-03-24T12:05:13.110283+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:13.110170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.110170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:29.861447+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182976+0000","last_change":"2024-03-24T12:03:10.080967+0000","last_active":"2024-03-24T12:05:08.182976+0000","last_peered":"2024-03-24T12:05:08.182976+0000","last_clean":"2024-03-24T12:05:08.182976+0000","last_became_active":"2024-03-24T11:59:19.099779+0000","last_became_peered":"2024-03-24T11:59:19.099779+0000","last_unstale":"2024-03-24T12:05:08.182976+0000","last_undegraded":"2024-03-24T12:05:08.182976+0000","last_fullsized":"2024-03-24T12:05:08.182976+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:10.080832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:03:10.080832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:25.929227+0000","last_change":"2024-03-24T12:05:25.929271+0000","last_active":"2024-03-24T12:05:25.929227+0000","last_peered":"2024-03-24T12:05:25.929227+0000","last_clean":"2024-03-24T12:05:25.929227+0000","last_became_active":"2024-03-24T11:52:21.381279+0000","last_became_peered":"2024-03-24T11:52:21.381279+0000","last_unstale":"2024-03-24T12:05:25.929227+0000","last_undegraded":"2024-03-24T12:05:25.929227+0000","last_fullsized":"2024-03-24T12:05:25.929227+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:25.928874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.928874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:26.018840+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":532,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:18.756798+0000","last_change":"2024-03-24T12:05:18.756844+0000","last_active":"2024-03-24T12:05:18.756798+0000","last_peered":"2024-03-24T12:05:18.756798+0000","last_clean":"2024-03-24T12:05:18.756798+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:05:18.756798+0000","last_undegraded":"2024-03-24T12:05:18.756798+0000","last_fullsized":"2024-03-24T12:05:18.756798+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:05:18.756720+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:05:18.756720+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:28.841436+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":512,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183399+0000","last_change":"2024-03-24T12:03:18.028601+0000","last_active":"2024-03-24T12:05:08.183399+0000","last_peered":"2024-03-24T12:05:08.183399+0000","last_clean":"2024-03-24T12:05:08.183399+0000","last_became_active":"2024-03-24T11:52:15.142130+0000","last_became_peered":"2024-03-24T11:52:15.142130+0000","last_unstale":"2024-03-24T12:05:08.183399+0000","last_undegraded":"2024-03-24T12:05:08.183399+0000","last_fullsized":"2024-03-24T12:05:08.183399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:18.028433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:18.028433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":519,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188106+0000","last_change":"2024-03-24T12:04:12.475747+0000","last_active":"2024-03-24T12:05:08.188106+0000","last_peered":"2024-03-24T12:05:08.188106+0000","last_clean":"2024-03-24T12:05:08.188106+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:05:08.188106+0000","last_undegraded":"2024-03-24T12:05:08.188106+0000","last_fullsized":"2024-03-24T12:05:08.188106+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.475631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.475631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:23.221135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":509,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179964+0000","last_change":"2024-03-24T12:04:50.160877+0000","last_active":"2024-03-24T12:05:08.179964+0000","last_peered":"2024-03-24T12:05:08.179964+0000","last_clean":"2024-03-24T12:05:08.179964+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:05:08.179964+0000","last_undegraded":"2024-03-24T12:05:08.179964+0000","last_fullsized":"2024-03-24T12:05:08.179964+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:50.160745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:04:50.160745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:18.457289+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"83'130","reported_seq":558,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.180103+0000","last_change":"2024-03-24T12:04:51.411706+0000","last_active":"2024-03-24T12:05:08.180103+0000","last_peered":"2024-03-24T12:05:08.180103+0000","last_clean":"2024-03-24T12:05:08.180103+0000","last_became_active":"2024-03-24T12:03:19.819346+0000","last_became_peered":"2024-03-24T12:03:19.819346+0000","last_unstale":"2024-03-24T12:05:08.180103+0000","last_undegraded":"2024-03-24T12:05:08.180103+0000","last_fullsized":"2024-03-24T12:05:08.180103+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:51.411487+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:51.411487+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:11.653420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":533,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185492+0000","last_change":"2024-03-24T12:03:00.601558+0000","last_active":"2024-03-24T12:05:08.185492+0000","last_peered":"2024-03-24T12:05:08.185492+0000","last_clean":"2024-03-24T12:05:08.185492+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:05:08.185492+0000","last_undegraded":"2024-03-24T12:05:08.185492+0000","last_fullsized":"2024-03-24T12:05:08.185492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:03:00.601428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:00.601428+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":519,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190374+0000","last_change":"2024-03-24T12:04:02.372915+0000","last_active":"2024-03-24T12:05:08.190374+0000","last_peered":"2024-03-24T12:05:08.190374+0000","last_clean":"2024-03-24T12:05:08.190374+0000","last_became_active":"2024-03-24T11:52:15.144820+0000","last_became_peered":"2024-03-24T11:52:15.144820+0000","last_unstale":"2024-03-24T12:05:08.190374+0000","last_undegraded":"2024-03-24T12:05:08.190374+0000","last_fullsized":"2024-03-24T12:05:08.190374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.372798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.372798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:28.536106+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":513,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179124+0000","last_change":"2024-03-24T12:04:12.188145+0000","last_active":"2024-03-24T12:05:08.179124+0000","last_peered":"2024-03-24T12:05:08.179124+0000","last_clean":"2024-03-24T12:05:08.179124+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:05:08.179124+0000","last_undegraded":"2024-03-24T12:05:08.179124+0000","last_fullsized":"2024-03-24T12:05:08.179124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.188028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.188028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:33.103624+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":494,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190657+0000","last_change":"2024-03-24T12:03:05.719881+0000","last_active":"2024-03-24T12:05:08.190657+0000","last_peered":"2024-03-24T12:05:08.190657+0000","last_clean":"2024-03-24T12:05:08.190657+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:05:08.190657+0000","last_undegraded":"2024-03-24T12:05:08.190657+0000","last_fullsized":"2024-03-24T12:05:08.190657+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:05.719621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:05.719621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":553,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:23.694174+0000","last_change":"2024-03-24T12:05:23.694227+0000","last_active":"2024-03-24T12:05:23.694174+0000","last_peered":"2024-03-24T12:05:23.694174+0000","last_clean":"2024-03-24T12:05:23.694174+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:05:23.694174+0000","last_undegraded":"2024-03-24T12:05:23.694174+0000","last_fullsized":"2024-03-24T12:05:23.694174+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:23.694067+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:05:23.694067+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:44.905208+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182258+0000","last_change":"2024-03-24T12:04:12.161106+0000","last_active":"2024-03-24T12:05:08.182258+0000","last_peered":"2024-03-24T12:05:08.182258+0000","last_clean":"2024-03-24T12:05:08.182258+0000","last_became_active":"2024-03-24T11:52:15.138502+0000","last_became_peered":"2024-03-24T11:52:15.138502+0000","last_unstale":"2024-03-24T12:05:08.182258+0000","last_undegraded":"2024-03-24T12:05:08.182258+0000","last_fullsized":"2024-03-24T12:05:08.182258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.160982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.160982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:18.654795+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":683,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.354260+0000","last_change":"2024-03-24T12:03:56.395856+0000","last_active":"2024-03-24T12:05:20.354260+0000","last_peered":"2024-03-24T12:05:20.354260+0000","last_clean":"2024-03-24T12:05:20.354260+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:05:20.354260+0000","last_undegraded":"2024-03-24T12:05:20.354260+0000","last_fullsized":"2024-03-24T12:05:20.354260+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.395650+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":527,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:20.119725+0000","last_change":"2024-03-24T12:05:20.119768+0000","last_active":"2024-03-24T12:05:20.119725+0000","last_peered":"2024-03-24T12:05:20.119725+0000","last_clean":"2024-03-24T12:05:20.119725+0000","last_became_active":"2024-03-24T11:52:21.390017+0000","last_became_peered":"2024-03-24T11:52:21.390017+0000","last_unstale":"2024-03-24T12:05:20.119725+0000","last_undegraded":"2024-03-24T12:05:20.119725+0000","last_fullsized":"2024-03-24T12:05:20.119725+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:05:20.119650+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:20.119650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:41.795539+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":563,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:25.602988+0000","last_change":"2024-03-24T12:05:25.603056+0000","last_active":"2024-03-24T12:05:25.602988+0000","last_peered":"2024-03-24T12:05:25.602988+0000","last_clean":"2024-03-24T12:05:25.602988+0000","last_became_active":"2024-03-24T11:52:17.358846+0000","last_became_peered":"2024-03-24T11:52:17.358846+0000","last_unstale":"2024-03-24T12:05:25.602988+0000","last_undegraded":"2024-03-24T12:05:25.602988+0000","last_fullsized":"2024-03-24T12:05:25.602988+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:05:25.602806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.602806+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:29.083088+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":507,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185136+0000","last_change":"2024-03-24T12:03:09.599073+0000","last_active":"2024-03-24T12:05:08.185136+0000","last_peered":"2024-03-24T12:05:08.185136+0000","last_clean":"2024-03-24T12:05:08.185136+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:05:08.185136+0000","last_undegraded":"2024-03-24T12:05:08.185136+0000","last_fullsized":"2024-03-24T12:05:08.185136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:03:09.598786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:09.598786+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":514,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:16.684820+0000","last_change":"2024-03-24T12:05:16.684854+0000","last_active":"2024-03-24T12:05:16.684820+0000","last_peered":"2024-03-24T12:05:16.684820+0000","last_clean":"2024-03-24T12:05:16.684820+0000","last_became_active":"2024-03-24T11:52:19.383854+0000","last_became_peered":"2024-03-24T11:52:19.383854+0000","last_unstale":"2024-03-24T12:05:16.684820+0000","last_undegraded":"2024-03-24T12:05:16.684820+0000","last_fullsized":"2024-03-24T12:05:16.684820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:16.684742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.684742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:35.877773+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.189208+0000","last_change":"2024-03-24T12:04:14.188719+0000","last_active":"2024-03-24T12:05:08.189208+0000","last_peered":"2024-03-24T12:05:08.189208+0000","last_clean":"2024-03-24T12:05:08.189208+0000","last_became_active":"2024-03-24T11:59:19.098557+0000","last_became_peered":"2024-03-24T11:59:19.098557+0000","last_unstale":"2024-03-24T12:05:08.189208+0000","last_undegraded":"2024-03-24T12:05:08.189208+0000","last_fullsized":"2024-03-24T12:05:08.189208+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:04:14.188600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:33.242073+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":576,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:14.526556+0000","last_change":"2024-03-24T12:05:14.526595+0000","last_active":"2024-03-24T12:05:14.526556+0000","last_peered":"2024-03-24T12:05:14.526556+0000","last_clean":"2024-03-24T12:05:14.526556+0000","last_became_active":"2024-03-24T11:52:17.358906+0000","last_became_peered":"2024-03-24T11:52:17.358906+0000","last_unstale":"2024-03-24T12:05:14.526556+0000","last_undegraded":"2024-03-24T12:05:14.526556+0000","last_fullsized":"2024-03-24T12:05:14.526556+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:05:14.526488+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.526488+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:39.548612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":527,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:21.571028+0000","last_change":"2024-03-24T12:05:21.571066+0000","last_active":"2024-03-24T12:05:21.571028+0000","last_peered":"2024-03-24T12:05:21.571028+0000","last_clean":"2024-03-24T12:05:21.571028+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:05:21.571028+0000","last_undegraded":"2024-03-24T12:05:21.571028+0000","last_fullsized":"2024-03-24T12:05:21.571028+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:05:21.570940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:23.609689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":273,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.187896+0000","last_change":"2024-03-24T12:04:57.823173+0000","last_active":"2024-03-24T12:05:08.187896+0000","last_peered":"2024-03-24T12:05:08.187896+0000","last_clean":"2024-03-24T12:05:08.187896+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:05:08.187896+0000","last_undegraded":"2024-03-24T12:05:08.187896+0000","last_fullsized":"2024-03-24T12:05:08.187896+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:57.823045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:57.823045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.912865+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":512,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186872+0000","last_change":"2024-03-24T12:04:53.809154+0000","last_active":"2024-03-24T12:05:08.186872+0000","last_peered":"2024-03-24T12:05:08.186872+0000","last_clean":"2024-03-24T12:05:08.186872+0000","last_became_active":"2024-03-24T11:52:21.851101+0000","last_became_peered":"2024-03-24T11:52:21.851101+0000","last_unstale":"2024-03-24T12:05:08.186872+0000","last_undegraded":"2024-03-24T12:05:08.186872+0000","last_fullsized":"2024-03-24T12:05:08.186872+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:53.809021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:53.809021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:09.097308+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":565,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.178475+0000","last_change":"2024-03-24T12:05:01.677576+0000","last_active":"2024-03-24T12:05:08.178475+0000","last_peered":"2024-03-24T12:05:08.178475+0000","last_clean":"2024-03-24T12:05:08.178475+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:05:08.178475+0000","last_undegraded":"2024-03-24T12:05:08.178475+0000","last_fullsized":"2024-03-24T12:05:08.178475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:05:01.675875+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.675875+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:03.534752+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188844+0000","last_change":"2024-03-24T12:03:51.776115+0000","last_active":"2024-03-24T12:05:08.188844+0000","last_peered":"2024-03-24T12:05:08.188844+0000","last_clean":"2024-03-24T12:05:08.188844+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:05:08.188844+0000","last_undegraded":"2024-03-24T12:05:08.188844+0000","last_fullsized":"2024-03-24T12:05:08.188844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:03:51.775978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.775978+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:15.993859+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":504,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.186318+0000","last_change":"2024-03-24T12:03:07.915966+0000","last_active":"2024-03-24T12:05:08.186318+0000","last_peered":"2024-03-24T12:05:08.186318+0000","last_clean":"2024-03-24T12:05:08.186318+0000","last_became_active":"2024-03-24T11:52:19.382369+0000","last_became_peered":"2024-03-24T11:52:19.382369+0000","last_unstale":"2024-03-24T12:05:08.186318+0000","last_undegraded":"2024-03-24T12:05:08.186318+0000","last_fullsized":"2024-03-24T12:05:08.186318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:07.915894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:07.915894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":503,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.188104+0000","last_change":"2024-03-24T12:03:19.844533+0000","last_active":"2024-03-24T12:05:08.188104+0000","last_peered":"2024-03-24T12:05:08.188104+0000","last_clean":"2024-03-24T12:05:08.188104+0000","last_became_active":"2024-03-24T11:52:21.851015+0000","last_became_peered":"2024-03-24T11:52:21.851015+0000","last_unstale":"2024-03-24T12:05:08.188104+0000","last_undegraded":"2024-03-24T12:05:08.188104+0000","last_fullsized":"2024-03-24T12:05:08.188104+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:19.844413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.844413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":541,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184015+0000","last_change":"2024-03-24T12:03:44.894651+0000","last_active":"2024-03-24T12:05:08.184015+0000","last_peered":"2024-03-24T12:05:08.184015+0000","last_clean":"2024-03-24T12:05:08.184015+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:05:08.184015+0000","last_undegraded":"2024-03-24T12:05:08.184015+0000","last_fullsized":"2024-03-24T12:05:08.184015+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:03:44.894523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:44.894523+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":532,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:09.177593+0000","last_change":"2024-03-24T12:05:09.177628+0000","last_active":"2024-03-24T12:05:09.177593+0000","last_peered":"2024-03-24T12:05:09.177593+0000","last_clean":"2024-03-24T12:05:09.177593+0000","last_became_active":"2024-03-24T11:52:15.144098+0000","last_became_peered":"2024-03-24T11:52:15.144098+0000","last_unstale":"2024-03-24T12:05:09.177593+0000","last_undegraded":"2024-03-24T12:05:09.177593+0000","last_fullsized":"2024-03-24T12:05:09.177593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.177509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:30.995931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":504,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183677+0000","last_change":"2024-03-24T12:03:52.099674+0000","last_active":"2024-03-24T12:05:08.183677+0000","last_peered":"2024-03-24T12:05:08.183677+0000","last_clean":"2024-03-24T12:05:08.183677+0000","last_became_active":"2024-03-24T11:52:19.381056+0000","last_became_peered":"2024-03-24T11:52:19.381056+0000","last_unstale":"2024-03-24T12:05:08.183677+0000","last_undegraded":"2024-03-24T12:05:08.183677+0000","last_fullsized":"2024-03-24T12:05:08.183677+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.099551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.099551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":501,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:18.690524+0000","last_change":"2024-03-24T12:05:18.690563+0000","last_active":"2024-03-24T12:05:18.690524+0000","last_peered":"2024-03-24T12:05:18.690524+0000","last_clean":"2024-03-24T12:05:18.690524+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:05:18.690524+0000","last_undegraded":"2024-03-24T12:05:18.690524+0000","last_fullsized":"2024-03-24T12:05:18.690524+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:18.690224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:05:18.690224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:42.629606+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":531,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184891+0000","last_change":"2024-03-24T12:03:49.858697+0000","last_active":"2024-03-24T12:05:08.184891+0000","last_peered":"2024-03-24T12:05:08.184891+0000","last_clean":"2024-03-24T12:05:08.184891+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:05:08.184891+0000","last_undegraded":"2024-03-24T12:05:08.184891+0000","last_fullsized":"2024-03-24T12:05:08.184891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:03:49.858566+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:03:49.858566+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":529,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179432+0000","last_change":"2024-03-24T12:04:52.100350+0000","last_active":"2024-03-24T12:05:08.179432+0000","last_peered":"2024-03-24T12:05:08.179432+0000","last_clean":"2024-03-24T12:05:08.179432+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:05:08.179432+0000","last_undegraded":"2024-03-24T12:05:08.179432+0000","last_fullsized":"2024-03-24T12:05:08.179432+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:52.100251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:04:52.100251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:06.420776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.179498+0000","last_change":"2024-03-24T12:05:05.120365+0000","last_active":"2024-03-24T12:05:08.179498+0000","last_peered":"2024-03-24T12:05:08.179498+0000","last_clean":"2024-03-24T12:05:08.179498+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:05:08.179498+0000","last_undegraded":"2024-03-24T12:05:08.179498+0000","last_fullsized":"2024-03-24T12:05:08.179498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:05.120099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.120099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:09.215137+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.184342+0000","last_change":"2024-03-24T12:03:56.118050+0000","last_active":"2024-03-24T12:05:08.184342+0000","last_peered":"2024-03-24T12:05:08.184342+0000","last_clean":"2024-03-24T12:05:08.184342+0000","last_became_active":"2024-03-24T11:52:21.388613+0000","last_became_peered":"2024-03-24T11:52:21.388613+0000","last_unstale":"2024-03-24T12:05:08.184342+0000","last_undegraded":"2024-03-24T12:05:08.184342+0000","last_fullsized":"2024-03-24T12:05:08.184342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.117875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.117875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:15.713771+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":545,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182960+0000","last_change":"2024-03-24T12:05:06.830698+0000","last_active":"2024-03-24T12:05:08.182960+0000","last_peered":"2024-03-24T12:05:08.182960+0000","last_clean":"2024-03-24T12:05:08.182960+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:05:08.182960+0000","last_undegraded":"2024-03-24T12:05:08.182960+0000","last_fullsized":"2024-03-24T12:05:08.182960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:05:06.830559+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.830559+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.321532+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":515,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.181199+0000","last_change":"2024-03-24T12:03:55.732013+0000","last_active":"2024-03-24T12:05:08.181199+0000","last_peered":"2024-03-24T12:05:08.181199+0000","last_clean":"2024-03-24T12:05:08.181199+0000","last_became_active":"2024-03-24T11:52:15.140914+0000","last_became_peered":"2024-03-24T11:52:15.140914+0000","last_unstale":"2024-03-24T12:05:08.181199+0000","last_undegraded":"2024-03-24T12:05:08.181199+0000","last_fullsized":"2024-03-24T12:05:08.181199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:55.731711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:03:55.731711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:12.824844+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.182101+0000","last_change":"2024-03-24T12:04:10.294033+0000","last_active":"2024-03-24T12:05:08.182101+0000","last_peered":"2024-03-24T12:05:08.182101+0000","last_clean":"2024-03-24T12:05:08.182101+0000","last_became_active":"2024-03-24T11:59:19.117244+0000","last_became_peered":"2024-03-24T11:59:19.117244+0000","last_unstale":"2024-03-24T12:05:08.182101+0000","last_undegraded":"2024-03-24T12:05:08.182101+0000","last_fullsized":"2024-03-24T12:05:08.182101+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:10.293805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:10.293805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:39.610672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.183928+0000","last_change":"2024-03-24T12:04:12.242117+0000","last_active":"2024-03-24T12:05:08.183928+0000","last_peered":"2024-03-24T12:05:08.183928+0000","last_clean":"2024-03-24T12:05:08.183928+0000","last_became_active":"2024-03-24T11:52:21.391075+0000","last_became_peered":"2024-03-24T11:52:21.391075+0000","last_unstale":"2024-03-24T12:05:08.183928+0000","last_undegraded":"2024-03-24T12:05:08.183928+0000","last_fullsized":"2024-03-24T12:05:08.183928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.241824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.241824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:05:21.596862+0000","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":560,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.185592+0000","last_change":"2024-03-24T12:04:50.623302+0000","last_active":"2024-03-24T12:05:08.185592+0000","last_peered":"2024-03-24T12:05:08.185592+0000","last_clean":"2024-03-24T12:05:08.185592+0000","last_became_active":"2024-03-24T11:52:17.353845+0000","last_became_peered":"2024-03-24T11:52:17.353845+0000","last_unstale":"2024-03-24T12:05:08.185592+0000","last_undegraded":"2024-03-24T12:05:08.185592+0000","last_fullsized":"2024-03-24T12:05:08.185592+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:04:50.623172+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:04:50.623172+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:15.353509+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":514,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:17.651910+0000","last_change":"2024-03-24T12:05:17.651948+0000","last_active":"2024-03-24T12:05:17.651910+0000","last_peered":"2024-03-24T12:05:17.651910+0000","last_clean":"2024-03-24T12:05:17.651910+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:05:17.651910+0000","last_undegraded":"2024-03-24T12:05:17.651910+0000","last_fullsized":"2024-03-24T12:05:17.651910+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.651826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.651826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:20.532994+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:12.732900+0000","last_change":"2024-03-24T12:05:12.732940+0000","last_active":"2024-03-24T12:05:12.732900+0000","last_peered":"2024-03-24T12:05:12.732900+0000","last_clean":"2024-03-24T12:05:12.732900+0000","last_became_active":"2024-03-24T11:52:19.384423+0000","last_became_peered":"2024-03-24T11:52:19.384423+0000","last_unstale":"2024-03-24T12:05:12.732900+0000","last_undegraded":"2024-03-24T12:05:12.732900+0000","last_fullsized":"2024-03-24T12:05:12.732900+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:05:12.732801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:30.917678+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":502,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-24T12:05:08.190796+0000","last_change":"2024-03-24T12:04:54.805094+0000","last_active":"2024-03-24T12:05:08.190796+0000","last_peered":"2024-03-24T12:05:08.190796+0000","last_clean":"2024-03-24T12:05:08.190796+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:05:08.190796+0000","last_undegraded":"2024-03-24T12:05:08.190796+0000","last_fullsized":"2024-03-24T12:05:08.190796+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.804969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.804969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:06:21.941068+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3525771288,"num_objects":53800,"num_object_clones":0,"num_object_copies":161400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197822,"num_write_kb":8051841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10614222848,"data_stored":10614210632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73716,"ondisk_log_size":73716,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":374,"num_read_kb":369,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":781,"ondisk_log_size":781,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005218,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2451824,"kb_used_data":1956968,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":91293328,"statfs":{"total":95995035648,"available":93484367872,"internally_reserved":0,"allocated":2003935232,"data_stored":2003507458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":506722762},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332265,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043560,"kb_used_data":661152,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92701592,"statfs":{"total":95995035648,"available":94926430208,"internally_reserved":0,"allocated":677019648,"data_stored":676599860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2514424,"kb_used_data":1963824,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":91230728,"statfs":{"total":95995035648,"available":93420265472,"internally_reserved":0,"allocated":2010955776,"data_stored":2010519207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738567,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1351084,"kb_used_data":860388,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92394068,"statfs":{"total":95995035648,"available":94611525632,"internally_reserved":0,"allocated":881037312,"data_stored":880612477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411517,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1102132,"kb_used_data":651564,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92643020,"statfs":{"total":95995035648,"available":94866452480,"internally_reserved":0,"allocated":667201536,"data_stored":666770003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117175,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2305160,"kb_used_data":1724032,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91439992,"statfs":{"total":95995035648,"available":93634551808,"internally_reserved":0,"allocated":1765408768,"data_stored":1764957185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":594868679},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920946,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1034680,"kb_used_data":652272,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92710472,"statfs":{"total":95995035648,"available":94935523328,"internally_reserved":0,"allocated":667926528,"data_stored":667489970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659309,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2523192,"kb_used_data":1986608,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91221960,"statfs":{"total":95995035648,"available":93411287040,"internally_reserved":0,"allocated":2034286592,"data_stored":2033850714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":874643456,"data_stored":874643456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1993936896,"data_stored":1993932824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":661979136,"data_stored":661979136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1742929920,"data_stored":1742929920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":660471808,"data_stored":660471808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2016219136,"data_stored":2016215064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":667222016,"data_stored":667222016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1996820480,"data_stored":1996816408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:05:31.772 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-24T12:05:31.772 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:05:31.772 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.3165247243566793 2024-03-24T12:05:31.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 2 0.3165247243566793 2024-03-24T12:05:31.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:31.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:31.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:32.000 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:32.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:32.081 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:32.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:32.157 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:32.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:32.259 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:32.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:32.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:32.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:32.425 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:32.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:32.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:32.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:32.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:32 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:05:32.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:32 smithi064 ceph-mon[30223]: from='client.25471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:05:32.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:32 smithi064 ceph-mon[30223]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T12:05:32.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:32 smithi064 ceph-mon[30223]: pgmap v597: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 962 op/s 2024-03-24T12:05:32.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:32 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:05:32.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:32 smithi064 ceph-mon[42243]: from='client.25471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:05:32.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:32 smithi064 ceph-mon[42243]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T12:05:32.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:32 smithi064 ceph-mon[42243]: pgmap v597: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 962 op/s 2024-03-24T12:05:32.566 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:32.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:32 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:05:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:32 smithi193 ceph-mon[33593]: from='client.25471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:05:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:32 smithi193 ceph-mon[33593]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T12:05:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:32 smithi193 ceph-mon[33593]: pgmap v597: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 962 op/s 2024-03-24T12:05:32.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:32.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:32.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:32.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:32.865 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:32.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:32.967 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:33.001 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:33.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:33.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:33.144 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:33.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:33.245 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:33.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:33.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:33.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:33.398 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:33.412 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:33.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:33.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:33.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:33.553 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:33.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:33.654 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:33.688 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:33.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:33.790 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:33.825 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:33.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:33.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:33.956 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:33.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:34.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:34.101 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:34.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:34.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:34.233 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:34.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:34.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:34.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:34.371 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:34.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:34.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:34.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:34.512 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:34.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:34.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:34.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:05:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:34.652 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:34.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:34.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:34 smithi064 ceph-mon[30223]: pgmap v598: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 947 op/s 2024-03-24T12:05:34.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:34 smithi064 ceph-mon[42243]: pgmap v598: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 947 op/s 2024-03-24T12:05:34.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:34.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:34 smithi193 ceph-mon[33593]: pgmap v598: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 947 op/s 2024-03-24T12:05:34.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:34.938 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:34.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:35.039 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:35.072 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:35.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:35.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:35.206 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:35.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:35.308 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:35.343 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:35.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:35.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:05:35.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:35.482 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:35.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:35.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:35.642 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:35.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:35.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:35 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:05:35.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:35 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:05:35.735 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:35 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:05:35.735 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:35 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:05:35.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:35.778 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:35.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:35.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:35 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:05:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:35 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:05:35.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:35.904 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:35.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:36.005 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:36.040 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:36.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:36.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:36.180 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:36.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:36.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:36.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:36.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:36.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:36.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:36.460 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:36.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:36.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:36.602 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:36.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:36.703 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:36.733 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:36.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:36.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:36 smithi064 ceph-mon[42243]: pgmap v599: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 947 op/s 2024-03-24T12:05:36.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:36 smithi064 ceph-mon[42243]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T12:05:36.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:36 smithi064 ceph-mon[30223]: pgmap v599: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 947 op/s 2024-03-24T12:05:36.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:36 smithi064 ceph-mon[30223]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T12:05:36.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:36.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:36 smithi193 ceph-mon[33593]: pgmap v599: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 947 op/s 2024-03-24T12:05:36.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:36 smithi193 ceph-mon[33593]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T12:05:36.869 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:36.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:36.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:37.007 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:37.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:37.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:37.146 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:37.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:37.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:37.284 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:37.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:37.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:37.424 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:37.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:37.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:37.569 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:37.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:37.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:37.707 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:37.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:37 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:05:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:37 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:05:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:37 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:05:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:37 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:05:37.802 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:05:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:05:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:37.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:37.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:05:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:05:37.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:37 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:05:37.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:37 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:05:37.846 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:37.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:37.949 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:37.981 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:37.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:38.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:38.112 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:38.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:38.214 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:38.242 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:38.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:38.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:38.380 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:38.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:38.481 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:38.504 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:38.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:38.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:38.640 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:38.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:38.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:38.766 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:38.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:38.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:38 smithi064 ceph-mon[30223]: pgmap v601: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:05:38.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:38 smithi064 ceph-mon[30223]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T12:05:38.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:38.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:38 smithi064 ceph-mon[42243]: pgmap v601: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:05:38.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:38 smithi064 ceph-mon[42243]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T12:05:38.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:38 smithi193 ceph-mon[33593]: pgmap v601: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:05:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:38 smithi193 ceph-mon[33593]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T12:05:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:38.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:38.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:38.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:39.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:39.033 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:39.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:39.134 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:39.158 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:39.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:39.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:39.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:39.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:39.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:39.440 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:39.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:39.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:39.576 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:39.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:39.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:39.714 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:39.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:39.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:39 smithi064 ceph-mon[30223]: 9.3 deep-scrub ok 2024-03-24T12:05:39.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:39 smithi064 ceph-mon[42243]: 9.3 deep-scrub ok 2024-03-24T12:05:39.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:39.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:39 smithi193 ceph-mon[33593]: 9.3 deep-scrub ok 2024-03-24T12:05:39.857 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:39.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:39.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:39.993 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:39.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:40.098 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:40.135 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:40.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:40.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:40.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:40.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:40.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:40.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:40.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:40.460 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:05:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:05:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:05:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:05:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215978 7.13 7:cbee3e48:::benchmark_data_smithi064_87013_object95021:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:44.912922+0000", 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 475.54393250800001, 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1576121580000001, 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215978 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912922+0000", 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912922+0000", 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912925+0000", 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0849999999999999e-06 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912930+0000", 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0660000000000003e-06 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.912940+0000", 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0710000000000001e-05 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:44.982216+0000", 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.069275701999999995 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006455+0000", 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024239369 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006466+0000", 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0155e-05 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.070430+0000", 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.06396409 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.070535+0000", 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104981 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215995 7.13 7:cee0aa74:::benchmark_data_smithi064_87013_object95038:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:45.006215+0000", 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 475.450639857, 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0807735059999999, 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215995 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006215+0000", 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006215+0000", 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006217+0000", 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006223+0000", 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5490000000000003e-06 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006365+0000", 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014170799999999999 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006376+0000", 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0444e-05 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006492+0000", 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116606 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006499+0000", 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.070882+0000", 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0643827809999999 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.071020+0000", 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013786199999999999 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,4,7", 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.071151+0000", 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131371 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.073770+0000", 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026192239999999999 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.073887+0000", 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116693 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.077318+0000", 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034309869999999999 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.083007+0000", 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056891340000000002 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.086764+0000", 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037570609999999999 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.086833+0000", 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8479000000000006e-05 2024-03-24T12:05:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.086989+0000", 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000155797 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:216009 7.13 7:ca212df6:::benchmark_data_smithi064_87013_object95052:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:45.006400+0000", 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 475.45045446799998, 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.091088863, 2024-03-24T12:05:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216009 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006400+0000", 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006400+0000", 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006402+0000", 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0729999999999999e-06 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006413+0000", 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1206000000000001e-05 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006421+0000", 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4889999999999995e-06 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006428+0000", 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8800000000000002e-06 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006524+0000", 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.611e-05 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.006531+0000", 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9940000000000003e-06 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.072292+0000", 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.065760874 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.072425+0000", 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132743 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,4,7", 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.072553+0000", 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012795999999999999 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.076497+0000", 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039439990000000001 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.076848+0000", 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035167100000000001 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.077325+0000", 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047677 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.083055+0000", 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057292860000000001 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.096675+0000", 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013620083999999999 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.097320+0000", 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064500600000000005 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.097489+0000", 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016971799999999999 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:229174 7.e 7:7d82ef5e:::benchmark_data_smithi064_87013_object108217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-03-24T12:05:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:59.457010+0000", 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 460.99984454299999, 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12239267099999999, 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 229174 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.457010+0000", 2024-03-24T12:05:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.457010+0000", 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.457012+0000", 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1880000000000001e-06 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.457016+0000", 2024-03-24T12:05:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.493e-06 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.457026+0000", 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.006e-05 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.457033+0000", 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0759999999999997e-06 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460416+0000", 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003382698 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460543+0000", 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012682899999999999 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460821+0000", 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027825499999999998 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.467248+0000", 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064268329999999999 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.469982+0000", 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002734649 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.579299+0000", 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10931618899999999 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.579352+0000", 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3557999999999998e-05 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.579403+0000", 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0843000000000001e-05 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:229177 7.e 7:70f3da66:::benchmark_data_smithi064_87013_object108220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:59.460314+0000", 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 460.99654109599999, 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119218478, 2024-03-24T12:05:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:05:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:05:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 229177 2024-03-24T12:05:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460314+0000", 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460314+0000", 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460316+0000", 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7230000000000001e-06 2024-03-24T12:05:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460321+0000", 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3249999999999998e-06 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460339+0000", 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8478000000000002e-05 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.460347+0000", 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9049999999999997e-06 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.464119+0000", 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037717369999999998 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.464239+0000", 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011968000000000001 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.464599+0000", 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000360653 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.561094+0000", 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096494637999999994 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.564784+0000", 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036898109999999999 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.579449+0000", 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014664959 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.579493+0000", 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4511999999999997e-05 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:59.579532+0000", 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9057000000000002e-05 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:72990 8.a 8:5a5d6a33:::benchmark_data_smithi064_120708_object72989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:05:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.012505+0000", 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 358.44434963700002, 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14825962200000001, 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 72990 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012505+0000", 2024-03-24T12:05:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012505+0000", 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012507+0000", 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1900000000000002e-06 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012529+0000", 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1361000000000001e-05 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012545+0000", 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6024000000000001e-05 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012553+0000", 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4349999999999997e-06 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012614+0000", 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0362999999999998e-05 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012877+0000", 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263007 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.013019+0000", 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014215199999999999 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.018482+0000", 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054628760000000002 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.018522+0000", 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0413000000000001e-05 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.160670+0000", 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14214798300000001 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.160731+0000", 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1137999999999996e-05 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.160765+0000", 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3680000000000003e-05 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:72992 8.a 8:558bd69d:::benchmark_data_smithi064_120708_object72991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.015224+0000", 2024-03-24T12:05:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 358.44163127500002, 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14865694300000001, 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 72992 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015224+0000", 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015224+0000", 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015226+0000", 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9089999999999998e-06 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015249+0000", 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3125999999999998e-05 2024-03-24T12:05:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015259+0000", 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0322999999999999e-05 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015267+0000", 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5270000000000001e-06 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015702+0000", 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043556900000000002 2024-03-24T12:05:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015829+0000", 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127119 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015993+0000", 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016342900000000001 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.019252+0000", 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032590649999999998 2024-03-24T12:05:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.027970+0000", 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087185219999999994 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163751+0000", 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13578027500000001 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163809+0000", 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8662999999999997e-05 2024-03-24T12:05:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163881+0000", 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1415999999999995e-05 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:73001 8.a 8:54eaffae:::benchmark_data_smithi064_120708_object73000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.020718+0000", 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 358.43613691399997, 2024-03-24T12:05:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14328966200000001, 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73001 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020718+0000", 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020718+0000", 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020720+0000", 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3530000000000002e-06 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020744+0000", 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3294000000000001e-05 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020757+0000", 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.306e-05 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020764+0000", 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4089999999999997e-06 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020808+0000", 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3523999999999999e-05 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020930+0000", 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012192 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.021097+0000", 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016748500000000001 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.027982+0000", 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068853250000000003 2024-03-24T12:05:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.028019+0000", 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6984999999999999e-05 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163926+0000", 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.135906215 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163972+0000", 2024-03-24T12:05:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6162999999999998e-05 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.164008+0000", 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5929000000000001e-05 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:73008 8.a 8:545958b9:::benchmark_data_smithi064_120708_object73007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.024320+0000", 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 358.43253523300001, 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.142947031, 2024-03-24T12:05:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73008 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024320+0000", 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024320+0000", 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024322+0000", 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.012e-06 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024345+0000", 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3402000000000001e-05 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024359+0000", 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4269e-05 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024367+0000", 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0870000000000007e-06 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024419+0000", 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.198e-05 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024544+0000", 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012440800000000001 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024675+0000", 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013087600000000001 2024-03-24T12:05:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.028147+0000", 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003472691 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.029846+0000", 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016989259999999999 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.167145+0000", 2024-03-24T12:05:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137298485 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.167218+0000", 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3440000000000002e-05 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.167267+0000", 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8455000000000003e-05 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:73013 8.a 8:5ddefa19:::benchmark_data_smithi064_120708_object73012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.030757+0000", 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 358.42609813399997, 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13935418199999999, 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73013 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030757+0000", 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030757+0000", 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030759+0000", 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2299999999999998e-06 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030782+0000", 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3312999999999999e-05 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030794+0000", 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2130000000000001e-05 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030802+0000", 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.841e-06 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030857+0000", 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4812000000000003e-05 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030982+0000", 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012492499999999999 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.031123+0000", 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140987 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.033678+0000", 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025544970000000002 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.035597+0000", 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001919104 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.169997+0000", 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13440000399999999 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.170056+0000", 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9839000000000001e-05 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.170111+0000", 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-05 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:177782 9.0 e295/246 9:05919d2f:::benchmark_data_smithi064_148274_object51969:head v 295'11103, mlcod=295'11100)", 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:07.752208+0000", 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.704646697999998, 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028175049000000001, 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.752208+0000", 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.752208+0000", 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.752210+0000", 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.587e-06 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.752220+0000", 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0013000000000001e-05 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.752235+0000", 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5012e-05 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.752243+0000", 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5150000000000006e-06 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.753219+0000", 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097566500000000002 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.753239+0000", 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9941000000000001e-05 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.780208+0000", 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026969165 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:07.780383+0000", 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017515099999999999 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200634 9.2 e296/246 9:4deab3a2:::benchmark_data_smithi064_148274_object74821:head v 296'12490, mlcod=296'12487)", 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.20933571, 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.1099523, 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247521+0000", 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8390000000000001e-06 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247528+0000", 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.374e-06 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247540+0000", 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1324e-05 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247548+0000", 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3110000000000006e-06 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.248032+0000", 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000483993 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.248052+0000", 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0441999999999998e-05 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357431+0000", 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10937805 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357472+0000", 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0967000000000003e-05 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200636 9.0 e296/246 9:059695cb:::benchmark_data_smithi064_148274_object74823:head v 296'12488, mlcod=296'12486)", 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.206816432, 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12092386400000001, 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:05:40.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250040+0000", 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250050+0000", 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7540000000000001e-06 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346892+0000", 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096842284000000001 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346904+0000", 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1429e-05 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.349515+0000", 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026114250000000001 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.349570+0000", 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5211999999999999e-05 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.370781+0000", 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021210360000000001 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.370962+0000", 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001818 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200640 9.6 e296/246 9:66c6e6e1:::benchmark_data_smithi064_148274_object74827:head v 296'12828, mlcod=296'12826)", 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.204433385, 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10521612800000001, 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.534 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252423+0000", 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.575e-06 2024-03-24T12:05:40.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252432+0000", 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9539999999999993e-06 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346205+0000", 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093772446999999995 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346228+0000", 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3819999999999999e-05 2024-03-24T12:05:40.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.348705+0000", 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024768310000000001 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.348731+0000", 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6122000000000002e-05 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357526+0000", 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087951639999999994 2024-03-24T12:05:40.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357638+0000", 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111215 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200637 9.0 e296/246 9:07f00a43:::benchmark_data_smithi064_148274_object74824:head v 296'12489, mlcod=296'12486)", 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.346932+0000", 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.109922430999999, 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024216919, 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:40.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346932+0000", 2024-03-24T12:05:40.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346932+0000", 2024-03-24T12:05:40.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346934+0000", 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2330000000000001e-06 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346941+0000", 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5329999999999997e-06 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346950+0000", 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.568e-06 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346956+0000", 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8749999999999997e-06 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.352613+0000", 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056574249999999998 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.352634+0000", 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-05 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.370968+0000", 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01833417 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371149+0000", 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018108499999999999 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200644 9.0 e296/246 9:0db93ad1:::benchmark_data_smithi064_148274_object74831:head v 296'12490, mlcod=296'12486)", 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.346964+0000", 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.109891287, 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024375709999999998, 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346964+0000", 2024-03-24T12:05:40.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346964+0000", 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346964+0000", 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9900000000000001e-07 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346969+0000", 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9860000000000002e-06 2024-03-24T12:05:40.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346974+0000", 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7840000000000003e-06 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346978+0000", 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5449999999999997e-06 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.355438+0000", 2024-03-24T12:05:40.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084600270000000002 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.355459+0000", 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0194999999999999e-05 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371153+0000", 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015693928999999999 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371339+0000", 2024-03-24T12:05:40.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018674499999999999 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200645 9.0 e296/246 9:03b42ab6:::benchmark_data_smithi064_148274_object74832:head v 296'12491, mlcod=296'12486)", 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.346985+0000", 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.109870159, 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024555286999999999, 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346985+0000", 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346985+0000", 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346985+0000", 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0200000000000003e-07 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346995+0000", 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0324e-05 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347001+0000", 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6169999999999998e-06 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347006+0000", 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0499999999999999e-06 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.358321+0000", 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011314849 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.358340+0000", 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.872e-05 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371342+0000", 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013002642999999999 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371540+0000", 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000197682 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200655 9.0 e296/246 9:021341bf:::benchmark_data_smithi064_148274_object74842:head v 296'12492, mlcod=296'12486)", 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.347013+0000", 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.109841482, 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024733766000000001, 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347013+0000", 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347013+0000", 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347014+0000", 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9499999999999998e-07 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347018+0000", 2024-03-24T12:05:40.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1949999999999996e-06 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347023+0000", 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5549999999999996e-06 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347027+0000", 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0289999999999999e-06 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.365205+0000", 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018178711 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.365226+0000", 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0186e-05 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371543+0000", 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006317652 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371747+0000", 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000204043 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200667 9.0 e296/246 9:0f4d4acb:::benchmark_data_smithi064_148274_object74854:head v 296'12493, mlcod=296'12486)", 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.347032+0000", 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.109822493999999, 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024933561, 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347032+0000", 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347032+0000", 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347033+0000", 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9000000000000002e-07 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347036+0000", 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.636e-06 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347041+0000", 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.442e-06 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.347045+0000", 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3089999999999997e-06 2024-03-24T12:05:40.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.368128+0000", 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021082876 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.368149+0000", 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0596000000000001e-05 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371750+0000", 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003601651 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.371966+0000", 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021566099999999999 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:202401 9.2 e297/246 9:44a84362:::benchmark_data_smithi064_148274_object76588:head v 297'12608, mlcod=297'12605)", 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:31.732263+0000", 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7245915210000007, 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026721431, 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:05:40.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.732263+0000", 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.732263+0000", 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.732265+0000", 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.732272+0000", 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2660000000000004e-06 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.732283+0000", 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0787e-05 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.732290+0000", 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.311e-06 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.732465+0000", 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017519700000000001 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.732532+0000", 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6051999999999997e-05 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.758858+0000", 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026326313 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:31.758985+0000", 2024-03-24T12:05:40.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126985 2024-03-24T12:05:40.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:05:40.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:05:40.564 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:05:40.571 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:40.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:40.673 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:40.706 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:40.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:40 smithi064 ceph-mon[30223]: pgmap v603: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-24T12:05:40.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:40 smithi064 ceph-mon[42243]: pgmap v603: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-24T12:05:40.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:40.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:40 smithi193 ceph-mon[33593]: pgmap v603: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-24T12:05:40.848 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:40.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:40.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:40.974 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:40.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:41.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:41.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:41.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:41.217 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:41.243 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:41.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:41.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:41.381 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:41.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:41.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:41.519 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:41.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:41.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:41.659 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:41.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:41.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:41.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:41.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:41.901 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:41.942 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:41.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:42.043 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:42.080 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:42.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:42.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:42.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:42.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:42.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:05:42.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:42.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:42.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:42.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:42.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:42.512 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:42.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:42.612 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:42.626 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:42.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:42.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:42.792 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:42.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:42.893 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:42.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:42 smithi064 ceph-mon[30223]: pgmap v604: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 822 op/s 2024-03-24T12:05:42.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:42.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:42.926 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:42.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:42 smithi064 ceph-mon[42243]: pgmap v604: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 822 op/s 2024-03-24T12:05:42.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:42.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:42.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:43.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:43.068 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:43.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:43.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:42 smithi193 ceph-mon[33593]: pgmap v604: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 822 op/s 2024-03-24T12:05:43.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:43.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:43.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:43.211 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:43.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:43.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:43.351 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:43.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:43.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:43.490 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:43.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:43.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:43.607 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:43.625 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:43.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:43.725 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 0.316525 (5107) 2024-03-24T12:05:43.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:43.757 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:43.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:43.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:43.882 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:43.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:43.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:44.007 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:44.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:44.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:43 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/615677182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]: dispatch 2024-03-24T12:05:44.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:43 smithi064 ceph-mon[30223]: from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]: dispatch 2024-03-24T12:05:44.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:43 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/615677182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]: dispatch 2024-03-24T12:05:44.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:43 smithi064 ceph-mon[42243]: from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]: dispatch 2024-03-24T12:05:44.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:43 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/615677182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]: dispatch 2024-03-24T12:05:44.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:43 smithi193 ceph-mon[33593]: from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]: dispatch 2024-03-24T12:05:44.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:44.133 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:44.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:44.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:44.258 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:44.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:44.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:44.383 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:44.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:44.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:44.510 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:44.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:44.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:44.634 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:44.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:44.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:44.768 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:44.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[30223]: pgmap v605: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[30223]: from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]': finished 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[30223]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[42243]: pgmap v605: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[42243]: from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]': finished 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[42243]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:05:44.846 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:44 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:05:44.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:44.893 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:44.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:44.995 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:45.021 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:45.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:45.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:44 smithi193 ceph-mon[33593]: pgmap v605: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-24T12:05:45.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:44 smithi193 ceph-mon[33593]: from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.3165247243566793}]': finished 2024-03-24T12:05:45.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:44 smithi193 ceph-mon[33593]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T12:05:45.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:44 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:05:45.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:44 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:05:45.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:45.145 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:45.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:45.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:45.270 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:45.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:45.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:45.399 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:45.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:45.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:45.501 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:45.528 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:45.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:45.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:45.654 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:45.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:45.756 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:45.782 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:45.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:45.884 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:45.918 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:45.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:46.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:45 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:05:46.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:45 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:05:46.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:45 smithi064 ceph-mon[30223]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T12:05:46.009 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:45 smithi064 ceph-mon[30223]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T12:05:46.010 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:45 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:05:46.010 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:45 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:05:46.010 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:45 smithi064 ceph-mon[42243]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T12:05:46.010 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:45 smithi064 ceph-mon[42243]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T12:05:46.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:46.053 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:46.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:46.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:45 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:05:46.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:45 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:05:46.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:45 smithi193 ceph-mon[33593]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T12:05:46.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:45 smithi193 ceph-mon[33593]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T12:05:46.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:46.191 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:46.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:46.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:46.331 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:46.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:46.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:46.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:46.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:46.568 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:46.595 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:46.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:46.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:46.742 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:46.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:46.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:46.884 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:46.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:46.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:47.012 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:47.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:47.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:46 smithi064 ceph-mon[30223]: pgmap v608: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 646 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:05:47.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:46 smithi064 ceph-mon[30223]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 299'13526 2024-03-24T12:05:47.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:46 smithi064 ceph-mon[30223]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 299'13826 2024-03-24T12:05:47.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:46 smithi064 ceph-mon[42243]: pgmap v608: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 646 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:05:47.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:46 smithi064 ceph-mon[42243]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 299'13526 2024-03-24T12:05:47.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:46 smithi064 ceph-mon[42243]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 299'13826 2024-03-24T12:05:47.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:46 smithi193 ceph-mon[33593]: pgmap v608: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 646 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:05:47.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:46 smithi193 ceph-mon[33593]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 299'13526 2024-03-24T12:05:47.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:46 smithi193 ceph-mon[33593]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 299'13826 2024-03-24T12:05:47.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:47.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:47.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:47.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:47.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:47.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:47.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:47.425 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:47.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:47.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:47.562 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:47.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:47.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:47.705 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:47.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:47.752 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:05:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:05:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:47.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:47.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:47 smithi193 ceph-mon[33593]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T12:05:47.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:47 smithi193 ceph-mon[33593]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T12:05:47.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:05:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:05:47.843 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:47.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:47.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:47.986 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:47.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:48.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:47 smithi064 ceph-mon[30223]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T12:05:48.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:47 smithi064 ceph-mon[30223]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T12:05:48.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:47 smithi064 ceph-mon[42243]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T12:05:48.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:47 smithi064 ceph-mon[42243]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T12:05:48.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:48.122 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:48.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:48.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:48.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:48.260 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:48.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:48.361 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:48.395 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:48.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:48.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:48.536 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:48.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:48.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:48.678 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:48.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:48.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:48.814 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:48.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:48.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:48.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:49.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:48 smithi064 ceph-mon[30223]: pgmap v611: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 165 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 546 op/s; 2122/104664 objects misplaced (2.027%); 217 B/s, 4 objects/s recovering 2024-03-24T12:05:49.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:48 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:49.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:48 smithi064 ceph-mon[30223]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:05:49.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:48 smithi064 ceph-mon[42243]: pgmap v611: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 165 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 546 op/s; 2122/104664 objects misplaced (2.027%); 217 B/s, 4 objects/s recovering 2024-03-24T12:05:49.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:48 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:49.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:48 smithi064 ceph-mon[42243]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:05:49.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:48 smithi193 ceph-mon[33593]: pgmap v611: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 165 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 546 op/s; 2122/104664 objects misplaced (2.027%); 217 B/s, 4 objects/s recovering 2024-03-24T12:05:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:48 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:48 smithi193 ceph-mon[33593]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:05:49.089 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:49.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:49.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:49.267 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:49.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:49.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:49.416 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:49.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:49.517 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:49.548 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:49.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:49.575 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:05:49.605 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:05:49.605 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:05:49.605 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-24T12:05:49.605 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T12:05:49.605 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T12:05:49.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-24T12:05:49.623 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:05:49.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:49.682 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:05:49.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:49.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:49.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:49.744 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:05:49.794 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:05:49.817 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:49.839 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:05:49.846 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:49.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:49.881 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:05:49.921 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:05:49.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:05:49.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:05:49.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:05:49.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:05:49.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:05:49.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:05:49.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:05:49.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:05:49.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:49 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4212047741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:49.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:49 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4212047741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:49.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:49.974 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:49.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:50.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:50.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:49 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4212047741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:05:50.114 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:50.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:50.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:50.216 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:50.251 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:50.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:50.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:50.395 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:50.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:50.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:50.527 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:05:50.527 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:05:50.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:50.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:50.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:50.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:50.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:50.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:50.800 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:05:50.820 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:50.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:50.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:50.957 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:50.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:51.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:50 smithi064 ceph-mon[30223]: pgmap v613: 168 pgs: 2 active+remapped+backfill_toofull, 2 active+undersized+remapped, 164 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 845 op/s; 4095/99366 objects misplaced (4.121%); 232 B/s, 8 objects/s recovering 2024-03-24T12:05:51.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:50 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1955088369' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:50 smithi064 ceph-mon[42243]: pgmap v613: 168 pgs: 2 active+remapped+backfill_toofull, 2 active+undersized+remapped, 164 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 845 op/s; 4095/99366 objects misplaced (4.121%); 232 B/s, 8 objects/s recovering 2024-03-24T12:05:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:50 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1955088369' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:51.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:51.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:50 smithi193 ceph-mon[33593]: pgmap v613: 168 pgs: 2 active+remapped+backfill_toofull, 2 active+undersized+remapped, 164 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 845 op/s; 4095/99366 objects misplaced (4.121%); 232 B/s, 8 objects/s recovering 2024-03-24T12:05:51.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:50 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1955088369' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:05:51.098 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:51.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:51.200 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:51.239 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:51.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:51.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:51.384 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:51.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:51.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:51.519 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:51.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:51.597 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:51.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:51.660 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:51.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:51.761 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:51.798 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:51.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:51.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:51.938 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:51.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:52.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:51 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:05:52.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:51 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1955088369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:05:52.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:51 smithi064 ceph-mon[30223]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T12:05:52.035 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:51 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:05:52.035 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:51 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1955088369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:05:52.035 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:51 smithi064 ceph-mon[42243]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T12:05:52.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:52.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:05:52.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:51 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:05:52.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:51 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1955088369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:05:52.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:51 smithi193 ceph-mon[33593]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T12:05:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:52.087 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:52.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:52.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:52.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:52.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:52.340 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:52.384 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:52.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:52.485 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:52.528 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:52.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:52.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:52.680 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:52.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:52.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:52.828 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:52.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:52.929 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:52.980 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:52.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:53.056 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[30223]: pgmap v615: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+remapped, 165 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 4095/96984 objects misplaced (4.222%); 191 B/s, 7 objects/s recovering 2024-03-24T12:05:53.056 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:05:53.056 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:05:53.056 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[30223]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T12:05:53.056 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:05:53.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[42243]: pgmap v615: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+remapped, 165 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 4095/96984 objects misplaced (4.222%); 191 B/s, 7 objects/s recovering 2024-03-24T12:05:53.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:05:53.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:05:53.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[42243]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T12:05:53.057 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:52 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:05:53.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:53.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:52 smithi193 ceph-mon[33593]: pgmap v615: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+remapped, 165 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 4095/96984 objects misplaced (4.222%); 191 B/s, 7 objects/s recovering 2024-03-24T12:05:53.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:52 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:05:53.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:52 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:05:53.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:52 smithi193 ceph-mon[33593]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T12:05:53.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:52 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:05:53.131 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:53.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:53.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:05:53.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:53.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:53.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:53.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:53.409 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:53.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:53.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:53.548 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:53.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:53.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:53.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:53.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:53.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:53.772 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:53.788 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:53.821 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:53.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:53.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:53.963 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:53.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:54.064 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:54.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:53 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:05:54.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:53 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:05:54.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:53 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:54.071 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:53 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:05:54.071 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:53 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:05:54.071 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:53 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:54.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:53 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:05:54.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:53 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:05:54.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:53 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:54.094 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:54.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:54.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:54.230 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:54.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:54.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:54.367 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:54.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:54.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:54.505 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:54.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:54.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:05:54.607 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:54.639 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:54.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:54.741 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:54.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:54.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:54.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:54.905 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:54.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:55.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:55.045 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:55.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:55.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:54 smithi064 ceph-mon[42243]: pgmap v617: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 1.37k op/s; 3250/79938 objects misplaced (4.066%); 36 B/s, 3 objects/s recovering 2024-03-24T12:05:55.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:54 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:05:55.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:54 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:05:55.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:54 smithi064 ceph-mon[30223]: pgmap v617: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 1.37k op/s; 3250/79938 objects misplaced (4.066%); 36 B/s, 3 objects/s recovering 2024-03-24T12:05:55.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:54 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:05:55.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:54 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:05:55.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:54 smithi193 ceph-mon[33593]: pgmap v617: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 1.37k op/s; 3250/79938 objects misplaced (4.066%); 36 B/s, 3 objects/s recovering 2024-03-24T12:05:55.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:54 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:05:55.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:54 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:05:55.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:55.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:55.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:55.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:55.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:55.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:55.458 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:55.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:55.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:55.595 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:55.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:55.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:55.733 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:55.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:55.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:55.875 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:55.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:55.977 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:56.005 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:56.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:56.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[42243]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T12:05:56.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [7, 3]}]: dispatch 2024-03-24T12:05:56.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [1, 2]}]: dispatch 2024-03-24T12:05:56.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]: dispatch 2024-03-24T12:05:56.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-03-24T12:05:56.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[30223]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T12:05:56.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [7, 3]}]: dispatch 2024-03-24T12:05:56.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [1, 2]}]: dispatch 2024-03-24T12:05:56.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]: dispatch 2024-03-24T12:05:56.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:55 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-03-24T12:05:56.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:55 smithi193 ceph-mon[33593]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T12:05:56.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [7, 3]}]: dispatch 2024-03-24T12:05:56.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [1, 2]}]: dispatch 2024-03-24T12:05:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]: dispatch 2024-03-24T12:05:56.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:55 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-03-24T12:05:56.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:56.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:56.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:56.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:56.295 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:56.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:56.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:56.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:56.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:56.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:56.572 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:56.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:56.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:56.713 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:56.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:56.815 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:56.846 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:56.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:56.866 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:05:56.866 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T12:05:56.866 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:05:56.921 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:56.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:56.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:56.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:57.100 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:57.138 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:57.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:57.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:57.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:57.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[30223]: pgmap v619: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 3250/79938 objects misplaced (4.066%) 2024-03-24T12:05:57.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [7, 3]}]': finished 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [1, 2]}]': finished 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]': finished 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[30223]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[42243]: pgmap v619: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 3250/79938 objects misplaced (4.066%) 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [7, 3]}]': finished 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [1, 2]}]': finished 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]': finished 2024-03-24T12:05:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-03-24T12:05:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[42243]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T12:05:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:56 smithi193 ceph-mon[33593]: pgmap v619: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 3250/79938 objects misplaced (4.066%) 2024-03-24T12:05:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [7, 3]}]': finished 2024-03-24T12:05:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [1, 2]}]': finished 2024-03-24T12:05:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]': finished 2024-03-24T12:05:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-03-24T12:05:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:56 smithi193 ceph-mon[33593]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T12:05:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:05:57.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:57.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:57.422 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:57.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:57.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:57.570 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:57.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:57.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:57.697 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:05:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:57.708 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:57.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:57.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:05:57.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:05:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:05:57.855 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:57.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:05:57.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:57.964 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:05:57.964 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:05:57.964 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-mon[30223]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T12:05:57.964 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:05:57.964 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:05:57.964 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:05:57.965 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-mon[42243]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T12:05:57.965 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:05:57.965 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:05:57.965 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:57 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:05:57.990 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:57.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:58.091 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:05:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:58.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:58.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:05:58.232 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:58.267 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:58.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:57 smithi193 ceph-mon[33593]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T12:05:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:57 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:05:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:57 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:05:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:57 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:05:58.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:58.403 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:58.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:58.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:05:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:58.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:05:58.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:58.682 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:58.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:58.727 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:05:58.783 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:58.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:58.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:58.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:05:58.962 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:58.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:05:59.064 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:59.101 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:59.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:59.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:58 smithi064 ceph-mon[42243]: pgmap v622: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 915 B/s rd, 0 B/s wr, 1.18k op/s; 2838/68664 objects misplaced (4.133%) 2024-03-24T12:05:59.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:58 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:59.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:58 smithi064 ceph-mon[42243]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:05:59.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:58 smithi064 ceph-mon[42243]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T12:05:59.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:58 smithi064 ceph-mon[30223]: pgmap v622: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 915 B/s rd, 0 B/s wr, 1.18k op/s; 2838/68664 objects misplaced (4.133%) 2024-03-24T12:05:59.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:58 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:59.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:58 smithi064 ceph-mon[30223]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:05:59.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:58 smithi064 ceph-mon[30223]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T12:05:59.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:05:59.239 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:59.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:05:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:58 smithi193 ceph-mon[33593]: pgmap v622: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 915 B/s rd, 0 B/s wr, 1.18k op/s; 2838/68664 objects misplaced (4.133%) 2024-03-24T12:05:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:58 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:58 smithi193 ceph-mon[33593]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:05:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:58 smithi193 ceph-mon[33593]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T12:05:59.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:05:59.376 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:59.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:05:59.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:59.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:59.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:59.658 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:59.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:05:59.760 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:05:59.795 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:05:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:59.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:05:59.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:05:59.934 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:05:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:05:59.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:00.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:59 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:06:00.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:05:59 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:06:00.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:59 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:06:00.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:05:59 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:06:00.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:00.078 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:00.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:00.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:00.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:00.322 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:59 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:06:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:05:59 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:06:00.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:00.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:00.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:00.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:00.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:00.640 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:00.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:00.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:00.777 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:00.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:00.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:00.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:01.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:01.045 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:01.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:01.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:01.181 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:01.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:01.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:01.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:00 smithi064 ceph-mon[30223]: pgmap v624: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 1.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.47k op/s; 2429/59760 objects misplaced (4.065%) 2024-03-24T12:06:01.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:00 smithi064 ceph-mon[30223]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T12:06:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:00 smithi064 ceph-mon[42243]: pgmap v624: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 1.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.47k op/s; 2429/59760 objects misplaced (4.065%) 2024-03-24T12:06:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:00 smithi064 ceph-mon[42243]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T12:06:01.319 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:01.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:00 smithi193 ceph-mon[33593]: pgmap v624: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 1.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.47k op/s; 2429/59760 objects misplaced (4.065%) 2024-03-24T12:06:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:00 smithi193 ceph-mon[33593]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T12:06:01.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:01.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:01.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:01.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:01.594 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:01.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:01.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:01.737 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:01.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:01.839 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:01.876 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:01.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:01.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:02.012 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:02.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:02.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:02.151 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:02.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:02.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:02.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:02.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:02.394 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:02.427 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:02.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:02.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:02.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:02.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:02.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:02.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:02.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:02.842 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:02.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:02.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:02.978 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:02.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:03.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:02 smithi064 ceph-mon[42243]: pgmap v626: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.37k op/s; 2429/57150 objects misplaced (4.250%) 2024-03-24T12:06:03.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:02 smithi064 ceph-mon[42243]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T12:06:03.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:02 smithi064 ceph-mon[30223]: pgmap v626: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.37k op/s; 2429/57150 objects misplaced (4.250%) 2024-03-24T12:06:03.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:02 smithi064 ceph-mon[30223]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T12:06:03.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:03.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:03.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:03.254 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:03.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:03.310 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:06:03.310 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:06:03.310 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-24T12:06:03.310 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T12:06:03.310 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T12:06:03.310 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-24T12:06:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:02 smithi193 ceph-mon[33593]: pgmap v626: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.37k op/s; 2429/57150 objects misplaced (4.250%) 2024-03-24T12:06:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:02 smithi193 ceph-mon[33593]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T12:06:03.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:03.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:03.395 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:03.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:03.496 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:03.553 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:03.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:03.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:03.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:03.719 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:03.788 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:03.830 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:03.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:03.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:03.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:06:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:03.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:03.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:04.073 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:04.109 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:04.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:04.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:04.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:04.234 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:04.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:04.336 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:04.374 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:04.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:04.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:04.513 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:04.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:04.615 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:04.638 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:04.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:04.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:04.774 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:04.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:04.874 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:04.906 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:04.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:05.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:05.047 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:05.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:05.123 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:04 smithi193 ceph-mon[33593]: pgmap v627: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 787 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 781 B/s rd, 0 B/s wr, 1.54k op/s; 1552/38337 objects misplaced (4.048%) 2024-03-24T12:06:05.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:05.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:05.224 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:04 smithi064 ceph-mon[30223]: pgmap v627: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 787 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 781 B/s rd, 0 B/s wr, 1.54k op/s; 1552/38337 objects misplaced (4.048%) 2024-03-24T12:06:05.224 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:04 smithi064 ceph-mon[42243]: pgmap v627: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 787 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 781 B/s rd, 0 B/s wr, 1.54k op/s; 1552/38337 objects misplaced (4.048%) 2024-03-24T12:06:05.286 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:05.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:05.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:05.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:05.449 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:05.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:05.550 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:05.555 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:06:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:05.578 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:05.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:05.585 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:06:05.625 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:06:05.667 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:06:05.679 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:05.704 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:06:05.712 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:05.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:05.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:05.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:05.850 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:06:05.878 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:06:05.923 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:06:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:06:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:06:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:06:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:06:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:06:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:06:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:06:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:06:05.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:05.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:06.076 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:05 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1093316351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:06.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:05 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1093316351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:06.089 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:06.167 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:06.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:06.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:06.331 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:06.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:06.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:05 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1093316351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:06.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:06.479 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:06.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:06.581 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:06.614 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:06.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:06.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:06.755 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:06.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:06.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:06.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:06.884 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:06:06.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:07.023 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:07.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:07.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:06 smithi064 ceph-mon[30223]: pgmap v628: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 787 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 1552/38337 objects misplaced (4.048%) 2024-03-24T12:06:07.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:06 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/89013275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:06:07.026 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:06 smithi064 ceph-mon[42243]: pgmap v628: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 787 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 1552/38337 objects misplaced (4.048%) 2024-03-24T12:06:07.026 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:06 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/89013275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:06:07.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:07.166 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:07.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:07.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:07.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:06 smithi193 ceph-mon[33593]: pgmap v628: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 787 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 1552/38337 objects misplaced (4.048%) 2024-03-24T12:06:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:06 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/89013275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:06:07.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:07.429 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:07.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:07.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:07.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:07.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:07.710 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:07.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:07.789 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:06:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:07.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:06:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:06:07.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:07.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:07.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:07.996 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:07.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:08.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:08.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/89013275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:06:08.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[30223]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T12:06:08.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[30223]: 9.d scrub starts 2024-03-24T12:06:08.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[30223]: 9.d scrub ok 2024-03-24T12:06:08.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:08.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/89013275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:06:08.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[42243]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T12:06:08.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[42243]: 9.d scrub starts 2024-03-24T12:06:08.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:07 smithi064 ceph-mon[42243]: 9.d scrub ok 2024-03-24T12:06:08.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:07 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:08.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:07 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/89013275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:06:08.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:07 smithi193 ceph-mon[33593]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T12:06:08.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:07 smithi193 ceph-mon[33593]: 9.d scrub starts 2024-03-24T12:06:08.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:07 smithi193 ceph-mon[33593]: 9.d scrub ok 2024-03-24T12:06:08.098 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:08.135 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:08.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:08.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:08.274 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:08.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:08.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:08.423 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:08.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:08.523 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:08.559 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:08.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:08.661 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:08.699 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:08.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:08.801 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:08.837 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:08.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:08.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:08.981 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:08.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:09.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T12:06:09.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T12:06:09.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[30223]: pgmap v630: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 584 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 1201/28623 objects misplaced (4.196%) 2024-03-24T12:06:09.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:09.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[42243]: pgmap v630: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 584 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 1201/28623 objects misplaced (4.196%) 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:06:09.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:08 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:06:09.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:09.134 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:09.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:09.174 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:08 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T12:06:09.174 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:08 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T12:06:09.174 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:08 smithi193 ceph-mon[33593]: pgmap v630: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 584 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 1201/28623 objects misplaced (4.196%) 2024-03-24T12:06:09.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:08 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:09.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:08 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T12:06:09.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:08 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:06:09.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:08 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:06:09.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:09.274 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:09.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:09.375 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:09.418 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:09.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:09.484 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:06:09.484 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:06:09.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:06:09.484 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:06:09.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.484 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:215971 7.11 7:8c040365:::benchmark_data_smithi064_87013_object95014:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:45.007482+0000", 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 504.47340226699998, 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99587450600000005, 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 215971 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007482+0000", 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007482+0000", 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007484+0000", 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-03-24T12:06:09.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007488+0000", 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.895e-06 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007497+0000", 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007506+0000", 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9309999999999997e-06 2024-03-24T12:06:09.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009756+0000", 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002250499 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009765+0000", 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.738e-06 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.003219+0000", 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99345422900000002 2024-03-24T12:06:09.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.003357+0000", 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013740399999999999 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:216013 7.11 7:8df42d58:::benchmark_data_smithi064_87013_object95056:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e150)", 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:45.007629+0000", 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 504.473255198, 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.001424466, 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216013 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007629+0000", 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007629+0000", 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007630+0000", 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007635+0000", 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1359999999999999e-06 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007643+0000", 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.303e-06 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.007652+0000", 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7870000000000007e-06 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009789+0000", 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002137417 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:45.009796+0000", 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1180000000000003e-06 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.003646+0000", 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99385028799999997 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.003784+0000", 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013761099999999999 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:06:09.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.004116+0000", 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033185499999999999 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.006002+0000", 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001885821 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.006098+0000", 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6527000000000006e-05 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.006135+0000", 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6325999999999997e-05 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.008533+0000", 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00239822 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.008925+0000", 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039180900000000002 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:46.009054+0000", 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012919400000000001 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221871 7.14 7:2e392a63:::benchmark_data_smithi064_87013_object100914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.055097+0000", 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 496.42578807199999, 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002813153, 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221871 2024-03-24T12:06:09.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.497 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055097+0000", 2024-03-24T12:06:09.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055097+0000", 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055099+0000", 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.813e-06 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055106+0000", 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8170000000000001e-06 2024-03-24T12:06:09.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055120+0000", 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3472000000000001e-05 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055130+0000", 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0088000000000001e-05 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055217+0000", 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7029999999999999e-05 2024-03-24T12:06:09.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.055227+0000", 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0217999999999999e-05 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.053733+0000", 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99850598199999996 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.053837+0000", 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010372100000000001 2024-03-24T12:06:09.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.054103+0000", 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026637299999999998 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057656+0000", 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003553301 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057669+0000", 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2533e-05 2024-03-24T12:06:09.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057808+0000", 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013899800000000001 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057881+0000", 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3225000000000004e-05 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057910+0000", 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8581999999999998e-05 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221882 7.14 7:2c6e9666:::benchmark_data_smithi064_87013_object100925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.070802+0000", 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 496.41008303299998, 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.987318433, 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221882 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070802+0000", 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070802+0000", 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070804+0000", 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.824e-06 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070813+0000", 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6070000000000008e-06 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070832+0000", 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9219999999999999e-05 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070843+0000", 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0601999999999999e-05 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070915+0000", 2024-03-24T12:06:09.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2408000000000002e-05 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.070925+0000", 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5270000000000004e-06 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.055439+0000", 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98451412999999999 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.055547+0000", 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108251 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.055674+0000", 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012672399999999999 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.057999+0000", 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023249820000000002 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058019+0000", 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0438999999999999e-05 2024-03-24T12:06:09.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058045+0000", 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5896000000000001e-05 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058090+0000", 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4468999999999997e-05 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058120+0000", 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0354000000000001e-05 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.509 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221955 7.14 7:2cc02cd6:::benchmark_data_smithi064_87013_object100998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.134401+0000", 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 496.34648407999998, 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.92390937799999995, 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221955 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134401+0000", 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134401+0000", 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134402+0000", 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.711e-06 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134407+0000", 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9579999999999998e-06 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134424+0000", 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6609e-05 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134432+0000", 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2619999999999998e-06 2024-03-24T12:06:09.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134491+0000", 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8668999999999997e-05 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.134499+0000", 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1810000000000001e-06 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.056347+0000", 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.92184798899999998 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.056432+0000", 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4890999999999997e-05 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.056642+0000", 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021018600000000001 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058140+0000", 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001497623 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058181+0000", 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1705999999999997e-05 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058244+0000", 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2739999999999999e-05 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058273+0000", 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9241999999999999e-05 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058310+0000", 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6610999999999999e-05 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:221994 7.14 7:2d64b07a:::benchmark_data_smithi064_87013_object101037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.166370+0000", 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 496.31451412500002, 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.89684323600000004, 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:06:09.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 221994 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166370+0000", 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166370+0000", 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166373+0000", 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.571e-06 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166379+0000", 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6540000000000004e-06 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166394+0000", 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5044e-05 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166403+0000", 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9129999999999992e-06 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166581+0000", 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868100000000001 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.166591+0000", 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3710000000000006e-06 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058358+0000", 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.89176743599999997 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058420+0000", 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2348999999999995e-05 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.058552+0000", 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131366 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.061732+0000", 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031802409999999999 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063030+0000", 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012981259999999999 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063110+0000", 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9950999999999994e-05 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063167+0000", 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6508999999999997e-05 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063214+0000", 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7024000000000002e-05 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.34742.0:222042 7.14 7:2868e145:::benchmark_data_smithi064_87013_object101085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:57:53.204936+0000", 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 496.27594890500001, 2024-03-24T12:06:09.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86118150800000004, 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.34742", 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 222042 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.523 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:09.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204936+0000", 2024-03-24T12:06:09.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204936+0000", 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204937+0000", 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4610000000000001e-06 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204942+0000", 2024-03-24T12:06:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8960000000000002e-06 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204954+0000", 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2234e-05 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.204962+0000", 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8089999999999993e-06 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.205016+0000", 2024-03-24T12:06:09.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3952000000000001e-05 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:53.205024+0000", 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6279999999999996e-06 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.059312+0000", 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.85428786700000003 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.059427+0000", 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115158 2024-03-24T12:06:09.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.059608+0000", 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000181392 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.063072+0000", 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034640309999999998 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.064297+0000", 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012244879999999999 2024-03-24T12:06:09.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066020+0000", 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017236599999999999 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066074+0000", 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4057000000000001e-05 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:57:54.066117+0000", 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2874999999999999e-05 2024-03-24T12:06:09.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104600 8.18 8:195e728d:::benchmark_data_smithi064_120708_object104599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 356.72142938500002, 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0728234670000001, 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:06:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104600 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759458+0000", 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6309999999999999e-06 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759492+0000", 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4329000000000003e-05 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759509+0000", 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6705e-05 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759520+0000", 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.111e-05 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759583+0000", 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3483000000000002e-05 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759592+0000", 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-03-24T12:06:09.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804373+0000", 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0447811650000001 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804557+0000", 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018477399999999999 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804989+0000", 2024-03-24T12:06:09.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043201600000000002 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809404+0000", 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044143280000000003 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813653+0000", 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042489600000000004 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.815161+0000", 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001507911 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831615+0000", 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016453925000000001 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831679+0000", 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4657000000000001e-05 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832279+0000", 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059939900000000003 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104606 8.18 8:1f771a39:::benchmark_data_smithi064_120708_object104605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 356.72134393099998, 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.073005889, 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:06:09.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104606 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759564+0000", 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2459999999999998e-05 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759572+0000", 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8399999999999995e-06 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759578+0000", 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5830000000000003e-06 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759626+0000", 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7576e-05 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759632+0000", 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4500000000000001e-06 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.806996+0000", 2024-03-24T12:06:09.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0473636369999999 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.807166+0000", 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170104 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.807568+0000", 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000401455 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813591+0000", 2024-03-24T12:06:09.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006023919 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.815217+0000", 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016253210000000001 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831571+0000", 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016353873000000001 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832375+0000", 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080415300000000005 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832441+0000", 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5860999999999998e-05 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832547+0000", 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010589799999999999 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104642 8.18 8:1b5ea0e5:::benchmark_data_smithi064_120708_object104641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 356.72129660299998, 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.073407242, 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104642 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759589+0000", 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759610+0000", 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0948e-05 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759616+0000", 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8589999999999999e-06 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759622+0000", 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9519999999999999e-06 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759649+0000", 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6659999999999999e-05 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759653+0000", 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9419999999999997e-06 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809440+0000", 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0497873879999999 2024-03-24T12:06:09.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809603+0000", 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016285099999999999 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.810439+0000", 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083558799999999996 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813599+0000", 2024-03-24T12:06:09.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031601060000000002 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831369+0000", 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017770563 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832331+0000", 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096128100000000003 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832805+0000", 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000473864 2024-03-24T12:06:09.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832873+0000", 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8516999999999995e-05 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832995+0000", 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122139 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216862 9.2 9:4d09947b:::benchmark_data_smithi064_148274_object91049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.745660425000001, 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0907798720000002, 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216862 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735226+0000", 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.272e-06 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735233+0000", 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.494e-06 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735246+0000", 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.277e-05 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735253+0000", 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7889999999999995e-06 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.744476+0000", 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092227190000000007 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.744486+0000", 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5070000000000006e-06 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722078+0000", 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97759208500000006 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722094+0000", 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6337999999999998e-05 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818237+0000", 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096142892 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818340+0000", 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010314099999999999 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818710+0000", 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036944499999999998 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821730+0000", 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00302075 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825432+0000", 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037018089999999999 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825705+0000", 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027328399999999999 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825805+0000", 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9975000000000004e-05 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825875+0000", 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9665999999999993e-05 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826004+0000", 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012893599999999999 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216877 9.2 9:41202982:::benchmark_data_smithi064_148274_object91064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.723846266999999, 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0695391550000002, 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.559 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216877 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757041+0000", 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4339999999999999e-06 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757048+0000", 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6369999999999999e-06 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757062+0000", 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.359e-05 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757072+0000", 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0125999999999999e-05 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757145+0000", 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2819999999999997e-05 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757158+0000", 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2721999999999999e-05 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722171+0000", 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96501314900000001 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722181+0000", 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0164e-05 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821770+0000", 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0995892380000001 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821855+0000", 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4513999999999998e-05 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821947+0000", 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2659999999999997e-05 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825725+0000", 2024-03-24T12:06:09.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037774430000000001 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825754+0000", 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9048e-05 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826079+0000", 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000325015 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826391+0000", 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031238199999999999 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826454+0000", 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2303000000000003e-05 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826577+0000", 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012391 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216879 9.2 9:4db289b8:::benchmark_data_smithi064_148274_object91066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.722318453, 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0685427669999998, 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216879 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758568+0000", 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9309999999999998e-06 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758574+0000", 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000002e-06 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758586+0000", 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2075999999999999e-05 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758595+0000", 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5930000000000006e-06 2024-03-24T12:06:09.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758682+0000", 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7328000000000001e-05 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758694+0000", 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2271e-05 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722222+0000", 2024-03-24T12:06:09.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.963528159 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722232+0000", 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2650000000000002e-06 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824044+0000", 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1018119829999999 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824123+0000", 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9806000000000001e-05 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824206+0000", 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2086e-05 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826021+0000", 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001815126 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826703+0000", 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068264900000000002 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826730+0000", 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6710999999999999e-05 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826787+0000", 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7014e-05 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826892+0000", 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105351 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827109+0000", 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021650799999999999 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216888 9.2 9:4c303ed8:::benchmark_data_smithi064_148274_object91075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.575 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.720078679, 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.079817389, 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216888 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:06:09.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760808+0000", 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9139999999999998e-06 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760813+0000", 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6699999999999999e-06 2024-03-24T12:06:09.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760826+0000", 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2533e-05 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760835+0000", 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3319999999999998e-06 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760923+0000", 2024-03-24T12:06:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7826999999999998e-05 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760940+0000", 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6396999999999999e-05 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722255+0000", 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96131583300000001 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722263+0000", 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0669999999999992e-06 2024-03-24T12:06:09.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827147+0000", 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.104883244 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827325+0000", 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001784 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827519+0000", 2024-03-24T12:06:09.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019378400000000001 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835233+0000", 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077136690000000003 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835278+0000", 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5664000000000002e-05 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835321+0000", 2024-03-24T12:06:09.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2817e-05 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840424+0000", 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051024970000000001 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840504+0000", 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0844e-05 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840623+0000", 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118897 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216890 9.2 9:4575dfdc:::benchmark_data_smithi064_148274_object91077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.716784368999999, 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0768427539999998, 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216890 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764103+0000", 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4870000000000001e-06 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764110+0000", 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2609999999999996e-06 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764123+0000", 2024-03-24T12:06:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3369e-05 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764133+0000", 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8179999999999999e-06 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764206+0000", 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2471999999999993e-05 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764244+0000", 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8655000000000002e-05 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722287+0000", 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95804261999999996 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722295+0000", 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6129999999999998e-06 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830293+0000", 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079985539999999 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830416+0000", 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012251699999999999 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830550+0000", 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013488000000000001 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835368+0000", 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048179049999999999 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840637+0000", 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005268928 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840707+0000", 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9367000000000002e-05 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840755+0000", 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8789000000000003e-05 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840813+0000", 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7757999999999998e-05 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840943+0000", 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012976099999999999 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216938 9.2 9:41fc8627:::benchmark_data_smithi064_148274_object91125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.672913882, 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.033607655, 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216938 2024-03-24T12:06:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807972+0000", 2024-03-24T12:06:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807977+0000", 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8450000000000002e-06 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807990+0000", 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3091999999999999e-05 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807998+0000", 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6769999999999995e-06 2024-03-24T12:06:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.808072+0000", 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3719000000000003e-05 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.808082+0000", 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0648999999999999e-05 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722352+0000", 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.9142692 2024-03-24T12:06:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722361+0000", 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7030000000000001e-06 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832521+0000", 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.110159882 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832692+0000", 2024-03-24T12:06:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171242 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832843+0000", 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015096200000000001 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840408+0000", 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075646669999999997 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840644+0000", 2024-03-24T12:06:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023627500000000001 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840991+0000", 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034632899999999998 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841121+0000", 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130445 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841180+0000", 2024-03-24T12:06:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8777999999999999e-05 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841578+0000", 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039849699999999999 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216940 9.2 9:46d14649:::benchmark_data_smithi064_148274_object91127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.671350273000002, 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.032468546, 2024-03-24T12:06:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216940 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809536+0000", 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.57e-06 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809541+0000", 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0470000000000001e-06 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809555+0000", 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.417e-05 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809563+0000", 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.622e-06 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809619+0000", 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.643e-05 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809631+0000", 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1477e-05 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722387+0000", 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.91275614100000002 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722395+0000", 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4279999999999996e-06 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835404+0000", 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1130083449999999 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835526+0000", 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122069 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835639+0000", 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113329 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840651+0000", 2024-03-24T12:06:09.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050119439999999999 2024-03-24T12:06:09.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841055+0000", 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000404127 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841634+0000", 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057911699999999998 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841735+0000", 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100603 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841807+0000", 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1997999999999997e-05 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842003+0000", 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019612899999999999 2024-03-24T12:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216945 9.2 9:46734ef5:::benchmark_data_smithi064_148274_object91132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.665797878999999, 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.027242598, 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216945 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.607 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815089+0000", 2024-03-24T12:06:09.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5110000000000002e-06 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815098+0000", 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815115+0000", 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833999999999998e-05 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815125+0000", 2024-03-24T12:06:09.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7019999999999996e-06 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815205+0000", 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0593999999999994e-05 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815214+0000", 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7630000000000006e-06 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722420+0000", 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.90720568000000001 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722428+0000", 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3610000000000001e-06 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837728+0000", 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.115299724 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837849+0000", 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121298 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837993+0000", 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014408300000000001 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841007+0000", 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030141429999999999 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841682+0000", 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067442900000000004 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842071+0000", 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038890700000000002 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842145+0000", 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4214999999999993e-05 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842218+0000", 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3461999999999996e-05 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842329+0000", 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110983 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216966 9.2 9:44cdb63b:::benchmark_data_smithi064_148274_object91153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.628260562000001, 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0008592350000001, 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216966 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852626+0000", 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852629+0000", 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3390000000000001e-06 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852638+0000", 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5720000000000003e-06 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852643+0000", 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5369999999999997e-06 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852719+0000", 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6086000000000003e-05 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852728+0000", 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3000000000000002e-06 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722453+0000", 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86972569 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722461+0000", 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2830000000000001e-06 2024-03-24T12:06:09.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842378+0000", 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1199161879999999 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842492+0000", 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114769 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842616+0000", 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012353200000000001 2024-03-24T12:06:09.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851293+0000", 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086773370000000002 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851656+0000", 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036241099999999998 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851701+0000", 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5457999999999997e-05 2024-03-24T12:06:09.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853267+0000", 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001566095 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853330+0000", 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2955999999999998e-05 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853483+0000", 2024-03-24T12:06:09.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153022 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216969 9.2 9:4ad9ca10:::benchmark_data_smithi064_148274_object91156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.619485560000001, 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9923648649999999, 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:09.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216969 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861400+0000", 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.438e-06 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861404+0000", 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9469999999999996e-06 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861415+0000", 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0396e-05 2024-03-24T12:06:09.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861421+0000", 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2600000000000002e-06 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861479+0000", 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7921000000000003e-05 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861486+0000", 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6359999999999997e-06 2024-03-24T12:06:09.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722487+0000", 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86100120400000002 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722495+0000", 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2300000000000008e-06 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.844900+0000", 2024-03-24T12:06:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1224051230000001 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.845026+0000", 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126062 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.845364+0000", 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000337809 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851301+0000", 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059373220000000001 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851745+0000", 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044327599999999999 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851788+0000", 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3272999999999997e-05 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853579+0000", 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001791555 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853644+0000", 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4944999999999996e-05 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853764+0000", 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119468 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:09.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:09.630 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:06:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:09.630 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:09.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:09.697 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:06:09.697 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T12:06:09.697 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:06:09.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:09.769 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:09.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:09.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:06:09.869 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:09.905 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:09.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:10.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:10.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:10.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:10.182 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:10.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:10.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T12:06:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T12:06:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T12:06:10.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T12:06:10.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T12:06:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T12:06:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T12:06:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T12:06:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T12:06:10.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:09 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T12:06:10.320 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:10.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:09 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T12:06:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:09 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T12:06:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:09 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T12:06:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:09 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T12:06:10.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:09 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T12:06:10.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:10.493 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:10.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:10.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:10.631 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:10.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:10.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:06:10.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:10.751 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:10.774 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:10.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:10.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:06:10.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:10.916 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:10.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:11.017 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:11.066 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:11.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:11.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:10 smithi193 ceph-mon[33593]: pgmap v631: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 437 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.34k op/s; 890/21558 objects misplaced (4.128%) 2024-03-24T12:06:11.088 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:11.168 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:11.195 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:11.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:11.278 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:10 smithi064 ceph-mon[42243]: pgmap v631: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 437 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.34k op/s; 890/21558 objects misplaced (4.128%) 2024-03-24T12:06:11.278 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:10 smithi064 ceph-mon[30223]: pgmap v631: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 437 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.34k op/s; 890/21558 objects misplaced (4.128%) 2024-03-24T12:06:11.296 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:11.339 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:11.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:11.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:11.477 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:11.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:11.579 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:11.621 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:11.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:11.723 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:11.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:11.764 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:11.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:11.865 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:11.899 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:11.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:12.000 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:12.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:12.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:12.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T12:06:12.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T12:06:12.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[30223]: 1.10 scrub starts 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[30223]: 1.10 scrub ok 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[42243]: 1.10 scrub starts 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[42243]: 1.10 scrub ok 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T12:06:12.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:11 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:12.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:12.175 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:12.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:12.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:12.276 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:12.305 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:12.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:11 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T12:06:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:11 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T12:06:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:11 smithi193 ceph-mon[33593]: 1.10 scrub starts 2024-03-24T12:06:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:11 smithi193 ceph-mon[33593]: 1.10 scrub ok 2024-03-24T12:06:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:11 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T12:06:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:11 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T12:06:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:11 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:12.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:12.434 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:12.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:12.511 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Removed 125810 objects 2024-03-24T12:06:12.511 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Clean up completed and total clean up time :105.491 2024-03-24T12:06:12.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:12.559 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:12.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:12.661 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:12.685 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:12.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:12.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:12.812 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:12.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:12.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:12.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:12.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:12.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:13.020 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:12 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T12:06:13.020 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:12 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T12:06:13.021 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:12 smithi193 ceph-mon[33593]: pgmap v632: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 388 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 890/19194 objects misplaced (4.637%) 2024-03-24T12:06:13.021 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:12 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T12:06:13.021 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:12 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T12:06:13.021 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:12 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T12:06:13.021 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:12 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:06:13.021 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:12 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:06:13.038 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:13.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T12:06:13.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T12:06:13.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[42243]: pgmap v632: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 388 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 890/19194 objects misplaced (4.637%) 2024-03-24T12:06:13.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T12:06:13.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T12:06:13.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T12:06:13.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:06:13.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:06:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T12:06:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T12:06:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[30223]: pgmap v632: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 388 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 890/19194 objects misplaced (4.637%) 2024-03-24T12:06:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T12:06:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T12:06:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T12:06:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:06:13.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:12 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:06:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:13.062 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:13.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:13.163 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:13.186 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:13.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:13.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:13.311 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:13.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:13.328 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:13.413 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:13.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:13.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:13.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:13.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:13.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:13.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:13.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:13.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:13.812 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:13.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:13.914 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:13.940 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:13.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:13.949 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:06:14.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:14.065 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:14.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:14.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:14.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:13 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T12:06:14.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:13 smithi193 ceph-mon[33593]: 9.1 scrub starts 2024-03-24T12:06:14.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:13 smithi193 ceph-mon[33593]: 9.1 scrub ok 2024-03-24T12:06:14.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:13 smithi193 ceph-mon[33593]: 1.14 scrub starts 2024-03-24T12:06:14.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:13 smithi193 ceph-mon[33593]: 1.14 scrub ok 2024-03-24T12:06:14.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:13 smithi193 ceph-mon[33593]: 1.e scrub starts 2024-03-24T12:06:14.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:13 smithi193 ceph-mon[33593]: 1.e scrub ok 2024-03-24T12:06:14.175 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:13 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/822443577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:14.189 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:14.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:14.194 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:06:14.194 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:06:14.195 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-03-24T12:06:14.195 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:06:14.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:14.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T12:06:14.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[30223]: 9.1 scrub starts 2024-03-24T12:06:14.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[30223]: 9.1 scrub ok 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[30223]: 1.14 scrub starts 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[30223]: 1.14 scrub ok 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[30223]: 1.e scrub starts 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[30223]: 1.e scrub ok 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/822443577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[42243]: 9.1 scrub starts 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[42243]: 9.1 scrub ok 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[42243]: 1.14 scrub starts 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[42243]: 1.14 scrub ok 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[42243]: 1.e scrub starts 2024-03-24T12:06:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[42243]: 1.e scrub ok 2024-03-24T12:06:14.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:13 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/822443577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:14.316 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:14.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:14.418 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:14.442 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:14.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:14.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:14.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:14.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:14.668 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:14.692 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:14.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:14.794 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:14.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:14.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:14.863 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:06:14.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:14.945 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:14.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:15.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:15.074 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:15.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:15.097 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:15.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:15.200 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:15.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:15.220 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:14 smithi193 ceph-mon[33593]: 4.11 scrub starts 2024-03-24T12:06:15.220 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:14 smithi193 ceph-mon[33593]: 4.11 scrub ok 2024-03-24T12:06:15.220 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:14 smithi193 ceph-mon[33593]: pgmap v633: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 44 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 74/2673 objects misplaced (2.768%) 2024-03-24T12:06:15.220 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:14 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T12:06:15.220 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:14 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T12:06:15.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:14 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:06:15.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/822443577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:15.221 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:14 smithi193 ceph-mon[33593]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T12:06:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[30223]: 4.11 scrub starts 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[30223]: 4.11 scrub ok 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[30223]: pgmap v633: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 44 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 74/2673 objects misplaced (2.768%) 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/822443577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[30223]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[42243]: 4.11 scrub starts 2024-03-24T12:06:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[42243]: 4.11 scrub ok 2024-03-24T12:06:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[42243]: pgmap v633: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 44 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 74/2673 objects misplaced (2.768%) 2024-03-24T12:06:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T12:06:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T12:06:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:06:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/822443577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:15.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:14 smithi064 ceph-mon[42243]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T12:06:15.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:15.327 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:15.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:15.420 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:06:15.420 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:06:15.420 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-24T12:06:15.420 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T12:06:15.420 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T12:06:15.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-24T12:06:15.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:15.455 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:15.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:15.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:15.580 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:15.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:15.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:15.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:15.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:15.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:15.830 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:15.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:15.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:15.959 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:15.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:06:15.972 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:15 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:06:15.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:15 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:06:15.973 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:15 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2870026117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:15.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:15 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:06:15.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:15 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:06:15.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:15 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2870026117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:06:16.013 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:06:16.031 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:06:16.052 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:06:16.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:16.072 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:06:16.086 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:16.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:16.092 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:06:16.113 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:06:16.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:06:16.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:06:16.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:06:16.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:06:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:06:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:06:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:06:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:06:16.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:16.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:16.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:15 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:06:16.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:15 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:06:16.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:15 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2870026117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:16.300 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:16.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:16.334 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:16.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:16.436 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:16.459 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:16.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:16.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:16.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:16.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:06:16.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:16.709 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:16.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:16.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:16.834 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:16.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:16.932 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:16.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:16.966 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:16.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:17.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:17.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:16 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T12:06:17.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:16 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T12:06:17.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:16 smithi064 ceph-mon[30223]: pgmap v635: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 44 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 1.01k op/s; 74/2673 objects misplaced (2.768%) 2024-03-24T12:06:17.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:16 smithi064 ceph-mon[30223]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T12:06:17.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:16 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T12:06:17.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:16 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T12:06:17.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:16 smithi064 ceph-mon[42243]: pgmap v635: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 44 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 1.01k op/s; 74/2673 objects misplaced (2.768%) 2024-03-24T12:06:17.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:16 smithi064 ceph-mon[42243]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T12:06:17.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:16 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T12:06:17.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:16 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T12:06:17.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:16 smithi193 ceph-mon[33593]: pgmap v635: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 44 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 1.01k op/s; 74/2673 objects misplaced (2.768%) 2024-03-24T12:06:17.087 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:16 smithi193 ceph-mon[33593]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T12:06:17.092 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:17.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:17.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:17.218 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:17.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:17.301 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-24T12:06:17.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-24T12:06:17.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:17.367 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:17.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:17.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:17.492 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:17.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:17.497 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:06:17.497 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:06:17.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:17.617 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:17.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:17.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:17.743 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:17.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:17.790 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:06:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:06:17.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:06:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:06:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:17.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:17.869 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:17.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:17.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:17.995 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:17.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:18.082 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:17 smithi193 ceph-mon[33593]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T12:06:18.096 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:18.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:06:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:18.121 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:18.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:18.142 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:17 smithi064 ceph-mon[30223]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T12:06:18.142 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:17 smithi064 ceph-mon[42243]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T12:06:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:18.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:18.245 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:18.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:18.346 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:18.369 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:18.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:18.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:18.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:18.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:18.621 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:18.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:18.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:18.746 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:18.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:18.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:18.871 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:18.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:18.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:06:18.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:18.998 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:18.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:19.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:19.123 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:19.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:19.170 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:06:19.170 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "" 2024-03-24T12:06:19.170 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:06:19.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:18 smithi193 ceph-mon[33593]: pgmap v638: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:06:19.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:19.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:18 smithi193 ceph-mon[33593]: 6.13 scrub starts 2024-03-24T12:06:19.170 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:18 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:06:19.224 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:19.242 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:18 smithi064 ceph-mon[30223]: pgmap v638: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:06:19.242 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:19.242 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:18 smithi064 ceph-mon[30223]: 6.13 scrub starts 2024-03-24T12:06:19.242 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:18 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:06:19.242 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:18 smithi064 ceph-mon[42243]: pgmap v638: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:06:19.242 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:19.242 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:18 smithi064 ceph-mon[42243]: 6.13 scrub starts 2024-03-24T12:06:19.242 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:18 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:06:19.247 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:19.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:19.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:19.373 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:19.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:19.475 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:19.498 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:19.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:19.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:19.622 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:19.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:19.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:19.748 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:19.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:19.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:19.864 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:19.873 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:19.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:19.930 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:19.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:06:19.999 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:20.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:20.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:20.119 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:19 smithi193 ceph-mon[33593]: 6.13 scrub ok 2024-03-24T12:06:20.119 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:19 smithi193 ceph-mon[33593]: 1.5 scrub starts 2024-03-24T12:06:20.119 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:19 smithi193 ceph-mon[33593]: 1.5 scrub ok 2024-03-24T12:06:20.119 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:19 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2279579307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:20.125 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:20.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:20.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:19 smithi064 ceph-mon[30223]: 6.13 scrub ok 2024-03-24T12:06:20.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:19 smithi064 ceph-mon[30223]: 1.5 scrub starts 2024-03-24T12:06:20.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:19 smithi064 ceph-mon[30223]: 1.5 scrub ok 2024-03-24T12:06:20.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:19 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2279579307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:20.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:19 smithi064 ceph-mon[42243]: 6.13 scrub ok 2024-03-24T12:06:20.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:19 smithi064 ceph-mon[42243]: 1.5 scrub starts 2024-03-24T12:06:20.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:19 smithi064 ceph-mon[42243]: 1.5 scrub ok 2024-03-24T12:06:20.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:19 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2279579307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:20.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:20.250 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:20.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:20.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:20.375 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:20.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:20.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:20.500 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:20.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:20.602 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:20.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:20.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:20.752 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:20.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:20.854 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:20.877 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:20.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:20.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:21.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:21.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:21.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:21.105 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:20 smithi193 ceph-mon[33593]: pgmap v639: 168 pgs: 1 peering, 167 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 116 op/s 2024-03-24T12:06:21.105 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:20 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T12:06:21.105 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:20 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T12:06:21.105 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:20 smithi193 ceph-mon[33593]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:06:21.105 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:20 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:06:21.105 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:20 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:21.105 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:20 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2279579307' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:06:21.105 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:20 smithi193 ceph-mon[33593]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T12:06:21.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:21.129 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:21.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:21.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:21.253 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[30223]: pgmap v639: 168 pgs: 1 peering, 167 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 116 op/s 2024-03-24T12:06:21.253 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[30223]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2279579307' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[30223]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[42243]: pgmap v639: 168 pgs: 1 peering, 167 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 116 op/s 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[42243]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:06:21.254 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:06:21.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:21.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2279579307' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:06:21.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:20 smithi064 ceph-mon[42243]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T12:06:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:21.256 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:21.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:21.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:21.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:21.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:21.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:21.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:21.631 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:21.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:21.643 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:06:21.643 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:06:21.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:21.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:21.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:21.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:21.883 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:21.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:21.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:22.001 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_2' removed 2024-03-24T12:06:22.045 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:22.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:22.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:22.172 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:22.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:22.229 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1913001469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:06:22.229 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:21 smithi193 ceph-mon[33593]: from='client.35516 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:06:22.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:06:22.273 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:22.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1913001469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:06:22.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:21 smithi064 ceph-mon[30223]: from='client.35516 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:06:22.278 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1913001469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:06:22.278 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:21 smithi064 ceph-mon[42243]: from='client.35516 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:06:22.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:22.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:22.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:22.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:22.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:22.579 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:22.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:22.613 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:22.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:22.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:22.739 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:22.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:22.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:22.863 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:22.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:22.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:22.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:23.038 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:22 smithi064 ceph-mon[30223]: pgmap v641: 168 pgs: 1 peering, 167 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 117 op/s 2024-03-24T12:06:23.039 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:22 smithi064 ceph-mon[30223]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:06:23.039 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:22 smithi064 ceph-mon[30223]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T12:06:23.039 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:22 smithi064 ceph-mon[42243]: pgmap v641: 168 pgs: 1 peering, 167 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 117 op/s 2024-03-24T12:06:23.039 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:22 smithi064 ceph-mon[42243]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:06:23.039 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:22 smithi064 ceph-mon[42243]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T12:06:23.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:23.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:23.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:23.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:23.240 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:23.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:23.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:22 smithi193 ceph-mon[33593]: pgmap v641: 168 pgs: 1 peering, 167 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 117 op/s 2024-03-24T12:06:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:22 smithi193 ceph-mon[33593]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:06:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:22 smithi193 ceph-mon[33593]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T12:06:23.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:23.366 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:23.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:23.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:23.490 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:23.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:23.592 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:23.616 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:23.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:23.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:23.740 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:23.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:23.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:23.865 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:23.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:23.870 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T12:06:23.871 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T12:06:23.871 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-24T12:06:23.871 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-24T12:06:23.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:23.991 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:23.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:24.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:24.120 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:24.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:24.135 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:06:24.156 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:06:24.171 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:24.177 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:06:24.200 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:06:24.220 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:06:24.222 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:24.239 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:06:24.246 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:24.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:24.259 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:06:24.280 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:06:24.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:06:24.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:06:24.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:06:24.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:06:24.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:06:24.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:06:24.282 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:06:24.282 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:06:24.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:06:24.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:24.353 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:06:24.354 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:06:24.354 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-03-24T12:06:24.354 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:06:24.387 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:24.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:24.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:24.516 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:24.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:24.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:06:24.643 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:24.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:06:24.745 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:24.769 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:24.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:24.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:24.870 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:06:24.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:06:24.870 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:06:24.870 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T12:06:24.870 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.5 2024-03-24T12:06:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:24.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:24.969 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:24 smithi193 systemd[1]: Stopping Ceph osd.5 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:06:24.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:25.001 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T12:06:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:25.023 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:25.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:25.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:25.149 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:25.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:25.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:25.275 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:25.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:25.279 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:25.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:24 smithi064 ceph-mon[30223]: pgmap v643: 152 pgs: 152 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-03-24T12:06:25.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:24 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3853113555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:25.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:24 smithi064 ceph-mon[42243]: pgmap v643: 152 pgs: 152 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-03-24T12:06:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:24 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3853113555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:25.313 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:24 smithi193 ceph-mon[33593]: pgmap v643: 152 pgs: 152 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-03-24T12:06:25.313 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:24 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3853113555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:25.314 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:25 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[45137]: 2024-03-24T12:06:25.210+0000 7f9d08e0b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:06:25.314 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:25 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[45137]: 2024-03-24T12:06:25.210+0000 7f9d08e0b700 -1 osd.5 317 *** Got signal Terminated *** 2024-03-24T12:06:25.314 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:25 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[45137]: 2024-03-24T12:06:25.210+0000 7f9d08e0b700 -1 osd.5 317 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:06:25.377 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:25.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:25.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:25.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:25.530 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:25.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:25.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:25.656 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:25.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:25.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:25.781 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:25.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:25.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:25.902 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:25.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:26.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:26.032 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:26.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:26.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:25 smithi193 ceph-mon[33593]: osd.5 marked itself down and dead 2024-03-24T12:06:26.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:26.157 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:26.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:26.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:26.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:25 smithi064 ceph-mon[30223]: osd.5 marked itself down and dead 2024-03-24T12:06:26.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:25 smithi064 ceph-mon[42243]: osd.5 marked itself down and dead 2024-03-24T12:06:26.282 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:26.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:26.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:26.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:26.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:26.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:26.532 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:26.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:26.634 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:26.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:26.659 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:26.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:26.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:26.787 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:26.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:26.814 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:26 smithi193 podman[124532]: 2024-03-24 12:06:26.495589409 +0000 UTC m=+1.399098183 container died 70d23e3170c067977dbb3bad3441066c944705dcb2d5c9475c0d7bebaced3ff9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:06:26.888 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:26.912 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:26.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:27.013 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:27.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:27.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:27.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:26 smithi193 ceph-mon[33593]: pgmap v644: 152 pgs: 152 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:06:27.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:26 smithi193 ceph-mon[33593]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:06:27.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:26 smithi193 ceph-mon[33593]: osdmap e318: 8 total, 7 up, 8 in 2024-03-24T12:06:27.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:27.086 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:26 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:27.086 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:26 smithi193 podman[124532]: 2024-03-24 12:06:26.92207409 +0000 UTC m=+1.825582855 container cleanup 70d23e3170c067977dbb3bad3441066c944705dcb2d5c9475c0d7bebaced3ff9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T12:06:27.086 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:26 smithi193 bash[124532]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5 2024-03-24T12:06:27.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:27.163 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:27.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:27.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[42243]: pgmap v644: 152 pgs: 152 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:06:27.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[42243]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:06:27.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[42243]: osdmap e318: 8 total, 7 up, 8 in 2024-03-24T12:06:27.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:27.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:27.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[30223]: pgmap v644: 152 pgs: 152 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:06:27.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[30223]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:06:27.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[30223]: osdmap e318: 8 total, 7 up, 8 in 2024-03-24T12:06:27.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:27.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:26 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:27.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:27.287 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:27.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:27.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:06:27.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:27.415 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:27.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:27.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:27.530 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:27 smithi193 podman[124736]: 2024-03-24 12:06:27.248978315 +0000 UTC m=+0.749737397 container remove 70d23e3170c067977dbb3bad3441066c944705dcb2d5c9475c0d7bebaced3ff9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:06:27.540 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:27.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:27.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:06:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:06:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:27.642 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:27.667 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:27.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:27.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:27.784 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:06:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:06:27.787 INFO:teuthology.orchestra.run.smithi193.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-24T12:06:27.793 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:27.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:27.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:27.918 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:27.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:28.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:28.044 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:28.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:28.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:27 smithi193 ceph-mon[33593]: osdmap e319: 8 total, 7 up, 8 in 2024-03-24T12:06:28.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:27 smithi064 ceph-mon[30223]: osdmap e319: 8 total, 7 up, 8 in 2024-03-24T12:06:28.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:27 smithi064 ceph-mon[42243]: osdmap e319: 8 total, 7 up, 8 in 2024-03-24T12:06:28.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:28.169 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:28.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:28.270 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:28.294 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:28.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:28.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:28.420 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:28.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:06:28.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T12:06:28.521 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:28.546 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:28.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:28.605 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:28.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:28.671 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:28.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:28.729 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:28.774 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:28.798 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:28.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:28.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:28.923 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:28.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:29.013 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:06:29.014 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_3' created 2024-03-24T12:06:29.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:29.050 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:29.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:29.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:29.175 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:29.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:29.216 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:28 smithi193 podman[125122]: 2024-03-24 12:06:28.767623476 +0000 UTC m=+0.019805411 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:06:29.216 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:29 smithi193 podman[125122]: 2024-03-24 12:06:29.084943491 +0000 UTC m=+0.337125427 container create a4bd0318756c5d98e7854310c8905e888173519d60877a2853385e462c2401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T12:06:29.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:28 smithi193 ceph-mon[33593]: pgmap v647: 152 pgs: 2 active+undersized+degraded, 32 peering, 12 stale+active+clean, 3 active+undersized, 103 active+clean; 457 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7 op/s; 2/594 objects degraded (0.337%); 66 KiB/s, 0 objects/s recovering 2024-03-24T12:06:29.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:29.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:28 smithi193 ceph-mon[33593]: Health check failed: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-24T12:06:29.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:28 smithi193 ceph-mon[33593]: Health check failed: Degraded data redundancy: 2/594 objects degraded (0.337%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:29.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:28 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3151844036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:06:29.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:28 smithi193 ceph-mon[33593]: from='client.25531 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:06:29.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:28 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/379400272' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:29.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:28 smithi193 ceph-mon[33593]: from='client.35534 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[30223]: pgmap v647: 152 pgs: 2 active+undersized+degraded, 32 peering, 12 stale+active+clean, 3 active+undersized, 103 active+clean; 457 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7 op/s; 2/594 objects degraded (0.337%); 66 KiB/s, 0 objects/s recovering 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[30223]: Health check failed: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[30223]: Health check failed: Degraded data redundancy: 2/594 objects degraded (0.337%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3151844036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[30223]: from='client.25531 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/379400272' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[30223]: from='client.35534 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:29.254 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[42243]: pgmap v647: 152 pgs: 2 active+undersized+degraded, 32 peering, 12 stale+active+clean, 3 active+undersized, 103 active+clean; 457 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7 op/s; 2/594 objects degraded (0.337%); 66 KiB/s, 0 objects/s recovering 2024-03-24T12:06:29.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:29.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[42243]: Health check failed: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-24T12:06:29.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[42243]: Health check failed: Degraded data redundancy: 2/594 objects degraded (0.337%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:29.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3151844036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:06:29.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[42243]: from='client.25531 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:06:29.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/379400272' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:29.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:28 smithi064 ceph-mon[42243]: from='client.35534 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:29.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:29.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:29.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:29.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:29.426 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:29.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:29.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:29.552 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:29.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:29.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:29.676 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:29.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:29.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:29.803 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:29.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:29.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:29.927 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:29.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:30.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:30.052 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:30.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:30.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:30.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:30.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:30.257 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:30 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:30.257 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:30 smithi193 ceph-mon[33593]: from='client.25531 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T12:06:30.257 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:30 smithi193 ceph-mon[33593]: from='client.35534 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:30.257 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:30 smithi193 ceph-mon[33593]: osdmap e320: 8 total, 7 up, 8 in 2024-03-24T12:06:30.258 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:30 smithi193 podman[125122]: 2024-03-24 12:06:30.150439356 +0000 UTC m=+1.402621281 container init a4bd0318756c5d98e7854310c8905e888173519d60877a2853385e462c2401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T12:06:30.258 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:30 smithi193 podman[125122]: 2024-03-24 12:06:30.156559296 +0000 UTC m=+1.408741228 container start a4bd0318756c5d98e7854310c8905e888173519d60877a2853385e462c2401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-24T12:06:30.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:30 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:30 smithi064 ceph-mon[30223]: from='client.25531 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T12:06:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:30 smithi064 ceph-mon[30223]: from='client.35534 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:30 smithi064 ceph-mon[30223]: osdmap e320: 8 total, 7 up, 8 in 2024-03-24T12:06:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:30 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:30 smithi064 ceph-mon[42243]: from='client.25531 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T12:06:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:30 smithi064 ceph-mon[42243]: from='client.35534 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:30.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:30 smithi064 ceph-mon[42243]: osdmap e320: 8 total, 7 up, 8 in 2024-03-24T12:06:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:30.303 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:30.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:30.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:30.427 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:30.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:30.514 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:30 smithi193 podman[125122]: 2024-03-24 12:06:30.258696153 +0000 UTC m=+1.510878094 container attach a4bd0318756c5d98e7854310c8905e888173519d60877a2853385e462c2401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T12:06:30.514 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:06:30 smithi193 podman[125122]: 2024-03-24 12:06:30.499859622 +0000 UTC m=+1.752041550 container died a4bd0318756c5d98e7854310c8905e888173519d60877a2853385e462c2401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T12:06:30.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:30.552 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:30.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:30.654 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:30.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:30.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:30.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:30.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:30.906 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:30.931 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:30.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:31.032 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:31.056 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:31.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:31.095 INFO:teuthology.orchestra.run.smithi193.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-24T12:06:31.158 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:31.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:31.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:31.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:31.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:31 smithi064 ceph-mon[30223]: pgmap v649: 168 pgs: 16 unknown, 8 active+undersized+degraded, 32 peering, 23 active+undersized, 89 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 110 KiB/s wr, 17 op/s; 30/594 objects degraded (5.051%); 79 KiB/s, 0 objects/s recovering 2024-03-24T12:06:31.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:31 smithi064 ceph-mon[30223]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:06:31.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:31 smithi064 ceph-mon[30223]: osdmap e321: 8 total, 7 up, 8 in 2024-03-24T12:06:31.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:31 smithi064 ceph-mon[42243]: pgmap v649: 168 pgs: 16 unknown, 8 active+undersized+degraded, 32 peering, 23 active+undersized, 89 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 110 KiB/s wr, 17 op/s; 30/594 objects degraded (5.051%); 79 KiB/s, 0 objects/s recovering 2024-03-24T12:06:31.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:31 smithi064 ceph-mon[42243]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:06:31.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:31 smithi064 ceph-mon[42243]: osdmap e321: 8 total, 7 up, 8 in 2024-03-24T12:06:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:31.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:31 smithi193 ceph-mon[33593]: pgmap v649: 168 pgs: 16 unknown, 8 active+undersized+degraded, 32 peering, 23 active+undersized, 89 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 110 KiB/s wr, 17 op/s; 30/594 objects degraded (5.051%); 79 KiB/s, 0 objects/s recovering 2024-03-24T12:06:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:31 smithi193 ceph-mon[33593]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:06:31.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:31 smithi193 ceph-mon[33593]: osdmap e321: 8 total, 7 up, 8 in 2024-03-24T12:06:31.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:31.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:31.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:31.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-24T12:06:31.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:31.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:31.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:31.685 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:31.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:06:31.698 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:06:31.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:31.811 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:31.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:31.912 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:31.936 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:31.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:32.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:32.061 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:32.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:32.149 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T12:06:32.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:32.186 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:32.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:32.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:06:32.201 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T12:06:32.201 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-03-24T12:06:32.201 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi193.front.sepia.ceph.com 2024-03-24T12:06:32.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-03-24T12:06:32.272 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:32.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:32.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:32 smithi064 ceph-mon[30223]: osdmap e322: 8 total, 7 up, 8 in 2024-03-24T12:06:32.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:32 smithi064 ceph-mon[42243]: osdmap e322: 8 total, 7 up, 8 in 2024-03-24T12:06:32.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:32.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:32.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:32 smithi193 ceph-mon[33593]: osdmap e322: 8 total, 7 up, 8 in 2024-03-24T12:06:32.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:32.438 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:32.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:32.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:32.564 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:32.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:32.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:32.576 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:06:32.592 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:32.601 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:06:32.622 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:06:32.643 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:06:32.662 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:06:32.665 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:32.683 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:06:32.691 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:32.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:32.703 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:06:32.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:06:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:06:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:06:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:06:32.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:06:32.724 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:06:32.724 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:06:32.724 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:06:32.724 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:06:32.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:32.816 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:32.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:32.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:32.941 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:32.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:33.043 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:33.067 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:33.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:33.142 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:33 smithi064 ceph-mon[30223]: pgmap v652: 168 pgs: 8 unknown, 12 active+undersized+degraded, 14 peering, 40 active+undersized, 94 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 124 KiB/s wr, 20 op/s; 44/594 objects degraded (7.407%); 89 KiB/s, 0 objects/s recovering 2024-03-24T12:06:33.142 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:33 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4050534505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:33.143 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:33 smithi064 ceph-mon[42243]: pgmap v652: 168 pgs: 8 unknown, 12 active+undersized+degraded, 14 peering, 40 active+undersized, 94 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 124 KiB/s wr, 20 op/s; 44/594 objects degraded (7.407%); 89 KiB/s, 0 objects/s recovering 2024-03-24T12:06:33.143 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:33 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4050534505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:33.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:33.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:33.275 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:33 smithi193 ceph-mon[33593]: pgmap v652: 168 pgs: 8 unknown, 12 active+undersized+degraded, 14 peering, 40 active+undersized, 94 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 124 KiB/s wr, 20 op/s; 44/594 objects degraded (7.407%); 89 KiB/s, 0 objects/s recovering 2024-03-24T12:06:33.275 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:33 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4050534505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:33.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:33.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:33.318 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:33.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:33.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:33.442 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:33.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:33.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:06:33.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:06:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:33.568 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:33.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:33.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:33.692 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:33.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:33.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:33.817 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:33.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:33.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:06:33.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:33.970 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:33.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:34.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:34.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:34 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 67/594 objects degraded (11.279%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:34.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:34 smithi193 ceph-mon[33593]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-03-24T12:06:34.096 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:34.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:34.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:34.223 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:34.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:34.231 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:34 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 67/594 objects degraded (11.279%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:34.232 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:34 smithi064 ceph-mon[30223]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-03-24T12:06:34.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:34 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 67/594 objects degraded (11.279%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:34.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:34 smithi064 ceph-mon[42243]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-03-24T12:06:34.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:34.348 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:34.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:34.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:34.473 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:34.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:34.501 INFO:teuthology.orchestra.run.smithi193.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:06:34.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:34.586 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:34.599 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:34.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:34.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:34.727 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:34.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:34.828 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:34.853 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:34.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:34.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:34.977 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:34.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:35.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:35.102 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:35.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:35.197 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:06:35.197 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:06:35.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:35.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:35.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:35.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:35.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:35 smithi193 ceph-mon[33593]: pgmap v653: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.8 KiB/s rd, 18 KiB/s wr, 3 op/s; 67/594 objects degraded (11.279%); 13 KiB/s, 0 objects/s recovering 2024-03-24T12:06:35.352 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:35.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:35.454 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:35.478 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:35.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:35 smithi064 ceph-mon[30223]: pgmap v653: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.8 KiB/s rd, 18 KiB/s wr, 3 op/s; 67/594 objects degraded (11.279%); 13 KiB/s, 0 objects/s recovering 2024-03-24T12:06:35.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:35 smithi064 ceph-mon[42243]: pgmap v653: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.8 KiB/s rd, 18 KiB/s wr, 3 op/s; 67/594 objects degraded (11.279%); 13 KiB/s, 0 objects/s recovering 2024-03-24T12:06:35.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:06:35.580 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:35.605 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:35.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:35.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:35.730 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:35.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:35.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:35.855 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:35.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:35.877 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:06:35.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:35.984 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:35.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:36.086 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:36.110 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:36.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:36.129 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:36.212 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:36.236 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:36.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:36.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:36.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:36.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:36.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:36.486 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:36.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:36.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:36.612 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:36.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:36.686 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:36.714 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:36.739 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:36.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:36.840 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:36.864 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:36.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:36.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:36.990 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:36.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:37.078 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:06:37.088 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-24T12:06:37.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:37.119 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:37.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:37.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-mon[30223]: pgmap v654: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 799 B/s rd, 0 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:37.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/972123080' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:06:37.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-mon[30223]: from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:06:37.192 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/166963414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:06:37.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-mon[42243]: pgmap v654: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 799 B/s rd, 0 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:37.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/972123080' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:06:37.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-mon[42243]: from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:06:37.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/166963414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:06:37.220 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:37.246 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:37.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:37.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:37 smithi193 ceph-mon[33593]: pgmap v654: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 799 B/s rd, 0 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:37.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:37 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/972123080' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:06:37.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:37 smithi193 ceph-mon[33593]: from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:06:37.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:37 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/166963414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:06:37.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:37.371 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:37.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:37.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:37.497 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:37.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:37.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:06:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:06:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:37.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:06:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:06:37.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:37.623 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:37.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:37.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:37.749 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:37.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:37.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:37.874 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:37.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:37.894 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:06:37.894 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:06:37.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:38.000 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:38.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:38.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:38.125 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:38.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:38.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:38.249 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:38 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:38.250 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:38 smithi193 ceph-mon[33593]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:06:38.250 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/166963414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:06:38.250 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:38 smithi193 ceph-mon[33593]: osdmap e323: 8 total, 7 up, 8 in 2024-03-24T12:06:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:38.252 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:38.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:38.353 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:38.377 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:38.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:38.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:38 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:38.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:38 smithi064 ceph-mon[30223]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:06:38.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/166963414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:06:38.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:38 smithi064 ceph-mon[30223]: osdmap e323: 8 total, 7 up, 8 in 2024-03-24T12:06:38.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:38 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:38.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:38 smithi064 ceph-mon[42243]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:06:38.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/166963414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:06:38.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:38 smithi064 ceph-mon[42243]: osdmap e323: 8 total, 7 up, 8 in 2024-03-24T12:06:38.478 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:38.502 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:38.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:38.587 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:06:38.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:38.627 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:38.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:38.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:38.752 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:38.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:38.839 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:06:38.854 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:38.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:38.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:38.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:39.005 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:39.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:39.107 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:39.133 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:39.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:39.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:39.258 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:39.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:39.286 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:39 smithi193 ceph-mon[33593]: pgmap v656: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:39.286 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:39 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:39.287 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:39 smithi193 ceph-mon[33593]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:06:39.287 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:39 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:06:39.287 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:39 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:06:39.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[42243]: pgmap v656: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:39.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:39.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[42243]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:06:39.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:06:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:06:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[30223]: pgmap v656: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[30223]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:06:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:06:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:39 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:06:39.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:39.382 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:39.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:39.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:39.500 DEBUG:teuthology.orchestra.run.smithi064:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-24T12:06:39.502 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T12:06:39.502 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T12:06:39.508 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:39.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:39.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:39.632 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:39.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:39.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:hints = 1 2024-03-24T12:06:39.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T12:06:39.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object prefix: benchmark_data_smithi064_174908 2024-03-24T12:06:39.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:39.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:06:39.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 0 0 0 0 0 0 - 0 2024-03-24T12:06:39.781 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:39.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:39.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:39.923 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:39.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:6.e 2024-03-24T12:06:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:6.c 2024-03-24T12:06:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:6.4 2024-03-24T12:06:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:6.1e 2024-03-24T12:06:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:6.7 2024-03-24T12:06:39.945 INFO:teuthology.orchestra.run.smithi193.stdout:6.11 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:6.17 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:6.10 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.f 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.e 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:6.1d 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.d 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.b 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.7 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:6.19 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.4 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.3 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:6.a 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.2 2024-03-24T12:06:39.946 INFO:teuthology.orchestra.run.smithi193.stdout:5.1b 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:5.19 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:2.17 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:3.2 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:6.0 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:5.16 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:4.3 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:2.12 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:4.7 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:1.f 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:1.9 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:6.d 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:2.15 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:1.8 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:6.1f 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:1.10 2024-03-24T12:06:39.947 INFO:teuthology.orchestra.run.smithi193.stdout:1.a 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:1.5 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:1.b 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:1.4 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:4.1f 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:6.1c 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:4.4 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:2.14 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:1.11 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:1.1 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:1.12 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:4.19 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:6.1a 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:2.18 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:2.1c 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:2.2 2024-03-24T12:06:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:2.1e 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:2.3 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:2.b 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:2.c 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:4.15 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:4.1 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:5.6 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:4.2 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:4.8 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:5.12 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:2.16 2024-03-24T12:06:39.949 INFO:teuthology.orchestra.run.smithi193.stdout:5.15 2024-03-24T12:06:40.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:40.061 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:40.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:40.157 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:06:40.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:40.198 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:40.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:06:40.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:40 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:06:40.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:40 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:06:40.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:40 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T12:06:40.217 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:40 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T12:06:40.242 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:06:40.285 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:06:40.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:40 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:06:40.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:40 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:06:40.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:40 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T12:06:40.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:40 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T12:06:40.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:40 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:06:40.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:40 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:06:40.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:40 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T12:06:40.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:40 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T12:06:40.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:40.329 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:06:40.339 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:40.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:40.370 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:06:40.412 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:06:40.413 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:06:40.413 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:06:40.413 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:06:40.413 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:06:40.413 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:06:40.413 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:06:40.413 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:06:40.413 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:06:40.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:40.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:40.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:40.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:40.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:40.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:40.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 1 16 3737 3721 232.549 232.562 0.0061947 0.00428707 2024-03-24T12:06:40.754 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:40.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:40.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:40.895 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:40.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:40.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:40.907 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221103 7.11 7:8c18531b:::benchmark_data_smithi064_87013_object100146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.064318+0000", 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.66729006399999, 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.992514813, 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221103 2024-03-24T12:06:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064318+0000", 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064318+0000", 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064320+0000", 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8020000000000001e-06 2024-03-24T12:06:40.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064325+0000", 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7200000000000003e-06 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064335+0000", 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6320000000000003e-06 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064342+0000", 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7100000000000001e-06 2024-03-24T12:06:40.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064389+0000", 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7324999999999998e-05 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.064395+0000", 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6459999999999998e-06 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.052960+0000", 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.98856554500000005 2024-03-24T12:06:40.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.053057+0000", 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6861999999999995e-05 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.053258+0000", 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020089399999999999 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.054391+0000", 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0011329230000000001 2024-03-24T12:06:40.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056626+0000", 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022347259999999998 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056731+0000", 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010556 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056792+0000", 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0520999999999999e-05 2024-03-24T12:06:40.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056833+0000", 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0946999999999999e-05 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221132 7.11 7:8da10ffd:::benchmark_data_smithi064_87013_object100175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.088744+0000", 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.64286409399995, 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.97044713199999999, 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.914 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221132 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088744+0000", 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088744+0000", 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088746+0000", 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7089999999999998e-06 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088752+0000", 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2870000000000004e-06 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088763+0000", 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1039000000000001e-05 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088771+0000", 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0250000000000002e-06 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088853+0000", 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1977000000000005e-05 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.088863+0000", 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9310000000000003e-06 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.054709+0000", 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96584587899999996 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.054807+0000", 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8837000000000003e-05 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.054994+0000", 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000186906 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056868+0000", 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018733840000000001 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056880+0000", 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1989e-05 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059102+0000", 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022221060000000002 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059145+0000", 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3281000000000001e-05 2024-03-24T12:06:40.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059191+0000", 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5782000000000003e-05 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221143 7.11 7:8a258abe:::benchmark_data_smithi064_87013_object100186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.096184+0000", 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.63542387099994, 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96315127599999995, 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.920 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221143 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096184+0000", 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096184+0000", 2024-03-24T12:06:40.921 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096186+0000", 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6729999999999999e-06 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096190+0000", 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3560000000000003e-06 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096201+0000", 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.078e-05 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096209+0000", 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3340000000000002e-06 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096251+0000", 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1390999999999997e-05 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.096258+0000", 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3130000000000001e-06 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.055724+0000", 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95946589400000004 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.055837+0000", 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011359800000000001 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.055947+0000", 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000109696 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059053+0000", 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003105694 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059236+0000", 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018369099999999999 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059272+0000", 2024-03-24T12:06:40.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5809999999999998e-05 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059304+0000", 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1658000000000001e-05 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059335+0000", 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1387999999999998e-05 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221150 7.11 7:8e576a6c:::benchmark_data_smithi064_87013_object100193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.100516+0000", 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.63109210000005, 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95903160200000004, 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221150 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100516+0000", 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100516+0000", 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100518+0000", 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5459999999999998e-06 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100523+0000", 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0699999999999997e-06 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100536+0000", 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3057999999999999e-05 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100544+0000", 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9580000000000007e-06 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100608+0000", 2024-03-24T12:06:40.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3167e-05 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.100616+0000", 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0539999999999995e-06 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.056914+0000", 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.956298431 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.057046+0000", 2024-03-24T12:06:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000132352 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.057209+0000", 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016295099999999999 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059201+0000", 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00199175 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059379+0000", 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000178207 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059444+0000", 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4416000000000001e-05 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059501+0000", 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7535999999999999e-05 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059547+0000", 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6106000000000002e-05 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.932 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221183 7.11 7:8a724717:::benchmark_data_smithi064_87013_object100226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.130532+0000", 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.60107611700005, 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92921261700000002, 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221183 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130532+0000", 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130532+0000", 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130534+0000", 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9659999999999999e-06 2024-03-24T12:06:40.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130539+0000", 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1340000000000003e-06 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130551+0000", 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2312e-05 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130559+0000", 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5480000000000004e-06 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130614+0000", 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5717999999999997e-05 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.130622+0000", 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1709999999999996e-06 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.058069+0000", 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92744775800000001 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.058145+0000", 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5464999999999997e-05 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.058320+0000", 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00017536400000000001 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059343+0000", 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001023045 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059610+0000", 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00026698899999999997 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059652+0000", 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1705000000000001e-05 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059706+0000", 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4042000000000002e-05 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059744+0000", 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8399999999999998e-05 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221191 7.11 7:8949aaf4:::benchmark_data_smithi064_87013_object100234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.140691+0000", 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.59091727600003, 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92540493999999995, 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.939 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221191 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140691+0000", 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140691+0000", 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140692+0000", 2024-03-24T12:06:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.395e-06 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140697+0000", 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6800000000000001e-06 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140714+0000", 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6946999999999999e-05 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140721+0000", 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8970000000000008e-06 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140779+0000", 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7873e-05 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.140798+0000", 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8722000000000001e-05 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059783+0000", 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91898496299999999 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059873+0000", 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9926999999999995e-05 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.059994+0000", 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012076 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065889+0000", 2024-03-24T12:06:40.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0058947899999999996 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065905+0000", 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6121e-05 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065951+0000", 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6213e-05 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066051+0000", 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000100466 2024-03-24T12:06:40.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066096+0000", 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4186e-05 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221248 7.11 7:8cc49d56:::benchmark_data_smithi064_87013_object100291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.209849+0000", 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.52175865300001, 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85784487899999995, 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.945 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221248 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209849+0000", 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209849+0000", 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209851+0000", 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.671e-06 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209855+0000", 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4320000000000001e-06 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209866+0000", 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0913999999999999e-05 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209873+0000", 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1300000000000003e-06 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209919+0000", 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5596999999999997e-05 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.209927+0000", 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5789999999999998e-06 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.060800+0000", 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85087338700000004 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.060874+0000", 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4499000000000006e-05 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.061587+0000", 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00071300599999999997 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065910+0000", 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0043224789999999997 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066149+0000", 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000238605 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067582+0000", 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014336900000000001 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.950 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067644+0000", 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.143e-05 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067694+0000", 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0460000000000001e-05 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221301 7.11 7:8e59ab61:::benchmark_data_smithi064_87013_object100344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.951 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.269297+0000", 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.46231124600001, 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.79857248599999997, 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221301 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269297+0000", 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269297+0000", 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269298+0000", 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5600000000000001e-06 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269302+0000", 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0990000000000001e-06 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269311+0000", 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5210000000000002e-06 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269316+0000", 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1800000000000004e-06 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269366+0000", 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0099999999999998e-05 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.269376+0000", 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0369e-05 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.062706+0000", 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.79332951100000004 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.062817+0000", 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000110881 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.063002+0000", 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018522999999999999 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.065915+0000", 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0029131719999999999 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067525+0000", 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001609929 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067753+0000", 2024-03-24T12:06:40.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022786799999999999 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067825+0000", 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1600000000000006e-05 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067869+0000", 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4465999999999997e-05 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221321 7.11 7:8c46042a:::benchmark_data_smithi064_87013_object100364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.288567+0000", 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.44304129199998, 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.77949712999999998, 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221321 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288567+0000", 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288567+0000", 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288568+0000", 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9149999999999999e-06 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288576+0000", 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7409999999999992e-06 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288590+0000", 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3667e-05 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288598+0000", 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4530000000000002e-06 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288641+0000", 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2713000000000003e-05 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.288649+0000", 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5680000000000002e-06 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.064069+0000", 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.77542028200000002 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.064204+0000", 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013507999999999999 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.064429+0000", 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022490699999999999 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066106+0000", 2024-03-24T12:06:40.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001677109 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067920+0000", 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018139650000000001 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.067965+0000", 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4904000000000002e-05 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068019+0000", 2024-03-24T12:06:40.963 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3826999999999998e-05 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068064+0000", 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4999000000000001e-05 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221398 7.11 7:8907d6c7:::benchmark_data_smithi064_87013_object100441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.367034+0000", 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.36457391700003, 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.70618219199999999, 2024-03-24T12:06:40.964 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221398 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367034+0000", 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367034+0000", 2024-03-24T12:06:40.965 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367036+0000", 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9570000000000001e-06 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367040+0000", 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3839999999999999e-06 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367052+0000", 2024-03-24T12:06:40.966 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1479e-05 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367059+0000", 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5680000000000002e-06 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367122+0000", 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2909000000000004e-05 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.367133+0000", 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1003e-05 2024-03-24T12:06:40.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066169+0000", 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.69903535100000003 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066248+0000", 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9142999999999998e-05 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.066448+0000", 2024-03-24T12:06:40.968 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020039400000000001 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068083+0000", 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001635135 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068118+0000", 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4981999999999998e-05 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073116+0000", 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0049980170000000004 2024-03-24T12:06:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073170+0000", 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3362000000000002e-05 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073216+0000", 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6508000000000003e-05 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.970 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221445 7.11 7:8ba7bf8f:::benchmark_data_smithi064_87013_object100488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:51.438694+0000", 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 529.29291369800001, 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.63464338099999995, 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221445 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.971 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438694+0000", 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438694+0000", 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438695+0000", 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.269e-06 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.972 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438698+0000", 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0680000000000001e-06 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438706+0000", 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3390000000000004e-06 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438710+0000", 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.561e-06 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438751+0000", 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0349999999999998e-05 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:51.438757+0000", 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1630000000000001e-06 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068152+0000", 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.62939536500000004 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068258+0000", 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010558200000000001 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.068479+0000", 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022149299999999999 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073065+0000", 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045854399999999997 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073077+0000", 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2023e-05 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073257+0000", 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018018 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073302+0000", 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4758000000000001e-05 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:52.073338+0000", 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5790000000000001e-05 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221900 7.17 7:ed89da28:::benchmark_data_smithi064_87013_object100943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.089809+0000", 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.64179858199998, 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.97642383899999996, 2024-03-24T12:06:40.977 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221900 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089809+0000", 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089809+0000", 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089810+0000", 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.077e-06 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089815+0000", 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5290000000000002e-06 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089825+0000", 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0208e-05 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089833+0000", 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4909999999999999e-06 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089890+0000", 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7324000000000002e-05 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.089898+0000", 2024-03-24T12:06:40.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1049999999999995e-06 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.059780+0000", 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.969881993 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.059897+0000", 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000117451 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.060151+0000", 2024-03-24T12:06:40.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025358800000000001 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.064132+0000", 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0039813499999999998 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066047+0000", 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0019148139999999999 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.982 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066117+0000", 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9938999999999996e-05 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066189+0000", 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.148e-05 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066233+0000", 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4490000000000003e-05 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221927 7.17 7:ee472041:::benchmark_data_smithi064_87013_object100970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.113707+0000", 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.61790058999998, 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95626834000000005, 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221927 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.984 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113707+0000", 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113707+0000", 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113709+0000", 2024-03-24T12:06:40.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1909999999999999e-06 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113716+0000", 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.37e-06 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113728+0000", 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2583e-05 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113737+0000", 2024-03-24T12:06:40.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4630000000000001e-06 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113808+0000", 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0976000000000006e-05 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.113817+0000", 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2070000000000001e-06 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.062331+0000", 2024-03-24T12:06:40.987 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.94851381899999998 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.062532+0000", 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000201079 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.062753+0000", 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022100900000000001 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066273+0000", 2024-03-24T12:06:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0035203779999999998 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066360+0000", 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6365000000000004e-05 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.069880+0000", 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0035198579999999998 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.069929+0000", 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8952000000000002e-05 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.069976+0000", 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7089999999999998e-05 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221945 7.17 7:e85c526a:::benchmark_data_smithi064_87013_object100988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:40.990 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.127910+0000", 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.60369742199998, 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.94219929999999996, 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221945 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127910+0000", 2024-03-24T12:06:40.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127910+0000", 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127912+0000", 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.635e-06 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127919+0000", 2024-03-24T12:06:40.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3780000000000003e-06 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127930+0000", 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0708e-05 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127946+0000", 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5951e-05 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.127999+0000", 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3359000000000002e-05 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.128006+0000", 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4409999999999998e-06 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.064179+0000", 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.93617329199999999 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.064312+0000", 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013303800000000001 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.064556+0000", 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024340599999999999 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066308+0000", 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001752569 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.069989+0000", 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036805750000000002 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070033+0000", 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3878999999999998e-05 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070074+0000", 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1576999999999999e-05 2024-03-24T12:06:40.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070110+0000", 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5491999999999998e-05 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:221981 7.17 7:ea5818cd:::benchmark_data_smithi064_87013_object101024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.154381+0000", 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.57722705200001, 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92266110999999995, 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 221981 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154381+0000", 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154381+0000", 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154383+0000", 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.844e-06 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154388+0000", 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0930000000000002e-06 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:40.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154397+0000", 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2580000000000001e-06 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154403+0000", 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9689999999999999e-06 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154452+0000", 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9258999999999997e-05 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.154463+0000", 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1090999999999999e-05 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066406+0000", 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91194288999999995 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066526+0000", 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011948 2024-03-24T12:06:41.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.066731+0000", 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020494600000000001 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070123+0000", 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0033922480000000001 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073339+0000", 2024-03-24T12:06:41.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032162269999999999 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.076943+0000", 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036038250000000002 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.076996+0000", 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3486000000000003e-05 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.003 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077042+0000", 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5494000000000002e-05 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222006 7.17 7:e9e2a120:::benchmark_data_smithi064_87013_object101049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.173925+0000", 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.55768309899997, 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.90330706000000005, 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:41.004 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222006 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173925+0000", 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173925+0000", 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173926+0000", 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4169999999999999e-06 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173931+0000", 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.826e-06 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173942+0000", 2024-03-24T12:06:41.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1046999999999999e-05 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173950+0000", 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6469999999999998e-06 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.173997+0000", 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7599000000000003e-05 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:41.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.174003+0000", 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.412e-06 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.068132+0000", 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.89412962500000004 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.068239+0000", 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010655299999999999 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:41.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.068446+0000", 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020731699999999999 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073356+0000", 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0049102829999999997 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.076853+0000", 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034965450000000002 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.009 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077143+0000", 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028972399999999999 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077193+0000", 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0186999999999999e-05 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077232+0000", 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8878000000000003e-05 2024-03-24T12:06:41.010 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222025 7.17 7:edc5c377:::benchmark_data_smithi064_87013_object101068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.191039+0000", 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.54056879799998, 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88634509100000003, 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:41.011 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222025 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191039+0000", 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191039+0000", 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:41.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191040+0000", 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.37e-06 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191045+0000", 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3930000000000001e-06 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191054+0000", 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8089999999999999e-06 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191061+0000", 2024-03-24T12:06:41.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7659999999999998e-06 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191114+0000", 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2343999999999999e-05 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.191122+0000", 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6230000000000003e-06 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070149+0000", 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.879026851 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070253+0000", 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000103705 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.070456+0000", 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000203437 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073361+0000", 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0029049169999999999 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.076899+0000", 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0035376990000000001 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077293+0000", 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039371700000000003 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077343+0000", 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0634999999999999e-05 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077384+0000", 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0825000000000003e-05 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222028 7.17 7:ea98fedf:::benchmark_data_smithi064_87013_object101071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:41.017 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.193044+0000", 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.53856429999996, 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88705281400000002, 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222028 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.018 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193044+0000", 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193044+0000", 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193045+0000", 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8400000000000002e-07 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193048+0000", 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2899999999999998e-06 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193056+0000", 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3080000000000008e-06 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193061+0000", 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0629999999999996e-06 2024-03-24T12:06:41.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193107+0000", 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5776999999999999e-05 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.193117+0000", 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7820000000000005e-06 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.071768+0000", 2024-03-24T12:06:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.87865084000000004 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.071875+0000", 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107205 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.072057+0000", 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000181946 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.075219+0000", 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003162514 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077094+0000", 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001874887 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.079954+0000", 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028593780000000001 2024-03-24T12:06:41.023 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080043+0000", 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9956000000000004e-05 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080096+0000", 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2883999999999997e-05 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222050 7.17 7:eff9e39d:::benchmark_data_smithi064_87013_object101093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.214310+0000", 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.517297773, 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.86595284100000003, 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222050 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.025 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214310+0000", 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214310+0000", 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214312+0000", 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5e-06 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214316+0000", 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1139999999999999e-06 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214327+0000", 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1253999999999999e-05 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214335+0000", 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8520000000000004e-06 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214426+0000", 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1668000000000004e-05 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.214433+0000", 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9770000000000003e-06 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073387+0000", 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85895402700000001 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073495+0000", 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107889 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.073680+0000", 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000184728 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077109+0000", 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034289519999999999 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.029 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.078140+0000", 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0010306200000000001 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080165+0000", 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020256010000000001 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080222+0000", 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7197000000000001e-05 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:41.030 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080263+0000", 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0462e-05 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34742.0:222079 7.17 7:eeb9bced:::benchmark_data_smithi064_87013_object101122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:57:53.247113+0000", 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.48449450099997, 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.83330855000000004, 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:06:41.031 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34742", 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1376446502", 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222079 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247113+0000", 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247113+0000", 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247115+0000", 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.559e-06 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247119+0000", 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8260000000000003e-06 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247129+0000", 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8530000000000004e-06 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.033 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247134+0000", 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8019999999999997e-06 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247178+0000", 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3229000000000002e-05 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:53.247185+0000", 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9489999999999999e-06 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.034 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.075262+0000", 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.828077536 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.075380+0000", 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000118319 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.075570+0000", 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018975400000000001 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.077240+0000", 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016701400000000001 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.078175+0000", 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00093505400000000005 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080329+0000", 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021540510000000001 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080380+0000", 2024-03-24T12:06:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0824e-05 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:57:54.080422+0000", 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1653999999999998e-05 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:06:41.037 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:06:41.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:06:41.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:41.079 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:41.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:41.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:41.233 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:41.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:06:41.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:06:41.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:41.371 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:41.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:41.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:41.482 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:41.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:41 smithi193 ceph-mon[33593]: pgmap v657: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:41.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:41 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T12:06:41.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:41 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T12:06:41.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:41 smithi193 ceph-mon[33593]: 2.19 scrub starts 2024-03-24T12:06:41.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:41 smithi193 ceph-mon[33593]: 2.19 scrub ok 2024-03-24T12:06:41.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:41 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2303059106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:41.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:41.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[30223]: pgmap v657: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T12:06:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T12:06:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[30223]: 2.19 scrub starts 2024-03-24T12:06:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[30223]: 2.19 scrub ok 2024-03-24T12:06:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2303059106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[42243]: pgmap v657: 168 pgs: 18 active+undersized+degraded, 51 active+undersized, 99 active+clean; 483 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/594 objects degraded (11.279%) 2024-03-24T12:06:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T12:06:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T12:06:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[42243]: 2.19 scrub starts 2024-03-24T12:06:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[42243]: 2.19 scrub ok 2024-03-24T12:06:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:41 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2303059106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:41.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:41.658 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:41.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:41.734 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 2 16 7282 7266 227.044 221.562 0.00177645 0.00439563 2024-03-24T12:06:41.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:41.796 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:41.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:41.897 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:41.933 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:41.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:41.993 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 4.8 --file /var/log/ceph/exp.4.8.5 2024-03-24T12:06:42.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:42.073 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:42.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:42.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:42.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:42 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T12:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:42 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T12:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:42.199 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:42 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T12:06:42.199 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:42 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T12:06:42.199 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:42.199 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:42.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:42.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:42 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T12:06:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:42 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T12:06:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:42.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:42.355 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:42.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:42.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:42.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:42.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:42.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:42.634 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:42.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:42.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:06:42.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 3 16 10691 10675 222.364 213.062 0.00511264 0.00448986 2024-03-24T12:06:42.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:42.750 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:06:42.774 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:42.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:42.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:42.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:42.920 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:42.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:43.021 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:43.059 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:43.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:43.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:43.215 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:43.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:43.224 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:43 smithi193 ceph-mon[33593]: pgmap v658: 168 pgs: 21 active+undersized+degraded, 48 active+undersized, 99 active+clean; 179 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 286 op/s; 1110/9171 objects degraded (12.103%) 2024-03-24T12:06:43.224 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:43 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T12:06:43.224 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:43 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T12:06:43.224 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:43 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 1110/9171 objects degraded (12.103%), 21 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:43.224 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:43 smithi193 ceph-mon[33593]: 5.17 scrub starts 2024-03-24T12:06:43.224 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:43 smithi193 ceph-mon[33593]: 5.17 scrub ok 2024-03-24T12:06:43.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:06:43.248 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:43.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:43.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:43.389 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[30223]: pgmap v658: 168 pgs: 21 active+undersized+degraded, 48 active+undersized, 99 active+clean; 179 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 286 op/s; 1110/9171 objects degraded (12.103%) 2024-03-24T12:06:43.389 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 1110/9171 objects degraded (12.103%), 21 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[30223]: 5.17 scrub starts 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[30223]: 5.17 scrub ok 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[42243]: pgmap v658: 168 pgs: 21 active+undersized+degraded, 48 active+undersized, 99 active+clean; 179 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 286 op/s; 1110/9171 objects degraded (12.103%) 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 1110/9171 objects degraded (12.103%), 21 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[42243]: 5.17 scrub starts 2024-03-24T12:06:43.390 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:43 smithi064 ceph-mon[42243]: 5.17 scrub ok 2024-03-24T12:06:43.461 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:43.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:43.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:43.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:43.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:43.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 4 15 13709 13694 213.942 188.688 0.00770807 0.00466693 2024-03-24T12:06:43.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:43.763 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:43.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:43.864 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:43.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:43.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:44.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:44.050 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:44.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:44.150 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:44.185 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:44.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:44.287 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:44.323 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:44.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:44.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:44.448 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:44.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:44.550 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:44 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T12:06:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:44 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T12:06:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:44 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T12:06:44.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:44 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T12:06:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:44 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T12:06:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:44 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T12:06:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:44 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T12:06:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:44 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T12:06:44.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:44 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T12:06:44.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:44 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T12:06:44.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:44 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T12:06:44.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:44 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T12:06:44.588 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:44.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:44.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:44.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:44.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:44.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 5 16 16402 16386 204.801 168.25 0.00356943 0.00487623 2024-03-24T12:06:44.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:44.862 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:44.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:44.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:45.003 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:45.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:45.105 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:45.118 INFO:teuthology.orchestra.run.smithi193.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:06:45.144 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:45.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:45.146 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:06:45.161 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:45.246 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:45.283 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:45.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:45.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:45 smithi193 ceph-mon[33593]: pgmap v659: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 474 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 757 op/s; 2943/23307 objects degraded (12.627%) 2024-03-24T12:06:45.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:45 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:06:45.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:45 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:06:45.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:45 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3863897516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:45.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:45 smithi193 ceph-mon[33593]: from='client.35570 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:45.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:45.429 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:45.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:45.517 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[30223]: pgmap v659: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 474 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 757 op/s; 2943/23307 objects degraded (12.627%) 2024-03-24T12:06:45.517 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:06:45.517 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:06:45.518 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3863897516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:45.518 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[30223]: from='client.35570 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:45.518 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[42243]: pgmap v659: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 474 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 757 op/s; 2943/23307 objects degraded (12.627%) 2024-03-24T12:06:45.518 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:06:45.518 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:06:45.518 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3863897516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:45.518 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:45 smithi064 ceph-mon[42243]: from='client.35570 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:06:45.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:45.569 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:45.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:45.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:45.712 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:45.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:45.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 6 16 19341 19325 201.28 183.688 0.00427995 0.00496237 2024-03-24T12:06:45.814 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:45.852 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:45.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:45.921 INFO:teuthology.orchestra.run.smithi193.stderr:Exporting 4.8 info 4.8( v 241'27 (0'0,241'27] local-lis/les=68/69 n=7 ec=68/68 lis/c=68/68 les/c/f=69/69/0 sis=68) 2024-03-24T12:06:45.931 INFO:teuthology.orchestra.run.smithi193.stderr:Read #4:1111b601:::obj_delete_at_hint.0000000069:head# 2024-03-24T12:06:45.931 INFO:teuthology.orchestra.run.smithi193.stderr:Read #4:116b4168:::obj_delete_at_hint.0000000095:head# 2024-03-24T12:06:45.931 INFO:teuthology.orchestra.run.smithi193.stderr:Read #4:11d93ab7:::obj_delete_at_hint.0000000003:head# 2024-03-24T12:06:45.931 INFO:teuthology.orchestra.run.smithi193.stderr:Read #4:11fabdfc:::obj_delete_at_hint.0000000047:head# 2024-03-24T12:06:45.932 INFO:teuthology.orchestra.run.smithi193.stderr:Read #4:1467d28a:::obj_delete_at_hint.0000000035:head# 2024-03-24T12:06:45.932 INFO:teuthology.orchestra.run.smithi193.stderr:Read #4:15146f36:::obj_delete_at_hint.0000000037:head# 2024-03-24T12:06:45.932 INFO:teuthology.orchestra.run.smithi193.stderr:Read #4:17d7e8ed:gc::gc.23:head# 2024-03-24T12:06:45.932 INFO:teuthology.orchestra.run.smithi193.stderr:Export successful 2024-03-24T12:06:45.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:45.996 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:45.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:46.097 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:46.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:46.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:46.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:46.271 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:46.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:46 smithi193 ceph-mon[33593]: 5.1 scrub starts 2024-03-24T12:06:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:46 smithi193 ceph-mon[33593]: 5.1 scrub ok 2024-03-24T12:06:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:46 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:06:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:46 smithi193 ceph-mon[33593]: from='client.35570 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:46.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:46 smithi193 ceph-mon[33593]: osdmap e324: 8 total, 7 up, 8 in 2024-03-24T12:06:46.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:46.415 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:46.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:46.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[30223]: 5.1 scrub starts 2024-03-24T12:06:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[30223]: 5.1 scrub ok 2024-03-24T12:06:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:06:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[30223]: from='client.35570 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[30223]: osdmap e324: 8 total, 7 up, 8 in 2024-03-24T12:06:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[42243]: 5.1 scrub starts 2024-03-24T12:06:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[42243]: 5.1 scrub ok 2024-03-24T12:06:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:06:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[42243]: from='client.35570 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:06:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:46 smithi064 ceph-mon[42243]: osdmap e324: 8 total, 7 up, 8 in 2024-03-24T12:06:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:46.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:46.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:46.661 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:46.696 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:46.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:46.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 7 16 22221 22205 198.238 180 0.00590734 0.00503893 2024-03-24T12:06:46.799 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:46.836 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:46.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:46.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:46.975 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:46.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:47.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:47.115 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:47.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:47.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 4.8 2024-03-24T12:06:47.215 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:47.256 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:47.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:47.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:47 smithi193 ceph-mon[33593]: pgmap v661: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 474 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 614 B/s rd, 57 MiB/s wr, 908 op/s; 2943/23307 objects degraded (12.627%) 2024-03-24T12:06:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:47 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T12:06:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:47 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T12:06:47.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:47.388 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:47.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:47.411 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:06:47.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:47.533 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:47.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:47.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:06:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:06:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:47.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:47 smithi064 ceph-mon[42243]: pgmap v661: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 474 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 614 B/s rd, 57 MiB/s wr, 908 op/s; 2943/23307 objects degraded (12.627%) 2024-03-24T12:06:47.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:47 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T12:06:47.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:47 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T12:06:47.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:47 smithi064 ceph-mon[30223]: pgmap v661: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 474 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 614 B/s rd, 57 MiB/s wr, 908 op/s; 2943/23307 objects degraded (12.627%) 2024-03-24T12:06:47.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:47 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T12:06:47.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:47 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T12:06:47.634 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:47.665 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:47.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:47.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 8 16 24586 24570 191.933 147.812 0.005101 0.00520427 2024-03-24T12:06:47.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:47.798 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:47.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:47.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:06:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:06:47.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:47.938 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:47.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:48.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.080 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:48.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:48.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:48.219 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:48.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:48.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:48.357 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:48.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:48.492 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:48.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T12:06:48.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T12:06:48.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[42243]: 4.e scrub starts 2024-03-24T12:06:48.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[42243]: 4.e scrub ok 2024-03-24T12:06:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T12:06:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T12:06:48.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T12:06:48.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T12:06:48.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[30223]: 4.e scrub starts 2024-03-24T12:06:48.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[30223]: 4.e scrub ok 2024-03-24T12:06:48.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T12:06:48.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:48 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T12:06:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:48 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T12:06:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:48 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T12:06:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:48 smithi193 ceph-mon[33593]: 4.e scrub starts 2024-03-24T12:06:48.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:48 smithi193 ceph-mon[33593]: 4.e scrub ok 2024-03-24T12:06:48.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:48 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T12:06:48.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:48 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T12:06:48.594 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:48.626 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:48.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:48.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 9 16 27160 27144 188.481 160.875 0.00499352 0.00529991 2024-03-24T12:06:48.763 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:48.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:06:48.865 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:48.888 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:48.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:49.034 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:49.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:49.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:49.168 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:49.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:49.270 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:49.294 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:49.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:49.396 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:49.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:49 smithi064 ceph-mon[30223]: pgmap v662: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.90k op/s; 6794/57489 objects degraded (11.818%) 2024-03-24T12:06:49.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:49.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:49 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 6794/57489 objects degraded (11.818%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:49.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:49 smithi064 ceph-mon[42243]: pgmap v662: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.90k op/s; 6794/57489 objects degraded (11.818%) 2024-03-24T12:06:49.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:49.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:49 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 6794/57489 objects degraded (11.818%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:49.430 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:49.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:49.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:49.556 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:49.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:49 smithi193 ceph-mon[33593]: pgmap v662: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.90k op/s; 6794/57489 objects degraded (11.818%) 2024-03-24T12:06:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:49 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 6794/57489 objects degraded (11.818%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:49.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:49.693 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:49.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:49.736 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 10 16 28893 28877 180.463 108.312 0.00635509 0.00553542 2024-03-24T12:06:49.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:49.831 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:49.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:49.933 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:49.971 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:49.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:50.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:50.097 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:50.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:50.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:50.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:50.222 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:50.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:50.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:06:50.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T12:06:50.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:50 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T12:06:50.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:50.347 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:50.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:50.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:50.479 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:50.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:50.581 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:50 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:06:50.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:50 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:06:50.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:50 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T12:06:50.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:50 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T12:06:50.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:50 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T12:06:50.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:50 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T12:06:50.610 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:50.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:50.615 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:28523 10.6 10:67c849a3:::benchmark_data_smithi064_174908_object28522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:06:49.517662+0000", 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.052087865999999997, 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052121769999999998, 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:06:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 28523 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517662+0000", 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517662+0000", 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517663+0000", 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.429e-06 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517685+0000", 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1667000000000001e-05 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517696+0000", 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0791000000000001e-05 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517703+0000", 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7419999999999997e-06 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.519215+0000", 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015113819999999999 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.519351+0000", 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136253 2024-03-24T12:06:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.519503+0000", 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152254 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.520321+0000", 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081743899999999999 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:28501 10.4 10:2db2dba9:::benchmark_data_smithi064_174908_object28500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:06:49.509289+0000", 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.060460706000000003, 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060552768999999999, 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 28501 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.509289+0000", 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.509289+0000", 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.509291+0000", 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.584e-06 2024-03-24T12:06:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.509312+0000", 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1818999999999999e-05 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.509324+0000", 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1384e-05 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.509332+0000", 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7810000000000005e-06 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.512679+0000", 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033471799999999999 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.512829+0000", 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150338 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.513017+0000", 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018820299999999999 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514710+0000", 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001692289 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.518156+0000", 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003445947 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:28515 10.6 10:6eda6874:::benchmark_data_smithi064_174908_object28514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:06:49.514784+0000", 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.054965278999999999, 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.05511945, 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 28515 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514784+0000", 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514784+0000", 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514786+0000", 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0030000000000001e-06 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514809+0000", 2024-03-24T12:06:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2863000000000001e-05 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514822+0000", 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2286000000000001e-05 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514830+0000", 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7900000000000005e-06 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.515842+0000", 2024-03-24T12:06:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001012031 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.515983+0000", 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140452 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516113+0000", 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129763 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.519166+0000", 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030532200000000002 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:28516 10.4 10:29c1b873:::benchmark_data_smithi064_174908_object28515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:06:49.514896+0000", 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.054853906000000001, 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055056776000000002, 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:06:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 28516 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514896+0000", 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514896+0000", 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514897+0000", 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.102e-06 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514919+0000", 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1631999999999999e-05 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514928+0000", 2024-03-24T12:06:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4099999999999997e-06 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.514935+0000", 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.866e-06 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516152+0000", 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012167650000000001 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516286+0000", 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013456000000000001 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516856+0000", 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000569547 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.518540+0000", 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016844329999999999 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.518961+0000", 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042067999999999999 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:28520 10.6 10:674d5610:::benchmark_data_smithi064_174908_object28519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:06:49.516343+0000", 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.053406953, 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053676506999999998, 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:06:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 28520 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516343+0000", 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516343+0000", 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516345+0000", 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516369+0000", 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4011000000000001e-05 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516379+0000", 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0185e-05 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.516387+0000", 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6860000000000006e-06 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517656+0000", 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012693069999999999 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517777+0000", 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121396 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.517932+0000", 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015461500000000001 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:49.519173+0000", 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001241083 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 5 2024-03-24T12:06:50.639 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:06:50.711 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:50.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 11 16 30822 30806 175.017 120.562 0.00536909 0.00570814 2024-03-24T12:06:50.747 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:50.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:50.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:50.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:50.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:50.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:51.037 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:51.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:51.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:51.177 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:51.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:51.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[30223]: pgmap v663: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.4 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s; 8428/67764 objects degraded (12.437%) 2024-03-24T12:06:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[30223]: 1.6 scrub starts 2024-03-24T12:06:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[30223]: 1.6 scrub ok 2024-03-24T12:06:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[30223]: 5.9 deep-scrub starts 2024-03-24T12:06:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[30223]: 5.9 deep-scrub ok 2024-03-24T12:06:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[42243]: pgmap v663: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.4 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s; 8428/67764 objects degraded (12.437%) 2024-03-24T12:06:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[42243]: 1.6 scrub starts 2024-03-24T12:06:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[42243]: 1.6 scrub ok 2024-03-24T12:06:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[42243]: 5.9 deep-scrub starts 2024-03-24T12:06:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:51 smithi064 ceph-mon[42243]: 5.9 deep-scrub ok 2024-03-24T12:06:51.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:51.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:51.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:51.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:06:51.449 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:51.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:51.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:51.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:51 smithi193 ceph-mon[33593]: pgmap v663: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.4 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s; 8428/67764 objects degraded (12.437%) 2024-03-24T12:06:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:51 smithi193 ceph-mon[33593]: 1.6 scrub starts 2024-03-24T12:06:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:51 smithi193 ceph-mon[33593]: 1.6 scrub ok 2024-03-24T12:06:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:51 smithi193 ceph-mon[33593]: 5.9 deep-scrub starts 2024-03-24T12:06:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:51 smithi193 ceph-mon[33593]: 5.9 deep-scrub ok 2024-03-24T12:06:51.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:51.723 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:51.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:51.735 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 12 16 32953 32937 171.529 133.188 0.00394277 0.00582499 2024-03-24T12:06:51.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:51.826 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:51.857 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:51.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:51.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:51.989 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:51.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:52.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:52.128 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:52.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:52.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:52.253 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:52.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:52.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:52.398 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:52.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:52.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[30223]: pgmap v664: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 157 MiB/s wr, 2.51k op/s; 10625/84594 objects degraded (12.560%) 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[42243]: pgmap v664: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 157 MiB/s wr, 2.51k op/s; 10625/84594 objects degraded (12.560%) 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T12:06:52.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:52 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T12:06:52.499 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:52.534 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:52.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:52 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T12:06:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:52 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T12:06:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:52 smithi193 ceph-mon[33593]: pgmap v664: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 1.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 157 MiB/s wr, 2.51k op/s; 10625/84594 objects degraded (12.560%) 2024-03-24T12:06:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:52 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T12:06:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:52 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T12:06:52.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:52.676 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:52.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:52.736 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 13 15 34929 34914 167.838 123.562 0.0059849 0.00595193 2024-03-24T12:06:52.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:52.825 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:52.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:52.914 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:06:52.914 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:06:52.914 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:06:52.914 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:06:52.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:52.953 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:52.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:52.968 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:06:53.034 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:06:53.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:53.079 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:06:53.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:06:53.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:06:53.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:06:53.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:06:53.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:06:53.080 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:06:53.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:06:53.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:06:53.099 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:53.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:53.165 INFO:teuthology.orchestra.run.smithi193.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:06:53.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:53.234 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:53.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:53.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2278644075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:53.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:53 smithi064 ceph-mon[30223]: 1.13 scrub starts 2024-03-24T12:06:53.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:53 smithi064 ceph-mon[30223]: 1.13 scrub ok 2024-03-24T12:06:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2278644075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:53 smithi064 ceph-mon[42243]: 1.13 scrub starts 2024-03-24T12:06:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:53 smithi064 ceph-mon[42243]: 1.13 scrub ok 2024-03-24T12:06:53.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:53.374 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:53.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:53.476 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2278644075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:06:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:53 smithi193 ceph-mon[33593]: 1.13 scrub starts 2024-03-24T12:06:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:53 smithi193 ceph-mon[33593]: 1.13 scrub ok 2024-03-24T12:06:53.514 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:53.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:53.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:53.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:53.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:53.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:53.736 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 14 16 36724 36708 163.858 112.125 0.00443071 0.00609698 2024-03-24T12:06:53.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:53.760 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:06:53.809 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:53.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:53.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:53.957 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:53.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:54.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:54.100 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:54.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:54.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:54.233 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:54.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:54.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:54.357 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:54.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:54.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:54.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:54 smithi064 ceph-mon[30223]: pgmap v665: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.56k op/s; 12555/100161 objects degraded (12.535%) 2024-03-24T12:06:54.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:54 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:06:54.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:54 smithi064 ceph-mon[30223]: 4.14 scrub starts 2024-03-24T12:06:54.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:54 smithi064 ceph-mon[30223]: 4.14 scrub ok 2024-03-24T12:06:54.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:54 smithi064 ceph-mon[42243]: pgmap v665: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.56k op/s; 12555/100161 objects degraded (12.535%) 2024-03-24T12:06:54.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:54 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:06:54.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:54 smithi064 ceph-mon[42243]: 4.14 scrub starts 2024-03-24T12:06:54.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:54 smithi064 ceph-mon[42243]: 4.14 scrub ok 2024-03-24T12:06:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:54.497 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:54.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:54.567 INFO:teuthology.orchestra.run.smithi193.stdout: marking collection for removal 2024-03-24T12:06:54.567 INFO:teuthology.orchestra.run.smithi193.stdout:setting '_remove' omap key 2024-03-24T12:06:54.567 INFO:teuthology.orchestra.run.smithi193.stdout:finish_remove_pgs 4.8_head removing 4.8 2024-03-24T12:06:54.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:54 smithi193 ceph-mon[33593]: pgmap v665: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.56k op/s; 12555/100161 objects degraded (12.535%) 2024-03-24T12:06:54.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:54 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:06:54.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:54 smithi193 ceph-mon[33593]: 4.14 scrub starts 2024-03-24T12:06:54.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:54 smithi193 ceph-mon[33593]: 4.14 scrub ok 2024-03-24T12:06:54.576 INFO:teuthology.orchestra.run.smithi193.stdout:Remove successful 2024-03-24T12:06:54.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:54.635 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:54.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:54.736 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 15 16 38091 38075 158.63 85.4375 0.00434237 0.00628724 2024-03-24T12:06:54.737 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:54.779 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:54.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:54.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:54.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:54.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:55.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:55.051 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:55.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:55.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:55.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:55.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:55.313 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:55 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 12555/100161 objects degraded (12.535%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:55.313 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:55 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:06:55.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:55 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:06:55.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:55 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:06:55.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:55 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:06:55.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:55 smithi193 ceph-mon[33593]: 4.0 scrub starts 2024-03-24T12:06:55.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:55 smithi193 ceph-mon[33593]: 4.0 scrub ok 2024-03-24T12:06:55.333 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:55.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:55.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:55.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:55.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:55.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:55.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:55.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 12555/100161 objects degraded (12.535%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:55.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[30223]: 4.0 scrub starts 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[30223]: 4.0 scrub ok 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 12555/100161 objects degraded (12.535%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:06:55.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:06:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[42243]: 4.0 scrub starts 2024-03-24T12:06:55.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:55 smithi064 ceph-mon[42243]: 4.0 scrub ok 2024-03-24T12:06:55.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:55.619 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:55.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:55.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:55.736 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 16 15 40142 40127 156.725 128.25 0.00631738 0.006376 2024-03-24T12:06:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:55.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:55.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:55.888 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:55.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:55.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:55.970 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.8.5 2024-03-24T12:06:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:06:55.973 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:06:55.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:56.053 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:56.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:56.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:56.210 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:56.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:56.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:56.349 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:56.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:56.442 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:06:56.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:56.506 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:56.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:56.515 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[30223]: osdmap e325: 8 total, 7 up, 8 in 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[30223]: pgmap v667: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.56k op/s; 12555/100161 objects degraded (12.535%) 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[42243]: osdmap e325: 8 total, 7 up, 8 in 2024-03-24T12:06:56.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:56 smithi064 ceph-mon[42243]: pgmap v667: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.56k op/s; 12555/100161 objects degraded (12.535%) 2024-03-24T12:06:56.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:56 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:06:56.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:56 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:06:56.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:56 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T12:06:56.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:56 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T12:06:56.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:56 smithi193 ceph-mon[33593]: osdmap e325: 8 total, 7 up, 8 in 2024-03-24T12:06:56.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:56 smithi193 ceph-mon[33593]: pgmap v667: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.56k op/s; 12555/100161 objects degraded (12.535%) 2024-03-24T12:06:56.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:56.651 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:56.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:56.737 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 17 16 41941 41925 154.116 112.375 0.00962445 0.00648388 2024-03-24T12:06:56.752 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:56.790 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:56.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:56.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:56.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:57.036 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:57.072 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:57.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:57.174 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:57.211 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:57.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:57.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:57.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:57.450 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:06:57.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:57.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:06:57.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:06:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:57.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T12:06:57.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T12:06:57.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:06:57.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: osdmap e326: 8 total, 7 up, 8 in 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:06:57.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:06:57.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: osdmap e326: 8 total, 7 up, 8 in 2024-03-24T12:06:57.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:57.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:57.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T12:06:57.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T12:06:57.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:06:57.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:06:57.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:06:57.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:06:57.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: osdmap e326: 8 total, 7 up, 8 in 2024-03-24T12:06:57.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:06:57.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:57.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:57.628 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:57.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:57.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:57.737 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 18 16 43848 43832 152.174 119.188 0.00731379 0.0065658 2024-03-24T12:06:57.767 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:57.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:06:57.830 INFO:teuthology.orchestra.run.smithi193.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:06:57.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:06:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:06:57.869 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:57.921 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:57.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:57.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:06:58.022 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:58.058 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:58.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:58.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:58.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:58.211 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:58.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:58.282 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:06:58.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:58.356 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:58.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:58.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:58.503 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:58.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:58.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[30223]: 4.1b scrub starts 2024-03-24T12:06:58.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[30223]: 4.1b scrub ok 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/54545320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[30223]: from='client.35582 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[30223]: pgmap v669: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 14834/120303 objects degraded (12.331%) 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[42243]: 4.1b scrub starts 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[42243]: 4.1b scrub ok 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/54545320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:58.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[42243]: from='client.35582 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:58.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[42243]: pgmap v669: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 14834/120303 objects degraded (12.331%) 2024-03-24T12:06:58.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:58 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:58.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:58 smithi193 ceph-mon[33593]: 4.1b scrub starts 2024-03-24T12:06:58.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:58 smithi193 ceph-mon[33593]: 4.1b scrub ok 2024-03-24T12:06:58.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:58 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T12:06:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:58 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T12:06:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:58 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/54545320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:58 smithi193 ceph-mon[33593]: from='client.35582 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:06:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:58 smithi193 ceph-mon[33593]: pgmap v669: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 14834/120303 objects degraded (12.331%) 2024-03-24T12:06:58.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:58 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:58.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:58.645 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:58.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:58.666 INFO:teuthology.orchestra.run.smithi193.stdout:get_pg_num_history pg_num_history pg_num_history(e317 pg_nums {1={258=21},2={66=32},3={66=3},4={68=32},5={70=32},6={72=32},9={246=16}} deleted_pools 317,9) 2024-03-24T12:06:58.666 INFO:teuthology.orchestra.run.smithi193.stdout:Importing pgid 4.8 2024-03-24T12:06:58.666 INFO:teuthology.orchestra.run.smithi193.stdout:Write #4:1111b601:::obj_delete_at_hint.0000000069:head# 2024-03-24T12:06:58.666 INFO:teuthology.orchestra.run.smithi193.stdout:snapset 0=[]:{} 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:Write #4:116b4168:::obj_delete_at_hint.0000000095:head# 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:snapset 0=[]:{} 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:Write #4:11d93ab7:::obj_delete_at_hint.0000000003:head# 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:snapset 0=[]:{} 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:Write #4:11fabdfc:::obj_delete_at_hint.0000000047:head# 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:snapset 0=[]:{} 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:Write #4:1467d28a:::obj_delete_at_hint.0000000035:head# 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:snapset 0=[]:{} 2024-03-24T12:06:58.669 INFO:teuthology.orchestra.run.smithi193.stdout:Write #4:15146f36:::obj_delete_at_hint.0000000037:head# 2024-03-24T12:06:58.670 INFO:teuthology.orchestra.run.smithi193.stdout:snapset 0=[]:{} 2024-03-24T12:06:58.670 INFO:teuthology.orchestra.run.smithi193.stdout:Write #4:17d7e8ed:gc::gc.23:head# 2024-03-24T12:06:58.670 INFO:teuthology.orchestra.run.smithi193.stdout:snapset 0=[]:{} 2024-03-24T12:06:58.670 INFO:teuthology.orchestra.run.smithi193.stdout:write_pg epoch 296 info 4.8( v 241'27 (0'0,241'27] local-lis/les=68/69 n=7 ec=68/68 lis/c=68/68 les/c/f=69/69/0 sis=68) 2024-03-24T12:06:58.671 INFO:teuthology.orchestra.run.smithi193.stdout:Import successful 2024-03-24T12:06:58.712 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:06:58.712 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:06:58.737 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 19 15 45685 45670 150.211 114.875 0.00917653 0.00665283 2024-03-24T12:06:58.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:06:58.784 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:58.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:06:58.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:58.922 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:58.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:59.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:59.061 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:59.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:59.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:06:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:59.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:06:59.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:59.341 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:59.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:59.443 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:06:59.478 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:59.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:06:59.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:59 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T12:06:59.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:59 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T12:06:59.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:59 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:59.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:59 smithi193 ceph-mon[33593]: from='client.35582 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:06:59.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:06:59 smithi193 ceph-mon[33593]: osdmap e327: 8 total, 7 up, 8 in 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[30223]: from='client.35582 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[30223]: osdmap e327: 8 total, 7 up, 8 in 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:06:59.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[42243]: from='client.35582 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:06:59.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:06:59 smithi064 ceph-mon[42243]: osdmap e327: 8 total, 7 up, 8 in 2024-03-24T12:06:59.580 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:06:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:59.630 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:06:59.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:06:59.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:06:59.738 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:06:59.739849+0000 min lat: 0.00129496 max lat: 0.125534 avg lat: 0.0067306 2024-03-24T12:06:59.738 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:06:59.738 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 20 16 47535 47519 148.471 115.562 0.00790854 0.0067306 2024-03-24T12:06:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:59.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:59.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:06:59.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:06:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:59.908 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:06:59.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:00.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:00.048 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:00.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:00.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:07:00.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:00.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:00.298 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:00 smithi064 ceph-mon[30223]: pgmap v671: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.49k op/s; 15944/127065 objects degraded (12.548%) 2024-03-24T12:07:00.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:00 smithi064 ceph-mon[42243]: pgmap v671: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.49k op/s; 15944/127065 objects degraded (12.548%) 2024-03-24T12:07:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:00.331 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:00.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:00.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:00.490 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:00.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:00.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:00 smithi193 ceph-mon[33593]: pgmap v671: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.49k op/s; 15944/127065 objects degraded (12.548%) 2024-03-24T12:07:00.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:00.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:00.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:00.738 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 21 16 49160 49144 146.237 101.562 0.00266979 0.00682391 2024-03-24T12:07:00.743 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:00.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:00.845 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:00.882 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:00.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:00.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:01.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:01.127 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:01.161 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:01.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:01.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:01.304 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:01.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:01.337 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:01.406 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:01.440 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:01.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:01.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:01 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 15944/127065 objects degraded (12.548%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:01.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:01 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 15944/127065 objects degraded (12.548%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:01.582 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:01.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:01.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:07:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:01 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 15944/127065 objects degraded (12.548%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:01.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:01.731 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:01.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:01.738 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 22 16 50937 50921 144.637 111.062 0.00985127 0.00690881 2024-03-24T12:07:01.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:01.896 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:01.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:01.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:02.059 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:02.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:02.151 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/exp.4.8.5 2024-03-24T12:07:02.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:02.193 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:02.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:02.295 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:02.332 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:02.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:02.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:02.459 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:02 smithi193 ceph-mon[33593]: pgmap v672: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 136 MiB/s wr, 2.18k op/s; 17626/140721 objects degraded (12.525%) 2024-03-24T12:07:02.475 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:02.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:02.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:02 smithi064 ceph-mon[30223]: pgmap v672: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 136 MiB/s wr, 2.18k op/s; 17626/140721 objects degraded (12.525%) 2024-03-24T12:07:02.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:02 smithi064 ceph-mon[42243]: pgmap v672: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 136 MiB/s wr, 2.18k op/s; 17626/140721 objects degraded (12.525%) 2024-03-24T12:07:02.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:02.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:02.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:02.739 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 23 16 52645 52629 142.986 106.75 0.00684325 0.00698894 2024-03-24T12:07:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:02.768 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:02.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:02.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:02.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:03.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:03.053 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:03.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:03.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:03.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:03.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:03.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:03.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:03.518 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:03.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:03.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:03.657 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:03.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:03.739 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 24 16 54438 54422 141.697 112.062 0.00753419 0.00705211 2024-03-24T12:07:03.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:03.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:03.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:03.909 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:03.945 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:03.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:04.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:04.082 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:04.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:04.183 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:04.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:04.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:04.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:04.356 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:04.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:04.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:04.509 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:04.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:04 smithi064 ceph-mon[30223]: pgmap v673: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s; 19346/153942 objects degraded (12.567%) 2024-03-24T12:07:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:04 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:07:04.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:04 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:07:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:04 smithi064 ceph-mon[42243]: pgmap v673: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s; 19346/153942 objects degraded (12.567%) 2024-03-24T12:07:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:04 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:07:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:04 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:07:04.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:04.647 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:04.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:04.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:04 smithi193 ceph-mon[33593]: pgmap v673: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s; 19346/153942 objects degraded (12.567%) 2024-03-24T12:07:04.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:04 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:07:04.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:04 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:07:04.739 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 25 15 56155 56140 140.324 107.375 0.00468116 0.00712148 2024-03-24T12:07:04.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:04.785 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:04.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:04.888 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:04.917 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:04.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:05.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:05.033 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:07:05.033 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:07:05.059 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:05.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:05.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:05.161 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:05.203 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:05.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:05.255 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:05.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:05.354 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:05.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:05.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:05.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:05.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:05.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:05.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:05.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:05.741 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 26 16 57812 57796 138.902 103.5 0.00702351 0.00719447 2024-03-24T12:07:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:05.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:05.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:05 smithi064 ceph-mon[30223]: osdmap e328: 8 total, 7 up, 8 in 2024-03-24T12:07:05.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:05 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:05.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:05 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:05 smithi064 ceph-mon[42243]: osdmap e328: 8 total, 7 up, 8 in 2024-03-24T12:07:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:05 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:05.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:05 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:05.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:05 smithi193 ceph-mon[33593]: osdmap e328: 8 total, 7 up, 8 in 2024-03-24T12:07:05.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:05 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:05.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:05 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:05.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:07:05.870 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:05.912 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:05.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:06.013 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:06.045 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:06.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:06.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:06.189 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:06.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:06.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:06.329 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:06.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:06.431 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:06.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:06.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:06.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:06.603 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:06.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:06.705 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:06.739 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:06.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:06.740 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 27 16 59452 59436 137.554 102.5 0.00527789 0.00726542 2024-03-24T12:07:06.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:06.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:06.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:07.025 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:07.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:07.127 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:07.164 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:07.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:07.209 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:07:07.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:07:07.209 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-03-24T12:07:07.209 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-03-24T12:07:07.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:07:07.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:06 smithi064 ceph-mon[30223]: pgmap v675: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.40k op/s; 19346/153942 objects degraded (12.567%) 2024-03-24T12:07:07.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:06 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:07:07.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:06 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:07:07.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:06 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 19346/153942 objects degraded (12.567%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:07.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:06 smithi064 ceph-mon[42243]: pgmap v675: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.40k op/s; 19346/153942 objects degraded (12.567%) 2024-03-24T12:07:07.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:06 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:07:07.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:06 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:07:07.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:06 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 19346/153942 objects degraded (12.567%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:07.266 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:07:07.266 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:07.299 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:07.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:07.304 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:07:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:06 smithi193 ceph-mon[33593]: pgmap v675: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.40k op/s; 19346/153942 objects degraded (12.567%) 2024-03-24T12:07:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:06 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:07:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:06 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:07:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:06 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 19346/153942 objects degraded (12.567%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:07.343 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:07:07.388 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:07:07.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:07.427 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:07:07.436 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:07.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:07.473 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:07:07.518 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:07:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:07:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:07:07.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:07:07.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:07.565 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:07.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:07.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:07.705 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:07.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:07.740 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 28 16 60998 60982 136.092 96.625 0.00638795 0.00734292 2024-03-24T12:07:07.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:07.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:07:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:07:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:07.845 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:07.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:07.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:07.978 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:07.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:08.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:08.125 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:08.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:08.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:08.278 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:08.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:08 smithi064 ceph-mon[30223]: osdmap e329: 8 total, 7 up, 8 in 2024-03-24T12:07:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:08 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1100235444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:08 smithi064 ceph-mon[42243]: osdmap e329: 8 total, 7 up, 8 in 2024-03-24T12:07:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:08 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1100235444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:08 smithi193 ceph-mon[33593]: osdmap e329: 8 total, 7 up, 8 in 2024-03-24T12:07:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:08 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1100235444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:08.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:08.457 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:08.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:08.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:08.596 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:08.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:08.698 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:08.732 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:08.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:08.740 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 29 16 62613 62597 134.879 100.938 0.0123393 0.00740952 2024-03-24T12:07:08.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:08.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:08.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:09.014 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:09.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:09.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:09.173 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:09.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:09.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:09.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:09 smithi064 ceph-mon[30223]: pgmap v677: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 21498/173028 objects degraded (12.425%) 2024-03-24T12:07:09.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:09.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:09 smithi064 ceph-mon[42243]: pgmap v677: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 21498/173028 objects degraded (12.425%) 2024-03-24T12:07:09.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:09.316 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:09.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:09.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:09 smithi193 ceph-mon[33593]: pgmap v677: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 21498/173028 objects degraded (12.425%) 2024-03-24T12:07:09.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:09.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:09.459 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:09.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:09.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:09.592 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:09.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:09.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:09.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:09.740 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 30 16 64161 64145 133.608 96.75 0.0091341 0.00747943 2024-03-24T12:07:09.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:09.873 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:09.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:09.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:10.016 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:10.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:10.039 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:07:10.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:10.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:10.257 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:10.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3620606313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:10.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:10 smithi064 ceph-mon[30223]: from='client.25594 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:10.289 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3620606313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:10.290 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:10 smithi064 ceph-mon[42243]: from='client.25594 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:10.292 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:10.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3620606313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:10.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:10 smithi193 ceph-mon[33593]: from='client.25594 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:10.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:10.434 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:10.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:10.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:10.573 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:10.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:10.675 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:10.710 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:10.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:10.727 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:10.740 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 31 16 65799 65783 132.601 102.375 0.0134674 0.00753567 2024-03-24T12:07:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:07:10.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:07:10.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:10.872 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:10.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:10.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:11.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:11.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:11.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:11.164 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:11.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:11.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:11.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[30223]: pgmap v678: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s; 22564/179382 objects degraded (12.579%) 2024-03-24T12:07:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:07:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[30223]: from='client.25594 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:07:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[30223]: osdmap e330: 8 total, 7 up, 8 in 2024-03-24T12:07:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:07:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:07:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[42243]: pgmap v678: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s; 22564/179382 objects degraded (12.579%) 2024-03-24T12:07:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:07:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[42243]: from='client.25594 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:07:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[42243]: osdmap e330: 8 total, 7 up, 8 in 2024-03-24T12:07:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:07:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:11 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:07:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:11.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:11.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:11 smithi193 ceph-mon[33593]: pgmap v678: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s; 22564/179382 objects degraded (12.579%) 2024-03-24T12:07:11.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:11 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:07:11.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:11 smithi193 ceph-mon[33593]: from='client.25594 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:07:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:11 smithi193 ceph-mon[33593]: osdmap e330: 8 total, 7 up, 8 in 2024-03-24T12:07:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:11 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:07:11.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:11 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:07:11.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:11.440 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:11.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:11.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:11.584 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:11.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:11.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:11.732 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:11.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:11.741 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 32 16 67344 67328 131.471 96.5625 0.0130815 0.00760099 2024-03-24T12:07:11.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:11.871 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:11.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:11.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:12.006 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:12.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:12.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:12.155 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:12.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:12.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:12.291 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:12.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[30223]: osdmap e331: 8 total, 7 up, 8 in 2024-03-24T12:07:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 22564/179382 objects degraded (12.579%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[30223]: osdmap e332: 8 total, 7 up, 8 in 2024-03-24T12:07:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[42243]: osdmap e331: 8 total, 7 up, 8 in 2024-03-24T12:07:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 22564/179382 objects degraded (12.579%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:12 smithi064 ceph-mon[42243]: osdmap e332: 8 total, 7 up, 8 in 2024-03-24T12:07:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:12 smithi193 ceph-mon[33593]: osdmap e331: 8 total, 7 up, 8 in 2024-03-24T12:07:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:12 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 22564/179382 objects degraded (12.579%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:12 smithi193 ceph-mon[33593]: osdmap e332: 8 total, 7 up, 8 in 2024-03-24T12:07:12.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:12.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:12.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:12.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:12.572 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:12.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:12.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:12.709 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:12.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:12.741 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 33 16 68885 68869 130.406 96.3125 0.010108 0.00766364 2024-03-24T12:07:12.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:07:12.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:12.861 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:12.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:12.962 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:12.994 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:12.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:12.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:13.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:13.169 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:13.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:13.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:13 smithi064 ceph-mon[30223]: pgmap v681: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 130 MiB/s wr, 2.08k op/s; 24044/191430 objects degraded (12.560%) 2024-03-24T12:07:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:13 smithi064 ceph-mon[42243]: pgmap v681: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 130 MiB/s wr, 2.08k op/s; 24044/191430 objects degraded (12.560%) 2024-03-24T12:07:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:13.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:13 smithi193 ceph-mon[33593]: pgmap v681: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 130 MiB/s wr, 2.08k op/s; 24044/191430 objects degraded (12.560%) 2024-03-24T12:07:13.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:13.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:13.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:07:13.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:13.644 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:13.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:13.742 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 34 16 70356 70340 129.274 91.9375 0.0126504 0.00773105 2024-03-24T12:07:13.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:13.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:13.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:13.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:14.039 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:14.077 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:14.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:14.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:14.208 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:14.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:14.232 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:14.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:14 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:07:14.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:14 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:07:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:14 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:07:14.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:14 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:07:14.310 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:14 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:07:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:14 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:07:14.366 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:14.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:14.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:14.508 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:14.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:14.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:14.660 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:14.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:14.742 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 35 16 71732 71716 128.038 86 0.00449574 0.00780619 2024-03-24T12:07:14.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:14.822 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:14.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:14.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:14.955 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:14.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:14.991 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:72018 10.2 10:458ccba8:::benchmark_data_smithi064_174908_object72017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:14.983564+0000", 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.006768774, 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068014340000000003, 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:07:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 72018 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.983564+0000", 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.983564+0000", 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:07:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.983568+0000", 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5580000000000001e-06 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.983593+0000", 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5802000000000001e-05 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.983615+0000", 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1543999999999999e-05 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.983626+0000", 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0856000000000001e-05 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.983695+0000", 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9577000000000001e-05 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.983877+0000", 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018174599999999999 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.984024+0000", 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014725900000000001 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.989259+0000", 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052345619999999999 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.989542+0000", 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028292900000000001 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:72020 10.b e333/320 10:da96d1d5:::benchmark_data_smithi064_174908_object72019:head v 333'4346, mlcod=333'4344)", 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:14.986417+0000", 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0039156249999999998, 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040560689999999998, 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.986417+0000", 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.986417+0000", 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.986424+0000", 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7150000000000001e-06 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.986500+0000", 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5904999999999999e-05 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.986515+0000", 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5412000000000001e-05 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.986526+0000", 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0239e-05 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.986712+0000", 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018619400000000001 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:14.986772+0000", 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0439999999999997e-05 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 2 2024-03-24T12:07:15.000 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:07:15.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:15.086 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:15.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:15.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:15.226 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:15.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:15 smithi064 ceph-mon[30223]: pgmap v683: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s; 25534/203163 objects degraded (12.568%) 2024-03-24T12:07:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:15 smithi064 ceph-mon[30223]: osdmap e333: 8 total, 7 up, 8 in 2024-03-24T12:07:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:15 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:15 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:15 smithi064 ceph-mon[42243]: pgmap v683: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s; 25534/203163 objects degraded (12.568%) 2024-03-24T12:07:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:15 smithi064 ceph-mon[42243]: osdmap e333: 8 total, 7 up, 8 in 2024-03-24T12:07:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:15 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:15 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:15.328 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:15 smithi193 ceph-mon[33593]: pgmap v683: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s; 25534/203163 objects degraded (12.568%) 2024-03-24T12:07:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:15 smithi193 ceph-mon[33593]: osdmap e333: 8 total, 7 up, 8 in 2024-03-24T12:07:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:15 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:15.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:15 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:15.362 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:15.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:15.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:15.495 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:15.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:15.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:15.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:15.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:15.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:15.742 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 36 16 73124 73108 126.898 87 0.00293609 0.00787495 2024-03-24T12:07:15.759 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:15.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:15.861 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:15.895 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:15.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:15.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:16.035 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:16.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:16.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:16.173 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:16.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:16.274 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:16.309 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:16.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:16.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:16.446 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:16.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:16.548 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:16.582 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:16.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:16.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:16.715 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:16.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:16.742 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 37 16 74488 74472 125.772 85.25 0.00353057 0.00794501 2024-03-24T12:07:16.817 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:16.851 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:16.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:16.953 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:16.989 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:16.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:17.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:17.127 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:17.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:17.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:17.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: pgmap v685: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 948 B/s rd, 92 MiB/s wr, 1.47k op/s; 25534/203163 objects degraded (12.568%) 2024-03-24T12:07:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:07:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:07:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: osdmap e334: 8 total, 7 up, 8 in 2024-03-24T12:07:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:07:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 25534/203163 objects degraded (12.568%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: osdmap e335: 8 total, 7 up, 8 in 2024-03-24T12:07:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:07:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:17.373 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:17.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: pgmap v685: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 948 B/s rd, 92 MiB/s wr, 1.47k op/s; 25534/203163 objects degraded (12.568%) 2024-03-24T12:07:17.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:07:17.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: osdmap e334: 8 total, 7 up, 8 in 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 25534/203163 objects degraded (12.568%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: osdmap e335: 8 total, 7 up, 8 in 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: pgmap v685: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 948 B/s rd, 92 MiB/s wr, 1.47k op/s; 25534/203163 objects degraded (12.568%) 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: osdmap e334: 8 total, 7 up, 8 in 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:07:17.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 25534/203163 objects degraded (12.568%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:17.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: osdmap e335: 8 total, 7 up, 8 in 2024-03-24T12:07:17.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:07:17.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:17.397 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:17.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:17.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:17.535 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:17.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:17.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:17.675 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:17.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:17.742 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 38 16 75754 75738 124.545 79.125 0.00608989 0.00802442 2024-03-24T12:07:17.776 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:17.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:07:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:07:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:17.812 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:17.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:17.914 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:17.943 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:17.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:18.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:18.083 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:18.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:18.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:07:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:07:18.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:18.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:18.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:18.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:18.465 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:18.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:18.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:18.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:18.742 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 39 16 77196 77180 123.662 90.125 0.0128161 0.00808198 2024-03-24T12:07:18.744 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:18.781 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:18.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:18.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:18.908 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:18.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:19.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:19.046 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:19.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:19.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:19.187 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:19.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:19.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:19 smithi193 ceph-mon[33593]: pgmap v688: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 922 B/s rd, 88 MiB/s wr, 1.40k op/s; 27310/219213 objects degraded (12.458%) 2024-03-24T12:07:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:19.337 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:19.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:19.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:19.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:19 smithi064 ceph-mon[30223]: pgmap v688: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 922 B/s rd, 88 MiB/s wr, 1.40k op/s; 27310/219213 objects degraded (12.458%) 2024-03-24T12:07:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:19.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:19 smithi064 ceph-mon[42243]: pgmap v688: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 922 B/s rd, 88 MiB/s wr, 1.40k op/s; 27310/219213 objects degraded (12.458%) 2024-03-24T12:07:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:19.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:19.620 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:19.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:19.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:19.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:07:19.744784+0000 min lat: 0.00129496 max lat: 0.125534 avg lat: 0.00814657 2024-03-24T12:07:19.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:07:19.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 40 16 78545 78529 122.676 84.3125 0.0115511 0.00814657 2024-03-24T12:07:19.767 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:19.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:19.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:19.909 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:19.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:19.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:20.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:20.056 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:20.079 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:20.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:20.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:20.222 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:20.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:20.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:20.363 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:20.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:20.465 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:20.502 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:20.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:20.520 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:20.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:07:20.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:20.666 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:20.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:20.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 41 16 80019 80003 121.93 92.125 0.0124462 0.00819702 2024-03-24T12:07:20.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:20.817 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:20.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:20.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:20.973 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:20.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:21.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:21.106 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:21.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:21.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:21.207 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:21.240 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:21.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:21.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:21 smithi064 ceph-mon[30223]: pgmap v689: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 28180/224385 objects degraded (12.559%) 2024-03-24T12:07:21.339 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:21 smithi064 ceph-mon[42243]: pgmap v689: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 28180/224385 objects degraded (12.559%) 2024-03-24T12:07:21.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:21.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:21.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:21.517 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:21.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:21 smithi193 ceph-mon[33593]: pgmap v689: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 28180/224385 objects degraded (12.559%) 2024-03-24T12:07:21.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:21.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:21.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:21.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 42 16 81472 81456 121.189 90.8125 0.00388042 0.00824638 2024-03-24T12:07:21.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:21.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:21.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:21.942 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:21.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:22.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:22.081 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:22.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:22.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:22.217 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:22.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:22.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:22 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 28180/224385 objects degraded (12.559%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:22.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:22 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 28180/224385 objects degraded (12.559%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:22.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:22.348 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:22.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:22.445 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:07:22.448 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:07:22.450 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:22.480 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:22.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:22.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:22 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 28180/224385 objects degraded (12.559%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:22.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:22.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:22.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 43 16 82748 82732 120.226 79.75 0.012749 0.00831291 2024-03-24T12:07:22.755 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:22.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:22.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:22.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:22.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:22.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:23.035 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:23.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:23.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:23.174 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:23.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:23.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:23 smithi064 ceph-mon[30223]: pgmap v690: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.44k op/s; 29487/234918 objects degraded (12.552%) 2024-03-24T12:07:23.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:23 smithi064 ceph-mon[42243]: pgmap v690: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.44k op/s; 29487/234918 objects degraded (12.552%) 2024-03-24T12:07:23.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:23.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:23.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:23.443 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:23.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:23.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:23 smithi193 ceph-mon[33593]: pgmap v690: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.44k op/s; 29487/234918 objects degraded (12.552%) 2024-03-24T12:07:23.545 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:23.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:23.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:23.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:23.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:23.716 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:23.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:23.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 44 16 84209 84193 119.568 91.3125 0.00474405 0.00835855 2024-03-24T12:07:23.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:23.853 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:23.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:23.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:23.997 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:23.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:24.098 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:24.134 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:24.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:24.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:24.271 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:24.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:24.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:24 smithi064 ceph-mon[30223]: from='client.15756 v1:172.21.15.64:0/1219444655' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:07:24.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:24 smithi064 ceph-mon[42243]: from='client.15756 v1:172.21.15.64:0/1219444655' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:07:24.328 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:24 smithi193 ceph-mon[33593]: from='client.15756 v1:172.21.15.64:0/1219444655' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:07:24.352 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:24.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2024-03-24T12:07:21.430147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5119380527,"num_objects":78306,"num_object_clones":0,"num_object_copies":234918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78306,"num_whiteouts":0,"num_read":1296,"num_read_kb":1074,"num_write":78879,"num_write_kb":4999615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":484956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74909,"ondisk_log_size":74909,"up":435,"acting":435,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":507,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17444812,"kb_used_data":12924976,"kb_used_omap":12,"kb_used_meta":4519731,"kb_avail":732516404,"statfs":{"total":767960285184,"available":750096797696,"internally_reserved":0,"allocated":13235175424,"data_stored":13231582821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":4628205104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,6,4,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":693698560,"num_objects":10585,"num_object_clones":0,"num_object_copies":31755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10585,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10585,"num_write_kb":677440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.363325"},"pg_stats":[{"pgid":"10.6","version":"335'4670","reported_seq":4707,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:18.068662+0000","last_change":"2024-03-24T12:06:39.744919+0000","last_active":"2024-03-24T12:07:18.068662+0000","last_peered":"2024-03-24T12:07:18.068662+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.019918+0000","last_became_peered":"2024-03-24T12:06:30.019918+0000","last_unstale":"2024-03-24T12:07:18.068662+0000","last_undegraded":"2024-03-24T12:06:39.740916+0000","last_fullsized":"2024-03-24T12:06:30.013919+0000","mapping_epoch":320,"log_start":"323'100","ondisk_log_start":"323'100","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:32.923512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306053120,"num_objects":4670,"num_object_clones":0,"num_object_copies":14010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4670,"num_write_kb":298880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4670}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"83'130","reported_seq":469,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.088241+0000","last_change":"2024-03-24T12:05:08.153739+0000","last_active":"2024-03-24T12:07:16.088241+0000","last_peered":"2024-03-24T12:07:16.088241+0000","last_clean":"2024-03-24T12:07:16.088241+0000","last_became_active":"2024-03-24T12:03:19.822426+0000","last_became_peered":"2024-03-24T12:03:19.822426+0000","last_unstale":"2024-03-24T12:07:16.088241+0000","last_undegraded":"2024-03-24T12:07:16.088241+0000","last_fullsized":"2024-03-24T12:07:16.088241+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:08.153554+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:08.153554+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":796,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.087805+0000","last_change":"2024-03-24T12:06:50.162950+0000","last_active":"2024-03-24T12:07:16.087805+0000","last_peered":"2024-03-24T12:07:16.087805+0000","last_clean":"2024-03-24T12:07:16.087805+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:07:16.087805+0000","last_undegraded":"2024-03-24T12:07:16.087805+0000","last_fullsized":"2024-03-24T12:07:16.087805+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:06:50.162823+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:06:50.162823+0000","last_clean_scrub_stamp":"2024-03-24T12:06:50.162823+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:55.282075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":469,"reported_epoch":334,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.089142+0000","last_change":"2024-03-24T12:06:27.010354+0000","last_active":"2024-03-24T12:07:16.089142+0000","last_peered":"2024-03-24T12:07:16.089142+0000","last_clean":"2024-03-24T12:02:22.958798+0000","last_became_active":"2024-03-24T12:06:27.010354+0000","last_became_peered":"2024-03-24T12:06:27.010354+0000","last_unstale":"2024-03-24T12:07:16.089142+0000","last_undegraded":"2024-03-24T12:06:27.003330+0000","last_fullsized":"2024-03-24T12:06:27.003218+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:05:17.253630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.253630+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":81,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886179+0000","last_change":"2024-03-24T12:05:44.732301+0000","last_active":"2024-03-24T12:07:16.886179+0000","last_peered":"2024-03-24T12:07:16.886179+0000","last_clean":"2024-03-24T12:07:16.886179+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:07:16.886179+0000","last_undegraded":"2024-03-24T12:07:16.886179+0000","last_fullsized":"2024-03-24T12:07:16.886179+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:11.505956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.505956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":41,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.886277+0000","last_change":"2024-03-24T12:06:27.009659+0000","last_active":"2024-03-24T12:07:16.886277+0000","last_peered":"2024-03-24T12:07:16.886277+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.009659+0000","last_became_peered":"2024-03-24T12:06:27.009659+0000","last_unstale":"2024-03-24T12:07:16.886277+0000","last_undegraded":"2024-03-24T12:07:16.886277+0000","last_fullsized":"2024-03-24T12:06:26.998639+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.095464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.095464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"335'4747","reported_seq":4787,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:17.017572+0000","last_change":"2024-03-24T12:06:30.016914+0000","last_active":"2024-03-24T12:07:17.017572+0000","last_peered":"2024-03-24T12:07:17.017572+0000","last_clean":"2024-03-24T12:07:17.017572+0000","last_became_active":"2024-03-24T12:06:30.016602+0000","last_became_peered":"2024-03-24T12:06:30.016602+0000","last_unstale":"2024-03-24T12:07:17.017572+0000","last_undegraded":"2024-03-24T12:07:17.017572+0000","last_fullsized":"2024-03-24T12:07:17.017572+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:30.194689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311099392,"num_objects":4747,"num_object_clones":0,"num_object_copies":14241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4747,"num_write_kb":303808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'130","reported_seq":470,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.896832+0000","last_change":"2024-03-24T12:06:12.900023+0000","last_active":"2024-03-24T12:07:16.896832+0000","last_peered":"2024-03-24T12:07:16.896832+0000","last_clean":"2024-03-24T12:07:16.896832+0000","last_became_active":"2024-03-24T12:03:19.822938+0000","last_became_peered":"2024-03-24T12:03:19.822938+0000","last_unstale":"2024-03-24T12:07:16.896832+0000","last_undegraded":"2024-03-24T12:07:16.896832+0000","last_fullsized":"2024-03-24T12:07:16.896832+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:12.899943+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:12.899943+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:22.788474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":621,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891269+0000","last_change":"2024-03-24T12:06:07.971250+0000","last_active":"2024-03-24T12:07:16.891269+0000","last_peered":"2024-03-24T12:07:16.891269+0000","last_clean":"2024-03-24T12:07:16.891269+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:07:16.891269+0000","last_undegraded":"2024-03-24T12:07:16.891269+0000","last_fullsized":"2024-03-24T12:07:16.891269+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:06:07.971105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:07.971105+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":622,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.142651+0000","last_change":"2024-03-24T12:06:54.372697+0000","last_active":"2024-03-24T12:07:16.142651+0000","last_peered":"2024-03-24T12:07:16.142651+0000","last_clean":"2024-03-24T12:07:16.142651+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:07:16.142651+0000","last_undegraded":"2024-03-24T12:07:16.142651+0000","last_fullsized":"2024-03-24T12:07:16.142651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:54.372598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.372598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:21.803855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":80,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.894841+0000","last_change":"2024-03-24T12:05:44.728767+0000","last_active":"2024-03-24T12:07:16.894841+0000","last_peered":"2024-03-24T12:07:16.894841+0000","last_clean":"2024-03-24T12:07:16.894841+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:07:16.894841+0000","last_undegraded":"2024-03-24T12:07:16.894841+0000","last_fullsized":"2024-03-24T12:07:16.894841+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:52.456151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:52.456151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":586,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890428+0000","last_change":"2024-03-24T12:05:44.737121+0000","last_active":"2024-03-24T12:07:16.890428+0000","last_peered":"2024-03-24T12:07:16.890428+0000","last_clean":"2024-03-24T12:07:16.890428+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:07:16.890428+0000","last_undegraded":"2024-03-24T12:07:16.890428+0000","last_fullsized":"2024-03-24T12:07:16.890428+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:14.719129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.719129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"335'4793","reported_seq":4833,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:18.057836+0000","last_change":"2024-03-24T12:06:30.020617+0000","last_active":"2024-03-24T12:07:18.057836+0000","last_peered":"2024-03-24T12:07:18.057836+0000","last_clean":"2024-03-24T12:07:18.057836+0000","last_became_active":"2024-03-24T12:06:30.020223+0000","last_became_peered":"2024-03-24T12:06:30.020223+0000","last_unstale":"2024-03-24T12:07:18.057836+0000","last_undegraded":"2024-03-24T12:07:18.057836+0000","last_fullsized":"2024-03-24T12:07:18.057836+0000","mapping_epoch":320,"log_start":"323'300","ondisk_log_start":"323'300","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:55.951188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314114048,"num_objects":4793,"num_object_clones":0,"num_object_copies":14379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4793,"num_write_kb":306752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'130","reported_seq":659,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.889750+0000","last_change":"2024-03-24T12:06:27.005543+0000","last_active":"2024-03-24T12:07:16.889750+0000","last_peered":"2024-03-24T12:07:16.889750+0000","last_clean":"2024-03-24T12:06:22.267535+0000","last_became_active":"2024-03-24T12:06:27.005543+0000","last_became_peered":"2024-03-24T12:06:27.005543+0000","last_unstale":"2024-03-24T12:07:16.889750+0000","last_undegraded":"2024-03-24T12:07:16.889750+0000","last_fullsized":"2024-03-24T12:06:27.001250+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:02.370579+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.370579+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"296'44","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886969+0000","last_change":"2024-03-24T12:05:14.441279+0000","last_active":"2024-03-24T12:07:16.886969+0000","last_peered":"2024-03-24T12:07:16.886969+0000","last_clean":"2024-03-24T12:07:16.886969+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:07:16.886969+0000","last_undegraded":"2024-03-24T12:07:16.886969+0000","last_fullsized":"2024-03-24T12:07:16.886969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.441098+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":44,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.892788+0000","last_change":"2024-03-24T12:06:27.009675+0000","last_active":"2024-03-24T12:07:16.892788+0000","last_peered":"2024-03-24T12:07:16.892788+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.009675+0000","last_became_peered":"2024-03-24T12:06:27.009675+0000","last_unstale":"2024-03-24T12:07:16.892788+0000","last_undegraded":"2024-03-24T12:07:16.892788+0000","last_fullsized":"2024-03-24T12:06:26.998070+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:00.489543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:05:00.489543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":83,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890713+0000","last_change":"2024-03-24T12:06:27.009828+0000","last_active":"2024-03-24T12:07:16.890713+0000","last_peered":"2024-03-24T12:07:16.890713+0000","last_clean":"2024-03-24T12:06:22.261311+0000","last_became_active":"2024-03-24T12:06:27.009828+0000","last_became_peered":"2024-03-24T12:06:27.009828+0000","last_unstale":"2024-03-24T12:07:16.890713+0000","last_undegraded":"2024-03-24T12:07:16.890713+0000","last_fullsized":"2024-03-24T12:06:26.998323+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:48.399346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:48.399346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":600,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.087555+0000","last_change":"2024-03-24T12:05:44.737781+0000","last_active":"2024-03-24T12:07:16.087555+0000","last_peered":"2024-03-24T12:07:16.087555+0000","last_clean":"2024-03-24T12:07:16.087555+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:07:16.087555+0000","last_undegraded":"2024-03-24T12:07:16.087555+0000","last_fullsized":"2024-03-24T12:07:16.087555+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:14.128698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.128698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"335'4833","reported_seq":4870,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:17.622120+0000","last_change":"2024-03-24T12:06:39.741733+0000","last_active":"2024-03-24T12:07:17.622120+0000","last_peered":"2024-03-24T12:07:17.622120+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.019248+0000","last_became_peered":"2024-03-24T12:06:30.019248+0000","last_unstale":"2024-03-24T12:07:17.622120+0000","last_undegraded":"2024-03-24T12:06:39.741580+0000","last_fullsized":"2024-03-24T12:06:30.008982+0000","mapping_epoch":320,"log_start":"323'600","ondisk_log_start":"323'600","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:38.628409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316735488,"num_objects":4833,"num_object_clones":0,"num_object_copies":14499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4833,"num_write_kb":309312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":4833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":676,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890959+0000","last_change":"2024-03-24T12:05:44.750408+0000","last_active":"2024-03-24T12:07:16.890959+0000","last_peered":"2024-03-24T12:07:16.890959+0000","last_clean":"2024-03-24T12:07:16.890959+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:07:16.890959+0000","last_undegraded":"2024-03-24T12:07:16.890959+0000","last_fullsized":"2024-03-24T12:07:16.890959+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:03:52.852641+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.852641+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":603,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.894867+0000","last_change":"2024-03-24T12:06:27.004815+0000","last_active":"2024-03-24T12:07:16.894867+0000","last_peered":"2024-03-24T12:07:16.894867+0000","last_clean":"2024-03-24T12:06:21.980599+0000","last_became_active":"2024-03-24T12:06:27.004815+0000","last_became_peered":"2024-03-24T12:06:27.004815+0000","last_unstale":"2024-03-24T12:07:16.894867+0000","last_undegraded":"2024-03-24T12:06:27.001638+0000","last_fullsized":"2024-03-24T12:06:27.001523+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:05:05.492318+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.492318+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":344,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891144+0000","last_change":"2024-03-24T12:03:56.825579+0000","last_active":"2024-03-24T12:07:16.891144+0000","last_peered":"2024-03-24T12:07:16.891144+0000","last_clean":"2024-03-24T12:07:16.891144+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:07:16.891144+0000","last_undegraded":"2024-03-24T12:07:16.891144+0000","last_fullsized":"2024-03-24T12:07:16.891144+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.825474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.825474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481147+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":80,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891711+0000","last_change":"2024-03-24T12:05:44.734066+0000","last_active":"2024-03-24T12:07:16.891711+0000","last_peered":"2024-03-24T12:07:16.891711+0000","last_clean":"2024-03-24T12:07:16.891711+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:07:16.891711+0000","last_undegraded":"2024-03-24T12:07:16.891711+0000","last_fullsized":"2024-03-24T12:07:16.891711+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:12.726872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:12.726872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":338,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.894797+0000","last_change":"2024-03-24T12:06:27.007623+0000","last_active":"2024-03-24T12:07:16.894797+0000","last_peered":"2024-03-24T12:07:16.894797+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T12:06:27.007623+0000","last_became_peered":"2024-03-24T12:06:27.007623+0000","last_unstale":"2024-03-24T12:07:16.894797+0000","last_undegraded":"2024-03-24T12:07:16.894797+0000","last_fullsized":"2024-03-24T12:06:27.000534+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:19.243242+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.243242+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:20.743979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"334'4650","reported_seq":4688,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.702712+0000","last_change":"2024-03-24T12:06:30.019085+0000","last_active":"2024-03-24T12:07:16.702712+0000","last_peered":"2024-03-24T12:07:16.702712+0000","last_clean":"2024-03-24T12:07:16.702712+0000","last_became_active":"2024-03-24T12:06:30.018755+0000","last_became_peered":"2024-03-24T12:06:30.018755+0000","last_unstale":"2024-03-24T12:07:16.702712+0000","last_undegraded":"2024-03-24T12:07:16.702712+0000","last_fullsized":"2024-03-24T12:07:16.702712+0000","mapping_epoch":320,"log_start":"323'100","ondisk_log_start":"323'100","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:49.023062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304742400,"num_objects":4650,"num_object_clones":0,"num_object_copies":13950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":297600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":630,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890777+0000","last_change":"2024-03-24T12:05:01.942196+0000","last_active":"2024-03-24T12:07:16.890777+0000","last_peered":"2024-03-24T12:07:16.890777+0000","last_clean":"2024-03-24T12:07:16.890777+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:07:16.890777+0000","last_undegraded":"2024-03-24T12:07:16.890777+0000","last_fullsized":"2024-03-24T12:07:16.890777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:01.942039+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.942039+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":343,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.884550+0000","last_change":"2024-03-24T12:04:00.667345+0000","last_active":"2024-03-24T12:07:16.884550+0000","last_peered":"2024-03-24T12:07:16.884550+0000","last_clean":"2024-03-24T12:07:16.884550+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:07:16.884550+0000","last_undegraded":"2024-03-24T12:07:16.884550+0000","last_fullsized":"2024-03-24T12:07:16.884550+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:04:00.667221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"319'9","reported_seq":794,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.896047+0000","last_change":"2024-03-24T12:06:27.007547+0000","last_active":"2024-03-24T12:07:16.896047+0000","last_peered":"2024-03-24T12:07:16.896047+0000","last_clean":"2024-03-24T12:06:25.366499+0000","last_became_active":"2024-03-24T12:06:27.007547+0000","last_became_peered":"2024-03-24T12:06:27.007547+0000","last_unstale":"2024-03-24T12:07:16.896047+0000","last_undegraded":"2024-03-24T12:06:27.001189+0000","last_fullsized":"2024-03-24T12:06:27.001079+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:06.478884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.478884+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":601,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.900253+0000","last_change":"2024-03-24T12:06:27.012210+0000","last_active":"2024-03-24T12:07:16.900253+0000","last_peered":"2024-03-24T12:07:16.900253+0000","last_clean":"2024-03-24T12:06:22.267578+0000","last_became_active":"2024-03-24T12:06:27.012210+0000","last_became_peered":"2024-03-24T12:06:27.012210+0000","last_unstale":"2024-03-24T12:07:16.900253+0000","last_undegraded":"2024-03-24T12:06:27.002891+0000","last_fullsized":"2024-03-24T12:06:27.002772+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:03:47.909970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:47.909970+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481122+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":893,"reported_epoch":334,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.087420+0000","last_change":"2024-03-24T12:06:27.013805+0000","last_active":"2024-03-24T12:07:16.087420+0000","last_peered":"2024-03-24T12:07:16.087420+0000","last_clean":"2024-03-24T12:06:22.145375+0000","last_became_active":"2024-03-24T12:06:27.013805+0000","last_became_peered":"2024-03-24T12:06:27.013805+0000","last_unstale":"2024-03-24T12:07:16.087420+0000","last_undegraded":"2024-03-24T12:07:16.087420+0000","last_fullsized":"2024-03-24T12:06:27.003965+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.142552+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"335'5167","reported_seq":5207,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:21.364356+0000","last_change":"2024-03-24T12:06:30.019783+0000","last_active":"2024-03-24T12:07:21.364356+0000","last_peered":"2024-03-24T12:07:21.364356+0000","last_clean":"2024-03-24T12:07:21.364356+0000","last_became_active":"2024-03-24T12:06:30.019501+0000","last_became_peered":"2024-03-24T12:06:30.019501+0000","last_unstale":"2024-03-24T12:07:21.364356+0000","last_undegraded":"2024-03-24T12:07:21.364356+0000","last_fullsized":"2024-03-24T12:07:21.364356+0000","mapping_epoch":320,"log_start":"323'900","ondisk_log_start":"323'900","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:55.623811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338624512,"num_objects":5167,"num_object_clones":0,"num_object_copies":15501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":330688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":660,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.085506+0000","last_change":"2024-03-24T12:06:48.430894+0000","last_active":"2024-03-24T12:07:16.085506+0000","last_peered":"2024-03-24T12:07:16.085506+0000","last_clean":"2024-03-24T12:07:16.085506+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:07:16.085506+0000","last_undegraded":"2024-03-24T12:07:16.085506+0000","last_fullsized":"2024-03-24T12:07:16.085506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:06:48.430734+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:06:48.430734+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.923096+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":614,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.084074+0000","last_change":"2024-03-24T12:06:38.413964+0000","last_active":"2024-03-24T12:07:16.084074+0000","last_peered":"2024-03-24T12:07:16.084074+0000","last_clean":"2024-03-24T12:07:16.084074+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:07:16.084074+0000","last_undegraded":"2024-03-24T12:07:16.084074+0000","last_fullsized":"2024-03-24T12:07:16.084074+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:38.413878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:06:38.413878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.670119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"319'9","reported_seq":788,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.900160+0000","last_change":"2024-03-24T12:06:27.011265+0000","last_active":"2024-03-24T12:07:16.900160+0000","last_peered":"2024-03-24T12:07:16.900160+0000","last_clean":"2024-03-24T12:06:25.366490+0000","last_became_active":"2024-03-24T12:06:27.011265+0000","last_became_peered":"2024-03-24T12:06:27.011265+0000","last_unstale":"2024-03-24T12:07:16.900160+0000","last_undegraded":"2024-03-24T12:06:26.998885+0000","last_fullsized":"2024-03-24T12:06:26.998748+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:05.860814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.860814+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:24.386351+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":41,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.891415+0000","last_change":"2024-03-24T12:06:27.012535+0000","last_active":"2024-03-24T12:07:16.891415+0000","last_peered":"2024-03-24T12:07:16.891415+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.012535+0000","last_became_peered":"2024-03-24T12:06:27.012535+0000","last_unstale":"2024-03-24T12:07:16.891415+0000","last_undegraded":"2024-03-24T12:07:16.891415+0000","last_fullsized":"2024-03-24T12:06:26.998792+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.219541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.219541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":338,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.894373+0000","last_change":"2024-03-24T12:06:27.005493+0000","last_active":"2024-03-24T12:07:16.894373+0000","last_peered":"2024-03-24T12:07:16.894373+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T12:06:27.005493+0000","last_became_peered":"2024-03-24T12:06:27.005493+0000","last_unstale":"2024-03-24T12:07:16.894373+0000","last_undegraded":"2024-03-24T12:07:16.894373+0000","last_fullsized":"2024-03-24T12:06:27.001315+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:11.242568+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.242568+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:20.743961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"335'5102","reported_seq":5142,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:21.409831+0000","last_change":"2024-03-24T12:06:30.017186+0000","last_active":"2024-03-24T12:07:21.409831+0000","last_peered":"2024-03-24T12:07:21.409831+0000","last_clean":"2024-03-24T12:07:21.409831+0000","last_became_active":"2024-03-24T12:06:30.016735+0000","last_became_peered":"2024-03-24T12:06:30.016735+0000","last_unstale":"2024-03-24T12:07:21.409831+0000","last_undegraded":"2024-03-24T12:07:21.409831+0000","last_fullsized":"2024-03-24T12:07:21.409831+0000","mapping_epoch":320,"log_start":"323'600","ondisk_log_start":"323'600","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:35.337441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334364672,"num_objects":5102,"num_object_clones":0,"num_object_copies":15306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5102,"num_write_kb":326528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"83'130","reported_seq":475,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890237+0000","last_change":"2024-03-24T12:06:27.010346+0000","last_active":"2024-03-24T12:07:16.890237+0000","last_peered":"2024-03-24T12:07:16.890237+0000","last_clean":"2024-03-24T12:06:22.261133+0000","last_became_active":"2024-03-24T12:06:27.010346+0000","last_became_peered":"2024-03-24T12:06:27.010346+0000","last_unstale":"2024-03-24T12:07:16.890237+0000","last_undegraded":"2024-03-24T12:07:16.890237+0000","last_fullsized":"2024-03-24T12:06:26.997700+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:20.649114+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:20.649114+0000","last_clean_scrub_stamp":"2024-03-24T12:05:20.649114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":631,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.896934+0000","last_change":"2024-03-24T12:06:46.690666+0000","last_active":"2024-03-24T12:07:16.896934+0000","last_peered":"2024-03-24T12:07:16.896934+0000","last_clean":"2024-03-24T12:07:16.896934+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:07:16.896934+0000","last_undegraded":"2024-03-24T12:07:16.896934+0000","last_fullsized":"2024-03-24T12:07:16.896934+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:06:46.690548+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:06:46.690548+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:48.691195+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":614,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.089089+0000","last_change":"2024-03-24T12:06:55.146672+0000","last_active":"2024-03-24T12:07:16.089089+0000","last_peered":"2024-03-24T12:07:16.089089+0000","last_clean":"2024-03-24T12:07:16.089089+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:07:16.089089+0000","last_undegraded":"2024-03-24T12:07:16.089089+0000","last_fullsized":"2024-03-24T12:07:16.089089+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:55.146558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:06:55.146558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.951968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":42,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.892310+0000","last_change":"2024-03-24T12:06:27.008428+0000","last_active":"2024-03-24T12:07:16.892310+0000","last_peered":"2024-03-24T12:07:16.892310+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.008428+0000","last_became_peered":"2024-03-24T12:06:27.008428+0000","last_unstale":"2024-03-24T12:07:16.892310+0000","last_undegraded":"2024-03-24T12:07:16.892310+0000","last_fullsized":"2024-03-24T12:06:27.000650+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:15.334211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:15.334211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:33.010069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":592,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.885018+0000","last_change":"2024-03-24T12:06:27.006596+0000","last_active":"2024-03-24T12:07:16.885018+0000","last_peered":"2024-03-24T12:07:16.885018+0000","last_clean":"2024-03-24T12:06:21.985308+0000","last_became_active":"2024-03-24T12:06:27.006596+0000","last_became_peered":"2024-03-24T12:06:27.006596+0000","last_unstale":"2024-03-24T12:07:16.885018+0000","last_undegraded":"2024-03-24T12:07:16.885018+0000","last_fullsized":"2024-03-24T12:06:27.000436+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:06.995768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:06.995768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"335'5021","reported_seq":5059,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:21.419721+0000","last_change":"2024-03-24T12:06:30.016271+0000","last_active":"2024-03-24T12:07:21.419721+0000","last_peered":"2024-03-24T12:07:21.419721+0000","last_clean":"2024-03-24T12:07:21.419721+0000","last_became_active":"2024-03-24T12:06:30.015853+0000","last_became_peered":"2024-03-24T12:06:30.015853+0000","last_unstale":"2024-03-24T12:07:21.419721+0000","last_undegraded":"2024-03-24T12:07:21.419721+0000","last_fullsized":"2024-03-24T12:07:21.419721+0000","mapping_epoch":320,"log_start":"323'500","ondisk_log_start":"323'500","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:48.009869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329056256,"num_objects":5021,"num_object_clones":0,"num_object_copies":15063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":321344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":775,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.883572+0000","last_change":"2024-03-24T12:04:13.607127+0000","last_active":"2024-03-24T12:07:16.883572+0000","last_peered":"2024-03-24T12:07:16.883572+0000","last_clean":"2024-03-24T12:07:16.883572+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:07:16.883572+0000","last_undegraded":"2024-03-24T12:07:16.883572+0000","last_fullsized":"2024-03-24T12:07:16.883572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:04:13.606837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:13.606837+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":648,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.888730+0000","last_change":"2024-03-24T12:06:27.007808+0000","last_active":"2024-03-24T12:07:16.888730+0000","last_peered":"2024-03-24T12:07:16.888730+0000","last_clean":"2024-03-24T12:06:22.260785+0000","last_became_active":"2024-03-24T12:06:27.007808+0000","last_became_peered":"2024-03-24T12:06:27.007808+0000","last_unstale":"2024-03-24T12:07:16.888730+0000","last_undegraded":"2024-03-24T12:06:27.001358+0000","last_fullsized":"2024-03-24T12:06:27.001242+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:05:29.699282+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:05:29.699282+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":606,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890284+0000","last_change":"2024-03-24T12:05:04.423091+0000","last_active":"2024-03-24T12:07:16.890284+0000","last_peered":"2024-03-24T12:07:16.890284+0000","last_clean":"2024-03-24T12:07:16.890284+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:07:16.890284+0000","last_undegraded":"2024-03-24T12:07:16.890284+0000","last_fullsized":"2024-03-24T12:07:16.890284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:04.422989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:05:04.422989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":609,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.088201+0000","last_change":"2024-03-24T12:06:15.262318+0000","last_active":"2024-03-24T12:07:16.088201+0000","last_peered":"2024-03-24T12:07:16.088201+0000","last_clean":"2024-03-24T12:07:16.088201+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:07:16.088201+0000","last_undegraded":"2024-03-24T12:07:16.088201+0000","last_fullsized":"2024-03-24T12:07:16.088201+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:15.262209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:15.262209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:26.927975+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":630,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.885133+0000","last_change":"2024-03-24T12:06:27.013509+0000","last_active":"2024-03-24T12:07:16.885133+0000","last_peered":"2024-03-24T12:07:16.885133+0000","last_clean":"2024-03-24T12:06:21.993056+0000","last_became_active":"2024-03-24T12:06:27.013509+0000","last_became_peered":"2024-03-24T12:06:27.013509+0000","last_unstale":"2024-03-24T12:07:16.885133+0000","last_undegraded":"2024-03-24T12:07:16.885133+0000","last_fullsized":"2024-03-24T12:06:27.000946+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:15.627055+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:15.627055+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"335'5003","reported_seq":5040,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:21.381708+0000","last_change":"2024-03-24T12:06:39.747440+0000","last_active":"2024-03-24T12:07:21.381708+0000","last_peered":"2024-03-24T12:07:21.381708+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.017588+0000","last_became_peered":"2024-03-24T12:06:30.017588+0000","last_unstale":"2024-03-24T12:07:21.381708+0000","last_undegraded":"2024-03-24T12:06:39.746623+0000","last_fullsized":"2024-03-24T12:06:30.012557+0000","mapping_epoch":320,"log_start":"323'700","ondisk_log_start":"323'700","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:35.351060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327876608,"num_objects":5003,"num_object_clones":0,"num_object_copies":15009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5003,"num_write_kb":320192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5003}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":592,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.889016+0000","last_change":"2024-03-24T12:06:44.708778+0000","last_active":"2024-03-24T12:07:16.889016+0000","last_peered":"2024-03-24T12:07:16.889016+0000","last_clean":"2024-03-24T12:07:16.889016+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:07:16.889016+0000","last_undegraded":"2024-03-24T12:07:16.889016+0000","last_fullsized":"2024-03-24T12:07:16.889016+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:44.708677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:06:44.708677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.105771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":656,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.883475+0000","last_change":"2024-03-24T12:06:54.925013+0000","last_active":"2024-03-24T12:07:16.883475+0000","last_peered":"2024-03-24T12:07:16.883475+0000","last_clean":"2024-03-24T12:07:16.883475+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:07:16.883475+0000","last_undegraded":"2024-03-24T12:07:16.883475+0000","last_fullsized":"2024-03-24T12:07:16.883475+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:06:54.924751+0000","last_deep_scrub":"195'26","last_deep_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.924751+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:08.422773+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":359,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.887421+0000","last_change":"2024-03-24T12:06:10.642218+0000","last_active":"2024-03-24T12:07:16.887421+0000","last_peered":"2024-03-24T12:07:16.887421+0000","last_clean":"2024-03-24T12:07:16.887421+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:07:16.887421+0000","last_undegraded":"2024-03-24T12:07:16.887421+0000","last_fullsized":"2024-03-24T12:07:16.887421+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:06:10.642123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:10.642123+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":610,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.894966+0000","last_change":"2024-03-24T12:06:39.791204+0000","last_active":"2024-03-24T12:07:16.894966+0000","last_peered":"2024-03-24T12:07:16.894966+0000","last_clean":"2024-03-24T12:07:16.894966+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:07:16.894966+0000","last_undegraded":"2024-03-24T12:07:16.894966+0000","last_fullsized":"2024-03-24T12:07:16.894966+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:39.791081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.791081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:46.673514+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":647,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890163+0000","last_change":"2024-03-24T12:06:27.008961+0000","last_active":"2024-03-24T12:07:16.890163+0000","last_peered":"2024-03-24T12:07:16.890163+0000","last_clean":"2024-03-24T12:06:22.266015+0000","last_became_active":"2024-03-24T12:06:27.008961+0000","last_became_peered":"2024-03-24T12:06:27.008961+0000","last_unstale":"2024-03-24T12:07:16.890163+0000","last_undegraded":"2024-03-24T12:07:16.890163+0000","last_fullsized":"2024-03-24T12:06:27.001539+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:18.897352+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:18.897352+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:26.744049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"335'4982","reported_seq":5022,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:21.381662+0000","last_change":"2024-03-24T12:06:30.018221+0000","last_active":"2024-03-24T12:07:21.381662+0000","last_peered":"2024-03-24T12:07:21.381662+0000","last_clean":"2024-03-24T12:07:21.381662+0000","last_became_active":"2024-03-24T12:06:30.017829+0000","last_became_peered":"2024-03-24T12:06:30.017829+0000","last_unstale":"2024-03-24T12:07:21.381662+0000","last_undegraded":"2024-03-24T12:07:21.381662+0000","last_fullsized":"2024-03-24T12:07:21.381662+0000","mapping_epoch":320,"log_start":"323'500","ondisk_log_start":"323'500","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:40.288032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326500352,"num_objects":4982,"num_object_clones":0,"num_object_copies":14946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":318848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":602,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.887131+0000","last_change":"2024-03-24T12:06:27.009137+0000","last_active":"2024-03-24T12:07:16.887131+0000","last_peered":"2024-03-24T12:07:16.887131+0000","last_clean":"2024-03-24T12:06:22.261276+0000","last_became_active":"2024-03-24T12:06:27.009137+0000","last_became_peered":"2024-03-24T12:06:27.009137+0000","last_unstale":"2024-03-24T12:07:16.887131+0000","last_undegraded":"2024-03-24T12:07:16.887131+0000","last_fullsized":"2024-03-24T12:06:26.998630+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:24.576146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:05:24.576146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"241'29","reported_seq":643,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.890965+0000","last_change":"2024-03-24T12:06:27.010256+0000","last_active":"2024-03-24T12:07:16.890965+0000","last_peered":"2024-03-24T12:07:16.890965+0000","last_clean":"2024-03-24T12:06:22.261479+0000","last_became_active":"2024-03-24T12:06:27.010256+0000","last_became_peered":"2024-03-24T12:06:27.010256+0000","last_unstale":"2024-03-24T12:07:16.890965+0000","last_undegraded":"2024-03-24T12:06:26.998858+0000","last_fullsized":"2024-03-24T12:06:26.998745+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-24T12:05:09.620298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.620298+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":615,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.088527+0000","last_change":"2024-03-24T12:05:29.324724+0000","last_active":"2024-03-24T12:07:16.088527+0000","last_peered":"2024-03-24T12:07:16.088527+0000","last_clean":"2024-03-24T12:07:16.088527+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:07:16.088527+0000","last_undegraded":"2024-03-24T12:07:16.088527+0000","last_fullsized":"2024-03-24T12:07:16.088527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:29.324593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:05:29.324593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":357,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.885365+0000","last_change":"2024-03-24T12:06:40.600601+0000","last_active":"2024-03-24T12:07:16.885365+0000","last_peered":"2024-03-24T12:07:16.885365+0000","last_clean":"2024-03-24T12:07:16.885365+0000","last_became_active":"2024-03-24T12:05:44.738926+0000","last_became_peered":"2024-03-24T12:05:44.738926+0000","last_unstale":"2024-03-24T12:07:16.885365+0000","last_undegraded":"2024-03-24T12:07:16.885365+0000","last_fullsized":"2024-03-24T12:07:16.885365+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:40.600197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:40.600197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:55.282895+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":657,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.887618+0000","last_change":"2024-03-24T12:06:49.509554+0000","last_active":"2024-03-24T12:07:16.887618+0000","last_peered":"2024-03-24T12:07:16.887618+0000","last_clean":"2024-03-24T12:07:16.887618+0000","last_became_active":"2024-03-24T12:03:15.295100+0000","last_became_peered":"2024-03-24T12:03:15.295100+0000","last_unstale":"2024-03-24T12:07:16.887618+0000","last_undegraded":"2024-03-24T12:07:16.887618+0000","last_fullsized":"2024-03-24T12:07:16.887618+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:49.509435+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.509435+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:53.252520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"335'4714","reported_seq":4754,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:18.047119+0000","last_change":"2024-03-24T12:06:30.018544+0000","last_active":"2024-03-24T12:07:18.047119+0000","last_peered":"2024-03-24T12:07:18.047119+0000","last_clean":"2024-03-24T12:07:18.047119+0000","last_became_active":"2024-03-24T12:06:30.015684+0000","last_became_peered":"2024-03-24T12:06:30.015684+0000","last_unstale":"2024-03-24T12:07:18.047119+0000","last_undegraded":"2024-03-24T12:07:18.047119+0000","last_fullsized":"2024-03-24T12:07:18.047119+0000","mapping_epoch":320,"log_start":"323'200","ondisk_log_start":"323'200","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:48.679208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308936704,"num_objects":4714,"num_object_clones":0,"num_object_copies":14142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4714,"num_write_kb":301696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":42,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890216+0000","last_change":"2024-03-24T12:06:27.009282+0000","last_active":"2024-03-24T12:07:16.890216+0000","last_peered":"2024-03-24T12:07:16.890216+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.009282+0000","last_became_peered":"2024-03-24T12:06:27.009282+0000","last_unstale":"2024-03-24T12:07:16.890216+0000","last_undegraded":"2024-03-24T12:07:16.890216+0000","last_fullsized":"2024-03-24T12:06:26.997206+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:28.232417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:28.232417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":658,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890745+0000","last_change":"2024-03-24T12:06:39.802730+0000","last_active":"2024-03-24T12:07:16.890745+0000","last_peered":"2024-03-24T12:07:16.890745+0000","last_clean":"2024-03-24T12:07:16.890745+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:07:16.890745+0000","last_undegraded":"2024-03-24T12:07:16.890745+0000","last_fullsized":"2024-03-24T12:07:16.890745+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:06:39.802577+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.802577+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:56.670488+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":437,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890856+0000","last_change":"2024-03-24T12:05:04.552941+0000","last_active":"2024-03-24T12:07:16.890856+0000","last_peered":"2024-03-24T12:07:16.890856+0000","last_clean":"2024-03-24T12:07:16.890856+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:07:16.890856+0000","last_undegraded":"2024-03-24T12:07:16.890856+0000","last_fullsized":"2024-03-24T12:07:16.890856+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:04.552647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:04.552647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":608,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890148+0000","last_change":"2024-03-24T12:05:27.651120+0000","last_active":"2024-03-24T12:07:16.890148+0000","last_peered":"2024-03-24T12:07:16.890148+0000","last_clean":"2024-03-24T12:07:16.890148+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:07:16.890148+0000","last_undegraded":"2024-03-24T12:07:16.890148+0000","last_fullsized":"2024-03-24T12:07:16.890148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:05:27.650987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":598,"reported_epoch":334,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.084970+0000","last_change":"2024-03-24T12:06:27.015212+0000","last_active":"2024-03-24T12:07:16.084970+0000","last_peered":"2024-03-24T12:07:16.084970+0000","last_clean":"2024-03-24T12:06:21.982922+0000","last_became_active":"2024-03-24T12:06:27.015212+0000","last_became_peered":"2024-03-24T12:06:27.015212+0000","last_unstale":"2024-03-24T12:07:16.084970+0000","last_undegraded":"2024-03-24T12:07:16.084970+0000","last_fullsized":"2024-03-24T12:06:27.003034+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.016987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.016987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":657,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886498+0000","last_change":"2024-03-24T12:06:54.603424+0000","last_active":"2024-03-24T12:07:16.886498+0000","last_peered":"2024-03-24T12:07:16.886498+0000","last_clean":"2024-03-24T12:07:16.886498+0000","last_became_active":"2024-03-24T12:03:15.297422+0000","last_became_peered":"2024-03-24T12:03:15.297422+0000","last_unstale":"2024-03-24T12:07:16.886498+0000","last_undegraded":"2024-03-24T12:07:16.886498+0000","last_fullsized":"2024-03-24T12:07:16.886498+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:54.603313+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.603313+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:09.509099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"335'4907","reported_seq":4947,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:21.415122+0000","last_change":"2024-03-24T12:06:30.018406+0000","last_active":"2024-03-24T12:07:21.415122+0000","last_peered":"2024-03-24T12:07:21.415122+0000","last_clean":"2024-03-24T12:07:21.415122+0000","last_became_active":"2024-03-24T12:06:30.017471+0000","last_became_peered":"2024-03-24T12:06:30.017471+0000","last_unstale":"2024-03-24T12:07:21.415122+0000","last_undegraded":"2024-03-24T12:07:21.415122+0000","last_fullsized":"2024-03-24T12:07:21.415122+0000","mapping_epoch":320,"log_start":"323'400","ondisk_log_start":"323'400","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4507,"log_dups_size":0,"ondisk_log_size":4507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:50.227687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321585152,"num_objects":4907,"num_object_clones":0,"num_object_copies":14721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":314048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":602,"reported_epoch":334,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.088336+0000","last_change":"2024-03-24T12:06:27.009330+0000","last_active":"2024-03-24T12:07:16.088336+0000","last_peered":"2024-03-24T12:07:16.088336+0000","last_clean":"2024-03-24T12:06:22.145617+0000","last_became_active":"2024-03-24T12:06:27.009330+0000","last_became_peered":"2024-03-24T12:06:27.009330+0000","last_unstale":"2024-03-24T12:07:16.088336+0000","last_undegraded":"2024-03-24T12:07:16.088336+0000","last_fullsized":"2024-03-24T12:06:27.001828+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:06.100942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.100942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":841,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886831+0000","last_change":"2024-03-24T12:06:46.537107+0000","last_active":"2024-03-24T12:07:16.886831+0000","last_peered":"2024-03-24T12:07:16.886831+0000","last_clean":"2024-03-24T12:07:16.886831+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:07:16.886831+0000","last_undegraded":"2024-03-24T12:07:16.886831+0000","last_fullsized":"2024-03-24T12:07:16.886831+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:06:46.536897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:46.536897+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:51.463836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":887,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.089055+0000","last_change":"2024-03-24T12:06:57.082478+0000","last_active":"2024-03-24T12:07:16.089055+0000","last_peered":"2024-03-24T12:07:16.089055+0000","last_clean":"2024-03-24T12:07:16.089055+0000","last_became_active":"2024-03-24T12:03:15.298475+0000","last_became_peered":"2024-03-24T12:03:15.298475+0000","last_unstale":"2024-03-24T12:07:16.089055+0000","last_undegraded":"2024-03-24T12:07:16.089055+0000","last_fullsized":"2024-03-24T12:07:16.089055+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:57.082342+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:03:16.845832+0000","last_clean_scrub_stamp":"2024-03-24T12:06:57.082342+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:57.767780+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.895524+0000","last_change":"2024-03-24T12:06:27.007334+0000","last_active":"2024-03-24T12:07:16.895524+0000","last_peered":"2024-03-24T12:07:16.895524+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.007334+0000","last_became_peered":"2024-03-24T12:06:27.007334+0000","last_unstale":"2024-03-24T12:07:16.895524+0000","last_undegraded":"2024-03-24T12:07:16.895524+0000","last_fullsized":"2024-03-24T12:06:27.001839+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.356759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.356759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":606,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.883978+0000","last_change":"2024-03-24T12:06:27.004802+0000","last_active":"2024-03-24T12:07:16.883978+0000","last_peered":"2024-03-24T12:07:16.883978+0000","last_clean":"2024-03-24T12:06:21.984854+0000","last_became_active":"2024-03-24T12:06:27.004802+0000","last_became_peered":"2024-03-24T12:06:27.004802+0000","last_unstale":"2024-03-24T12:07:16.883978+0000","last_undegraded":"2024-03-24T12:06:26.998482+0000","last_fullsized":"2024-03-24T12:06:26.998367+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:04:08.679904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:08.679904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":41,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.885827+0000","last_change":"2024-03-24T12:06:27.007879+0000","last_active":"2024-03-24T12:07:16.885827+0000","last_peered":"2024-03-24T12:07:16.885827+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.007879+0000","last_became_peered":"2024-03-24T12:06:27.007879+0000","last_unstale":"2024-03-24T12:07:16.885827+0000","last_undegraded":"2024-03-24T12:07:16.885827+0000","last_fullsized":"2024-03-24T12:06:26.999940+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.463242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.463242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"334'4666","reported_seq":4701,"reported_epoch":334,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.603211+0000","last_change":"2024-03-24T12:06:39.746288+0000","last_active":"2024-03-24T12:07:16.603211+0000","last_peered":"2024-03-24T12:07:16.603211+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.019333+0000","last_became_peered":"2024-03-24T12:06:30.019333+0000","last_unstale":"2024-03-24T12:07:16.603211+0000","last_undegraded":"2024-03-24T12:06:39.740158+0000","last_fullsized":"2024-03-24T12:06:30.012518+0000","mapping_epoch":320,"log_start":"323'400","ondisk_log_start":"323'400","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:35.455957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305790976,"num_objects":4666,"num_object_clones":0,"num_object_copies":13998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":298624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":4666}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":66,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.885489+0000","last_change":"2024-03-24T12:06:27.009761+0000","last_active":"2024-03-24T12:07:16.885489+0000","last_peered":"2024-03-24T12:07:16.885489+0000","last_clean":"2024-03-24T12:05:54.824542+0000","last_became_active":"2024-03-24T12:06:27.009761+0000","last_became_peered":"2024-03-24T12:06:27.009761+0000","last_unstale":"2024-03-24T12:07:16.885489+0000","last_undegraded":"2024-03-24T12:07:16.885489+0000","last_fullsized":"2024-03-24T12:06:27.000267+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.646040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":638,"reported_epoch":334,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.085374+0000","last_change":"2024-03-24T12:06:27.011400+0000","last_active":"2024-03-24T12:07:16.085374+0000","last_peered":"2024-03-24T12:07:16.085374+0000","last_clean":"2024-03-24T12:06:21.982394+0000","last_became_active":"2024-03-24T12:06:27.011400+0000","last_became_peered":"2024-03-24T12:06:27.011400+0000","last_unstale":"2024-03-24T12:07:16.085374+0000","last_undegraded":"2024-03-24T12:06:27.002875+0000","last_fullsized":"2024-03-24T12:06:27.002745+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:05:08.083956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:08.083956+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.885168+0000","last_change":"2024-03-24T12:05:44.739463+0000","last_active":"2024-03-24T12:07:16.885168+0000","last_peered":"2024-03-24T12:07:16.885168+0000","last_clean":"2024-03-24T12:07:16.885168+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:07:16.885168+0000","last_undegraded":"2024-03-24T12:07:16.885168+0000","last_fullsized":"2024-03-24T12:07:16.885168+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.610545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.610545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":230,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:19.887170+0000","last_change":"2024-03-24T12:06:55.648298+0000","last_active":"2024-03-24T12:07:19.887170+0000","last_peered":"2024-03-24T12:07:19.887170+0000","last_clean":"2024-03-24T12:07:19.887170+0000","last_became_active":"2024-03-24T12:05:44.736437+0000","last_became_peered":"2024-03-24T12:05:44.736437+0000","last_unstale":"2024-03-24T12:07:19.887170+0000","last_undegraded":"2024-03-24T12:07:19.887170+0000","last_fullsized":"2024-03-24T12:07:19.887170+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:06:55.648166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:55.648166+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:10.161881+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":103,"num_read_kb":98,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":595,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.088435+0000","last_change":"2024-03-24T12:05:11.223509+0000","last_active":"2024-03-24T12:07:16.088435+0000","last_peered":"2024-03-24T12:07:16.088435+0000","last_clean":"2024-03-24T12:07:16.088435+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:07:16.088435+0000","last_undegraded":"2024-03-24T12:07:16.088435+0000","last_fullsized":"2024-03-24T12:07:16.088435+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:11.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":894,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.087912+0000","last_change":"2024-03-24T12:06:56.127437+0000","last_active":"2024-03-24T12:07:16.087912+0000","last_peered":"2024-03-24T12:07:16.087912+0000","last_clean":"2024-03-24T12:07:16.087912+0000","last_became_active":"2024-03-24T12:03:15.298007+0000","last_became_peered":"2024-03-24T12:03:15.298007+0000","last_unstale":"2024-03-24T12:07:16.087912+0000","last_undegraded":"2024-03-24T12:07:16.087912+0000","last_fullsized":"2024-03-24T12:07:16.087912+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:56.127305+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:56.127305+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:13.466011+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"335'5116","reported_seq":5153,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:21.404320+0000","last_change":"2024-03-24T12:06:39.741514+0000","last_active":"2024-03-24T12:07:21.404320+0000","last_peered":"2024-03-24T12:07:21.404320+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.016584+0000","last_became_peered":"2024-03-24T12:06:30.016584+0000","last_unstale":"2024-03-24T12:07:21.404320+0000","last_undegraded":"2024-03-24T12:06:39.740877+0000","last_fullsized":"2024-03-24T12:06:30.008363+0000","mapping_epoch":320,"log_start":"323'600","ondisk_log_start":"323'600","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:55.071631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335282176,"num_objects":5116,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5116,"num_write_kb":327424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":5116}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":593,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.892493+0000","last_change":"2024-03-24T12:06:47.944634+0000","last_active":"2024-03-24T12:07:16.892493+0000","last_peered":"2024-03-24T12:07:16.892493+0000","last_clean":"2024-03-24T12:07:16.892493+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:07:16.892493+0000","last_undegraded":"2024-03-24T12:07:16.892493+0000","last_fullsized":"2024-03-24T12:07:16.892493+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:47.944504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:06:47.944504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:10.660377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"319'43","reported_seq":579,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.890921+0000","last_change":"2024-03-24T12:06:27.007856+0000","last_active":"2024-03-24T12:07:16.890921+0000","last_peered":"2024-03-24T12:07:16.890921+0000","last_clean":"2024-03-24T12:02:22.995365+0000","last_became_active":"2024-03-24T12:06:27.007856+0000","last_became_peered":"2024-03-24T12:06:27.007856+0000","last_unstale":"2024-03-24T12:07:16.890921+0000","last_undegraded":"2024-03-24T12:06:27.001627+0000","last_fullsized":"2024-03-24T12:06:27.001501+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-24T12:04:07.131550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:07.131550+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":39,"reported_epoch":334,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.086132+0000","last_change":"2024-03-24T12:06:27.006538+0000","last_active":"2024-03-24T12:07:16.086132+0000","last_peered":"2024-03-24T12:07:16.086132+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.006538+0000","last_became_peered":"2024-03-24T12:06:27.006538+0000","last_unstale":"2024-03-24T12:07:16.086132+0000","last_undegraded":"2024-03-24T12:07:16.086132+0000","last_fullsized":"2024-03-24T12:06:27.003033+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.196749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.196749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":581,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886745+0000","last_change":"2024-03-24T12:04:02.704225+0000","last_active":"2024-03-24T12:07:16.886745+0000","last_peered":"2024-03-24T12:07:16.886745+0000","last_clean":"2024-03-24T12:07:16.886745+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:07:16.886745+0000","last_undegraded":"2024-03-24T12:07:16.886745+0000","last_fullsized":"2024-03-24T12:07:16.886745+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.704086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.704086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":345,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.889875+0000","last_change":"2024-03-24T12:06:27.005399+0000","last_active":"2024-03-24T12:07:16.889875+0000","last_peered":"2024-03-24T12:07:16.889875+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T12:06:27.005399+0000","last_became_peered":"2024-03-24T12:06:27.005399+0000","last_unstale":"2024-03-24T12:07:16.889875+0000","last_undegraded":"2024-03-24T12:07:16.889875+0000","last_fullsized":"2024-03-24T12:06:26.996752+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:47.152371+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:47.152371+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:21.143659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"334'4605","reported_seq":4643,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.612388+0000","last_change":"2024-03-24T12:06:30.018151+0000","last_active":"2024-03-24T12:07:16.612388+0000","last_peered":"2024-03-24T12:07:16.612388+0000","last_clean":"2024-03-24T12:07:16.612388+0000","last_became_active":"2024-03-24T12:06:30.017875+0000","last_became_peered":"2024-03-24T12:06:30.017875+0000","last_unstale":"2024-03-24T12:07:16.612388+0000","last_undegraded":"2024-03-24T12:07:16.612388+0000","last_fullsized":"2024-03-24T12:07:16.612388+0000","mapping_epoch":320,"log_start":"323'300","ondisk_log_start":"323'300","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:55.469562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301793280,"num_objects":4605,"num_object_clones":0,"num_object_copies":13815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4605,"num_write_kb":294720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"319'9","reported_seq":786,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.891372+0000","last_change":"2024-03-24T12:06:27.009093+0000","last_active":"2024-03-24T12:07:16.891372+0000","last_peered":"2024-03-24T12:07:16.891372+0000","last_clean":"2024-03-24T12:06:25.366554+0000","last_became_active":"2024-03-24T12:06:27.009093+0000","last_became_peered":"2024-03-24T12:06:27.009093+0000","last_unstale":"2024-03-24T12:07:16.891372+0000","last_undegraded":"2024-03-24T12:06:26.999413+0000","last_fullsized":"2024-03-24T12:06:26.999301+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:02.500062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.500062+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":641,"reported_epoch":334,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.143298+0000","last_change":"2024-03-24T12:06:27.010753+0000","last_active":"2024-03-24T12:07:16.143298+0000","last_peered":"2024-03-24T12:07:16.143298+0000","last_clean":"2024-03-24T12:06:21.984527+0000","last_became_active":"2024-03-24T12:06:27.010753+0000","last_became_peered":"2024-03-24T12:06:27.010753+0000","last_unstale":"2024-03-24T12:07:16.143298+0000","last_undegraded":"2024-03-24T12:06:27.004797+0000","last_fullsized":"2024-03-24T12:06:27.004688+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:05.061844+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.061844+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":351,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.897801+0000","last_change":"2024-03-24T12:06:49.982193+0000","last_active":"2024-03-24T12:07:16.897801+0000","last_peered":"2024-03-24T12:07:16.897801+0000","last_clean":"2024-03-24T12:07:16.897801+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:07:16.897801+0000","last_undegraded":"2024-03-24T12:07:16.897801+0000","last_fullsized":"2024-03-24T12:07:16.897801+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:49.981897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.981897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:05.069700+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":590,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890287+0000","last_change":"2024-03-24T12:06:27.011434+0000","last_active":"2024-03-24T12:07:16.890287+0000","last_peered":"2024-03-24T12:07:16.890287+0000","last_clean":"2024-03-24T12:06:22.260451+0000","last_became_active":"2024-03-24T12:06:27.011434+0000","last_became_peered":"2024-03-24T12:06:27.011434+0000","last_unstale":"2024-03-24T12:07:16.890287+0000","last_undegraded":"2024-03-24T12:07:16.890287+0000","last_fullsized":"2024-03-24T12:06:26.996364+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:13.648498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.648498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":649,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.889947+0000","last_change":"2024-03-24T12:05:23.690448+0000","last_active":"2024-03-24T12:07:16.889947+0000","last_peered":"2024-03-24T12:07:16.889947+0000","last_clean":"2024-03-24T12:07:16.889947+0000","last_became_active":"2024-03-24T12:03:15.298983+0000","last_became_peered":"2024-03-24T12:03:15.298983+0000","last_unstale":"2024-03-24T12:07:16.889947+0000","last_undegraded":"2024-03-24T12:07:16.889947+0000","last_fullsized":"2024-03-24T12:07:16.889947+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:05:23.690315+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"335'5132","reported_seq":5169,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:21.390830+0000","last_change":"2024-03-24T12:06:39.755347+0000","last_active":"2024-03-24T12:07:21.390830+0000","last_peered":"2024-03-24T12:07:21.390830+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.015883+0000","last_became_peered":"2024-03-24T12:06:30.015883+0000","last_unstale":"2024-03-24T12:07:21.390830+0000","last_undegraded":"2024-03-24T12:06:39.745221+0000","last_fullsized":"2024-03-24T12:06:30.008879+0000","mapping_epoch":320,"log_start":"323'600","ondisk_log_start":"323'600","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:45.406545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336330752,"num_objects":5132,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5132,"num_write_kb":328448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":5132}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"83'130","reported_seq":640,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.884787+0000","last_change":"2024-03-24T12:03:43.603483+0000","last_active":"2024-03-24T12:07:16.884787+0000","last_peered":"2024-03-24T12:07:16.884787+0000","last_clean":"2024-03-24T12:07:16.884787+0000","last_became_active":"2024-03-24T12:03:19.827283+0000","last_became_peered":"2024-03-24T12:03:19.827283+0000","last_unstale":"2024-03-24T12:07:16.884787+0000","last_undegraded":"2024-03-24T12:07:16.884787+0000","last_fullsized":"2024-03-24T12:07:16.884787+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:43.603335+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:43.603335+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":609,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.894931+0000","last_change":"2024-03-24T12:06:43.893998+0000","last_active":"2024-03-24T12:07:16.894931+0000","last_peered":"2024-03-24T12:07:16.894931+0000","last_clean":"2024-03-24T12:07:16.894931+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:07:16.894931+0000","last_undegraded":"2024-03-24T12:07:16.894931+0000","last_fullsized":"2024-03-24T12:07:16.894931+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:43.893854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:43.893854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:11.959417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":653,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.888932+0000","last_change":"2024-03-24T12:06:13.937378+0000","last_active":"2024-03-24T12:07:16.888932+0000","last_peered":"2024-03-24T12:07:16.888932+0000","last_clean":"2024-03-24T12:07:16.888932+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:07:16.888932+0000","last_undegraded":"2024-03-24T12:07:16.888932+0000","last_fullsized":"2024-03-24T12:07:16.888932+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:06:13.937275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:13.937275+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:30.540827+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":611,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.892153+0000","last_change":"2024-03-24T12:03:53.808708+0000","last_active":"2024-03-24T12:07:16.892153+0000","last_peered":"2024-03-24T12:07:16.892153+0000","last_clean":"2024-03-24T12:07:16.892153+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:07:16.892153+0000","last_undegraded":"2024-03-24T12:07:16.892153+0000","last_fullsized":"2024-03-24T12:07:16.892153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:03:53.808558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:53.808558+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481107+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":581,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886852+0000","last_change":"2024-03-24T12:04:55.858996+0000","last_active":"2024-03-24T12:07:16.886852+0000","last_peered":"2024-03-24T12:07:16.886852+0000","last_clean":"2024-03-24T12:07:16.886852+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:07:16.886852+0000","last_undegraded":"2024-03-24T12:07:16.886852+0000","last_fullsized":"2024-03-24T12:07:16.886852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:55.858730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:04:55.858730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"83'130","reported_seq":473,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.884560+0000","last_change":"2024-03-24T12:06:52.949501+0000","last_active":"2024-03-24T12:07:16.884560+0000","last_peered":"2024-03-24T12:07:16.884560+0000","last_clean":"2024-03-24T12:07:16.884560+0000","last_became_active":"2024-03-24T12:03:19.828385+0000","last_became_peered":"2024-03-24T12:03:19.828385+0000","last_unstale":"2024-03-24T12:07:16.884560+0000","last_undegraded":"2024-03-24T12:07:16.884560+0000","last_fullsized":"2024-03-24T12:07:16.884560+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:52.949381+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:52.949381+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.163619+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":625,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890848+0000","last_change":"2024-03-24T12:06:19.899098+0000","last_active":"2024-03-24T12:07:16.890848+0000","last_peered":"2024-03-24T12:07:16.890848+0000","last_clean":"2024-03-24T12:07:16.890848+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:07:16.890848+0000","last_undegraded":"2024-03-24T12:07:16.890848+0000","last_fullsized":"2024-03-24T12:07:16.890848+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:06:19.899003+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:06:19.899003+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:21.059792+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":89,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890596+0000","last_change":"2024-03-24T12:06:51.959021+0000","last_active":"2024-03-24T12:07:16.890596+0000","last_peered":"2024-03-24T12:07:16.890596+0000","last_clean":"2024-03-24T12:07:16.890596+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:07:16.890596+0000","last_undegraded":"2024-03-24T12:07:16.890596+0000","last_fullsized":"2024-03-24T12:07:16.890596+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:51.958902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:51.958902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:59.863779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":585,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.885740+0000","last_change":"2024-03-24T12:06:42.950243+0000","last_active":"2024-03-24T12:07:16.885740+0000","last_peered":"2024-03-24T12:07:16.885740+0000","last_clean":"2024-03-24T12:07:16.885740+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:07:16.885740+0000","last_undegraded":"2024-03-24T12:07:16.885740+0000","last_fullsized":"2024-03-24T12:07:16.885740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.950004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.950004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:10.968575+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":80,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890544+0000","last_change":"2024-03-24T12:05:44.736346+0000","last_active":"2024-03-24T12:07:16.890544+0000","last_peered":"2024-03-24T12:07:16.890544+0000","last_clean":"2024-03-24T12:07:16.890544+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:07:16.890544+0000","last_undegraded":"2024-03-24T12:07:16.890544+0000","last_fullsized":"2024-03-24T12:07:16.890544+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:49.349208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:49.349208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"83'130","reported_seq":337,"reported_epoch":334,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.085828+0000","last_change":"2024-03-24T12:06:27.009499+0000","last_active":"2024-03-24T12:07:16.085828+0000","last_peered":"2024-03-24T12:07:16.085828+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T12:06:27.009499+0000","last_became_peered":"2024-03-24T12:06:27.009499+0000","last_unstale":"2024-03-24T12:07:16.085828+0000","last_undegraded":"2024-03-24T12:06:27.003936+0000","last_fullsized":"2024-03-24T12:06:27.003811+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:48.080209+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.080209+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:16.311215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":615,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.885591+0000","last_change":"2024-03-24T12:06:49.874361+0000","last_active":"2024-03-24T12:07:16.885591+0000","last_peered":"2024-03-24T12:07:16.885591+0000","last_clean":"2024-03-24T12:07:16.885591+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:07:16.885591+0000","last_undegraded":"2024-03-24T12:07:16.885591+0000","last_fullsized":"2024-03-24T12:07:16.885591+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:06:49.873894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.873894+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:51.938555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":614,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.885537+0000","last_change":"2024-03-24T12:06:45.566720+0000","last_active":"2024-03-24T12:07:16.885537+0000","last_peered":"2024-03-24T12:07:16.885537+0000","last_clean":"2024-03-24T12:07:16.885537+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:07:16.885537+0000","last_undegraded":"2024-03-24T12:07:16.885537+0000","last_fullsized":"2024-03-24T12:07:16.885537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:45.566608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:06:45.566608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:50.604753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":42,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.886342+0000","last_change":"2024-03-24T12:06:27.013558+0000","last_active":"2024-03-24T12:07:16.886342+0000","last_peered":"2024-03-24T12:07:16.886342+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.013558+0000","last_became_peered":"2024-03-24T12:06:27.013558+0000","last_unstale":"2024-03-24T12:07:16.886342+0000","last_undegraded":"2024-03-24T12:07:16.886342+0000","last_fullsized":"2024-03-24T12:06:27.000884+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:11.242845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:11.242845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":597,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.088285+0000","last_change":"2024-03-24T12:05:17.165172+0000","last_active":"2024-03-24T12:07:16.088285+0000","last_peered":"2024-03-24T12:07:16.088285+0000","last_clean":"2024-03-24T12:07:16.088285+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:07:16.088285+0000","last_undegraded":"2024-03-24T12:07:16.088285+0000","last_fullsized":"2024-03-24T12:07:16.088285+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.165061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"83'130","reported_seq":487,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.889581+0000","last_change":"2024-03-24T12:06:27.004114+0000","last_active":"2024-03-24T12:07:16.889581+0000","last_peered":"2024-03-24T12:07:16.889581+0000","last_clean":"2024-03-24T12:06:22.260475+0000","last_became_active":"2024-03-24T12:06:27.004114+0000","last_became_peered":"2024-03-24T12:06:27.004114+0000","last_unstale":"2024-03-24T12:07:16.889581+0000","last_undegraded":"2024-03-24T12:07:16.889581+0000","last_fullsized":"2024-03-24T12:06:26.996349+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:21.650437+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:21.650437+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":632,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.884917+0000","last_change":"2024-03-24T12:06:53.944441+0000","last_active":"2024-03-24T12:07:16.884917+0000","last_peered":"2024-03-24T12:07:16.884917+0000","last_clean":"2024-03-24T12:07:16.884917+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:07:16.884917+0000","last_undegraded":"2024-03-24T12:07:16.884917+0000","last_fullsized":"2024-03-24T12:07:16.884917+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:06:53.944281+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:06:53.944281+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.886487+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":41,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.884709+0000","last_change":"2024-03-24T12:06:27.010653+0000","last_active":"2024-03-24T12:07:16.884709+0000","last_peered":"2024-03-24T12:07:16.884709+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.010653+0000","last_became_peered":"2024-03-24T12:06:27.010653+0000","last_unstale":"2024-03-24T12:07:16.884709+0000","last_undegraded":"2024-03-24T12:07:16.884709+0000","last_fullsized":"2024-03-24T12:06:27.000433+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.284540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.284540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"319'9","reported_seq":77,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.891754+0000","last_change":"2024-03-24T12:06:27.013138+0000","last_active":"2024-03-24T12:07:16.891754+0000","last_peered":"2024-03-24T12:07:16.891754+0000","last_clean":"2024-03-24T11:52:22.536407+0000","last_became_active":"2024-03-24T12:06:27.013138+0000","last_became_peered":"2024-03-24T12:06:27.013138+0000","last_unstale":"2024-03-24T12:07:16.891754+0000","last_undegraded":"2024-03-24T12:06:26.999373+0000","last_fullsized":"2024-03-24T12:06:26.999255+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:03:51.155727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.155727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":588,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890647+0000","last_change":"2024-03-24T12:06:12.019392+0000","last_active":"2024-03-24T12:07:16.890647+0000","last_peered":"2024-03-24T12:07:16.890647+0000","last_clean":"2024-03-24T12:07:16.890647+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:07:16.890647+0000","last_undegraded":"2024-03-24T12:07:16.890647+0000","last_fullsized":"2024-03-24T12:07:16.890647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:12.019278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:12.019278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:29.648882+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"83'130","reported_seq":748,"reported_epoch":334,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.087317+0000","last_change":"2024-03-24T12:06:27.008690+0000","last_active":"2024-03-24T12:07:16.087317+0000","last_peered":"2024-03-24T12:07:16.087317+0000","last_clean":"2024-03-24T12:03:16.298442+0000","last_became_active":"2024-03-24T12:06:27.008690+0000","last_became_peered":"2024-03-24T12:06:27.008690+0000","last_unstale":"2024-03-24T12:07:16.087317+0000","last_undegraded":"2024-03-24T12:07:16.087317+0000","last_fullsized":"2024-03-24T12:06:27.002281+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:10.347460+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:10.347460+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":451,"reported_epoch":334,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.088391+0000","last_change":"2024-03-24T12:06:27.013827+0000","last_active":"2024-03-24T12:07:16.088391+0000","last_peered":"2024-03-24T12:07:16.088391+0000","last_clean":"2024-03-24T12:02:22.991378+0000","last_became_active":"2024-03-24T12:06:27.013827+0000","last_became_peered":"2024-03-24T12:06:27.013827+0000","last_unstale":"2024-03-24T12:07:16.088391+0000","last_undegraded":"2024-03-24T12:06:27.004993+0000","last_fullsized":"2024-03-24T12:06:27.004873+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":592,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891180+0000","last_change":"2024-03-24T12:05:10.648412+0000","last_active":"2024-03-24T12:07:16.891180+0000","last_peered":"2024-03-24T12:07:16.891180+0000","last_clean":"2024-03-24T12:07:16.891180+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T12:07:16.891180+0000","last_undegraded":"2024-03-24T12:07:16.891180+0000","last_fullsized":"2024-03-24T12:07:16.891180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.648144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.648144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":792,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890792+0000","last_change":"2024-03-24T12:06:10.014367+0000","last_active":"2024-03-24T12:07:16.890792+0000","last_peered":"2024-03-24T12:07:16.890792+0000","last_clean":"2024-03-24T12:07:16.890792+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:07:16.890792+0000","last_undegraded":"2024-03-24T12:07:16.890792+0000","last_fullsized":"2024-03-24T12:07:16.890792+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:06:10.014224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:10.014224+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:35.223134+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":590,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.892488+0000","last_change":"2024-03-24T12:06:27.005026+0000","last_active":"2024-03-24T12:07:16.892488+0000","last_peered":"2024-03-24T12:07:16.892488+0000","last_clean":"2024-03-24T12:06:22.261051+0000","last_became_active":"2024-03-24T12:06:27.005026+0000","last_became_peered":"2024-03-24T12:06:27.005026+0000","last_unstale":"2024-03-24T12:07:16.892488+0000","last_undegraded":"2024-03-24T12:07:16.892488+0000","last_fullsized":"2024-03-24T12:06:27.000977+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.041566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.041566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481257+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":395,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886520+0000","last_change":"2024-03-24T12:06:08.593650+0000","last_active":"2024-03-24T12:07:16.886520+0000","last_peered":"2024-03-24T12:07:16.886520+0000","last_clean":"2024-03-24T12:07:16.886520+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:07:16.886520+0000","last_undegraded":"2024-03-24T12:07:16.886520+0000","last_fullsized":"2024-03-24T12:07:16.886520+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:06:08.593424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:08.593424+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":609,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.886418+0000","last_change":"2024-03-24T12:06:27.008866+0000","last_active":"2024-03-24T12:07:16.886418+0000","last_peered":"2024-03-24T12:07:16.886418+0000","last_clean":"2024-03-24T12:06:22.268390+0000","last_became_active":"2024-03-24T12:06:27.008866+0000","last_became_peered":"2024-03-24T12:06:27.008866+0000","last_unstale":"2024-03-24T12:07:16.886418+0000","last_undegraded":"2024-03-24T12:07:16.886418+0000","last_fullsized":"2024-03-24T12:06:27.000626+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:05.334379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:04:05.334379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":591,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886771+0000","last_change":"2024-03-24T12:05:56.893513+0000","last_active":"2024-03-24T12:07:16.886771+0000","last_peered":"2024-03-24T12:07:16.886771+0000","last_clean":"2024-03-24T12:07:16.886771+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:07:16.886771+0000","last_undegraded":"2024-03-24T12:07:16.886771+0000","last_fullsized":"2024-03-24T12:07:16.886771+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.669210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.669210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":591,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890060+0000","last_change":"2024-03-24T12:06:27.013848+0000","last_active":"2024-03-24T12:07:16.890060+0000","last_peered":"2024-03-24T12:07:16.890060+0000","last_clean":"2024-03-24T12:06:23.013980+0000","last_became_active":"2024-03-24T12:06:27.013848+0000","last_became_peered":"2024-03-24T12:06:27.013848+0000","last_unstale":"2024-03-24T12:07:16.890060+0000","last_undegraded":"2024-03-24T12:07:16.890060+0000","last_fullsized":"2024-03-24T12:06:27.006706+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:22.691279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T12:05:22.691279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":626,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.888898+0000","last_change":"2024-03-24T12:06:50.625611+0000","last_active":"2024-03-24T12:07:16.888898+0000","last_peered":"2024-03-24T12:07:16.888898+0000","last_clean":"2024-03-24T12:07:16.888898+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:07:16.888898+0000","last_undegraded":"2024-03-24T12:07:16.888898+0000","last_fullsized":"2024-03-24T12:07:16.888898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:06:50.625428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:50.625428+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:58.481142+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":617,"reported_epoch":334,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.142422+0000","last_change":"2024-03-24T12:06:27.015745+0000","last_active":"2024-03-24T12:07:16.142422+0000","last_peered":"2024-03-24T12:07:16.142422+0000","last_clean":"2024-03-24T12:06:21.984264+0000","last_became_active":"2024-03-24T12:06:27.015745+0000","last_became_peered":"2024-03-24T12:06:27.015745+0000","last_unstale":"2024-03-24T12:07:16.142422+0000","last_undegraded":"2024-03-24T12:07:16.142422+0000","last_fullsized":"2024-03-24T12:06:27.004103+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:13.110170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.110170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":44,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.886182+0000","last_change":"2024-03-24T12:06:27.010505+0000","last_active":"2024-03-24T12:07:16.886182+0000","last_peered":"2024-03-24T12:07:16.886182+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.010505+0000","last_became_peered":"2024-03-24T12:06:27.010505+0000","last_unstale":"2024-03-24T12:07:16.886182+0000","last_undegraded":"2024-03-24T12:07:16.886182+0000","last_fullsized":"2024-03-24T12:06:27.000808+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:10.080832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:03:10.080832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":584,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.884424+0000","last_change":"2024-03-24T12:06:27.006336+0000","last_active":"2024-03-24T12:07:16.884424+0000","last_peered":"2024-03-24T12:07:16.884424+0000","last_clean":"2024-03-24T12:06:22.633690+0000","last_became_active":"2024-03-24T12:06:27.006336+0000","last_became_peered":"2024-03-24T12:06:27.006336+0000","last_unstale":"2024-03-24T12:07:16.884424+0000","last_undegraded":"2024-03-24T12:07:16.884424+0000","last_fullsized":"2024-03-24T12:06:26.998332+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:25.928874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.928874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":611,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.883315+0000","last_change":"2024-03-24T12:05:18.756844+0000","last_active":"2024-03-24T12:07:16.883315+0000","last_peered":"2024-03-24T12:07:16.883315+0000","last_clean":"2024-03-24T12:07:16.883315+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:07:16.883315+0000","last_undegraded":"2024-03-24T12:07:16.883315+0000","last_fullsized":"2024-03-24T12:07:16.883315+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:05:18.756720+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:05:18.756720+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":560,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.886781+0000","last_change":"2024-03-24T12:06:27.011102+0000","last_active":"2024-03-24T12:07:16.886781+0000","last_peered":"2024-03-24T12:07:16.886781+0000","last_clean":"2024-03-24T12:05:37.533531+0000","last_became_active":"2024-03-24T12:06:27.011102+0000","last_became_peered":"2024-03-24T12:06:27.011102+0000","last_unstale":"2024-03-24T12:07:16.886781+0000","last_undegraded":"2024-03-24T12:07:16.886781+0000","last_fullsized":"2024-03-24T12:06:27.001860+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:18.028433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:18.028433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":606,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.885676+0000","last_change":"2024-03-24T12:06:42.538679+0000","last_active":"2024-03-24T12:07:16.885676+0000","last_peered":"2024-03-24T12:07:16.885676+0000","last_clean":"2024-03-24T12:07:16.885676+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:07:16.885676+0000","last_undegraded":"2024-03-24T12:07:16.885676+0000","last_fullsized":"2024-03-24T12:07:16.885676+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.538568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.538568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:59.350306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":594,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.087945+0000","last_change":"2024-03-24T12:06:39.164062+0000","last_active":"2024-03-24T12:07:16.087945+0000","last_peered":"2024-03-24T12:07:16.087945+0000","last_clean":"2024-03-24T12:07:16.087945+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:07:16.087945+0000","last_undegraded":"2024-03-24T12:07:16.087945+0000","last_fullsized":"2024-03-24T12:07:16.087945+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:39.163951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.163951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:47.639637+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"83'130","reported_seq":645,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.895206+0000","last_change":"2024-03-24T12:06:12.854883+0000","last_active":"2024-03-24T12:07:16.895206+0000","last_peered":"2024-03-24T12:07:16.895206+0000","last_clean":"2024-03-24T12:07:16.895206+0000","last_became_active":"2024-03-24T12:03:19.819346+0000","last_became_peered":"2024-03-24T12:03:19.819346+0000","last_unstale":"2024-03-24T12:07:16.895206+0000","last_undegraded":"2024-03-24T12:07:16.895206+0000","last_fullsized":"2024-03-24T12:07:16.895206+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:12.854773+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:12.854773+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:40.786739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":620,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.885318+0000","last_change":"2024-03-24T12:06:13.055590+0000","last_active":"2024-03-24T12:07:16.885318+0000","last_peered":"2024-03-24T12:07:16.885318+0000","last_clean":"2024-03-24T12:07:16.885318+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:07:16.885318+0000","last_undegraded":"2024-03-24T12:07:16.885318+0000","last_fullsized":"2024-03-24T12:07:16.885318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:06:13.055489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:13.055489+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:36.305181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":607,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890036+0000","last_change":"2024-03-24T12:06:27.005897+0000","last_active":"2024-03-24T12:07:16.890036+0000","last_peered":"2024-03-24T12:07:16.890036+0000","last_clean":"2024-03-24T12:06:22.267726+0000","last_became_active":"2024-03-24T12:06:27.005897+0000","last_became_peered":"2024-03-24T12:06:27.005897+0000","last_unstale":"2024-03-24T12:07:16.890036+0000","last_undegraded":"2024-03-24T12:07:16.890036+0000","last_fullsized":"2024-03-24T12:06:27.001669+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.372798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.372798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":598,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.089185+0000","last_change":"2024-03-24T12:06:12.258848+0000","last_active":"2024-03-24T12:07:16.089185+0000","last_peered":"2024-03-24T12:07:16.089185+0000","last_clean":"2024-03-24T12:07:16.089185+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:07:16.089185+0000","last_undegraded":"2024-03-24T12:07:16.089185+0000","last_fullsized":"2024-03-24T12:07:16.089185+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:12.258730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:06:12.258730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:32.118364+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":581,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.887038+0000","last_change":"2024-03-24T12:06:18.013886+0000","last_active":"2024-03-24T12:07:16.887038+0000","last_peered":"2024-03-24T12:07:16.887038+0000","last_clean":"2024-03-24T12:07:16.887038+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:07:16.887038+0000","last_undegraded":"2024-03-24T12:07:16.887038+0000","last_fullsized":"2024-03-24T12:07:16.887038+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:18.013768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:18.013768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:22.950266+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":632,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891345+0000","last_change":"2024-03-24T12:05:23.694227+0000","last_active":"2024-03-24T12:07:16.891345+0000","last_peered":"2024-03-24T12:07:16.891345+0000","last_clean":"2024-03-24T12:07:16.891345+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:07:16.891345+0000","last_undegraded":"2024-03-24T12:07:16.891345+0000","last_fullsized":"2024-03-24T12:07:16.891345+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:23.694067+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:05:23.694067+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":41,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.891753+0000","last_change":"2024-03-24T12:06:27.007241+0000","last_active":"2024-03-24T12:07:16.891753+0000","last_peered":"2024-03-24T12:07:16.891753+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.007241+0000","last_became_peered":"2024-03-24T12:06:27.007241+0000","last_unstale":"2024-03-24T12:07:16.891753+0000","last_undegraded":"2024-03-24T12:07:16.891753+0000","last_fullsized":"2024-03-24T12:06:27.000285+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.160982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.160982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":793,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890466+0000","last_change":"2024-03-24T12:06:41.566287+0000","last_active":"2024-03-24T12:07:16.890466+0000","last_peered":"2024-03-24T12:07:16.890466+0000","last_clean":"2024-03-24T12:07:16.890466+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:07:16.890466+0000","last_undegraded":"2024-03-24T12:07:16.890466+0000","last_fullsized":"2024-03-24T12:07:16.890466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:06:41.566151+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:06:41.566151+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:46.127618+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":606,"reported_epoch":334,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.088462+0000","last_change":"2024-03-24T12:06:27.009504+0000","last_active":"2024-03-24T12:07:16.088462+0000","last_peered":"2024-03-24T12:07:16.088462+0000","last_clean":"2024-03-24T12:06:22.145780+0000","last_became_active":"2024-03-24T12:06:27.009504+0000","last_became_peered":"2024-03-24T12:06:27.009504+0000","last_unstale":"2024-03-24T12:07:16.088462+0000","last_undegraded":"2024-03-24T12:06:27.002365+0000","last_fullsized":"2024-03-24T12:06:27.002251+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:05:20.119650+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:20.119650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":645,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.890514+0000","last_change":"2024-03-24T12:06:27.007431+0000","last_active":"2024-03-24T12:07:16.890514+0000","last_peered":"2024-03-24T12:07:16.890514+0000","last_clean":"2024-03-24T12:06:22.268056+0000","last_became_active":"2024-03-24T12:06:27.007431+0000","last_became_peered":"2024-03-24T12:06:27.007431+0000","last_unstale":"2024-03-24T12:07:16.890514+0000","last_undegraded":"2024-03-24T12:06:27.002523+0000","last_fullsized":"2024-03-24T12:06:27.002390+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:05:25.602806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.602806+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":594,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.884950+0000","last_change":"2024-03-24T12:06:40.021255+0000","last_active":"2024-03-24T12:07:16.884950+0000","last_peered":"2024-03-24T12:07:16.884950+0000","last_clean":"2024-03-24T12:07:16.884950+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:07:16.884950+0000","last_undegraded":"2024-03-24T12:07:16.884950+0000","last_fullsized":"2024-03-24T12:07:16.884950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:06:40.021114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:40.021114+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:44.158067+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":599,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891449+0000","last_change":"2024-03-24T12:05:44.742947+0000","last_active":"2024-03-24T12:07:16.891449+0000","last_peered":"2024-03-24T12:07:16.891449+0000","last_clean":"2024-03-24T12:07:16.891449+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:07:16.891449+0000","last_undegraded":"2024-03-24T12:07:16.891449+0000","last_fullsized":"2024-03-24T12:07:16.891449+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:16.684742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.684742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":43,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890367+0000","last_change":"2024-03-24T12:06:27.011917+0000","last_active":"2024-03-24T12:07:16.890367+0000","last_peered":"2024-03-24T12:07:16.890367+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.011917+0000","last_became_peered":"2024-03-24T12:06:27.011917+0000","last_unstale":"2024-03-24T12:07:16.890367+0000","last_undegraded":"2024-03-24T12:07:16.890367+0000","last_fullsized":"2024-03-24T12:06:27.001875+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:04:14.188600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":543,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891154+0000","last_change":"2024-03-24T12:05:44.763045+0000","last_active":"2024-03-24T12:07:16.891154+0000","last_peered":"2024-03-24T12:07:16.891154+0000","last_clean":"2024-03-24T12:07:16.891154+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:07:16.891154+0000","last_undegraded":"2024-03-24T12:07:16.891154+0000","last_fullsized":"2024-03-24T12:07:16.891154+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:05:14.526488+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:05:14.526488+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":614,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.887056+0000","last_change":"2024-03-24T12:06:57.653082+0000","last_active":"2024-03-24T12:07:16.887056+0000","last_peered":"2024-03-24T12:07:16.887056+0000","last_clean":"2024-03-24T12:07:16.887056+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:07:16.887056+0000","last_undegraded":"2024-03-24T12:07:16.887056+0000","last_fullsized":"2024-03-24T12:07:16.887056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:57.653004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:06:57.653004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:27.542506+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":352,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891272+0000","last_change":"2024-03-24T12:04:57.823173+0000","last_active":"2024-03-24T12:07:16.891272+0000","last_peered":"2024-03-24T12:07:16.891272+0000","last_clean":"2024-03-24T12:07:16.891272+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:07:16.891272+0000","last_undegraded":"2024-03-24T12:07:16.891272+0000","last_fullsized":"2024-03-24T12:07:16.891272+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:57.823045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:57.823045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481311+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":594,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.897030+0000","last_change":"2024-03-24T12:06:27.008492+0000","last_active":"2024-03-24T12:07:16.897030+0000","last_peered":"2024-03-24T12:07:16.897030+0000","last_clean":"2024-03-24T12:06:22.267344+0000","last_became_active":"2024-03-24T12:06:27.008492+0000","last_became_peered":"2024-03-24T12:06:27.008492+0000","last_unstale":"2024-03-24T12:07:16.897030+0000","last_undegraded":"2024-03-24T12:07:16.897030+0000","last_fullsized":"2024-03-24T12:06:27.002393+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:53.809021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:53.809021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481245+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":643,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.885342+0000","last_change":"2024-03-24T12:05:01.677576+0000","last_active":"2024-03-24T12:07:16.885342+0000","last_peered":"2024-03-24T12:07:16.885342+0000","last_clean":"2024-03-24T12:07:16.885342+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:07:16.885342+0000","last_undegraded":"2024-03-24T12:07:16.885342+0000","last_fullsized":"2024-03-24T12:07:16.885342+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:05:01.675875+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:05:01.675875+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":599,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.886722+0000","last_change":"2024-03-24T12:03:51.776115+0000","last_active":"2024-03-24T12:07:16.886722+0000","last_peered":"2024-03-24T12:07:16.886722+0000","last_clean":"2024-03-24T12:07:16.886722+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:07:16.886722+0000","last_undegraded":"2024-03-24T12:07:16.886722+0000","last_fullsized":"2024-03-24T12:07:16.886722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:03:51.775978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.775978+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":597,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890724+0000","last_change":"2024-03-24T12:06:42.722854+0000","last_active":"2024-03-24T12:07:16.890724+0000","last_peered":"2024-03-24T12:07:16.890724+0000","last_clean":"2024-03-24T12:07:16.890724+0000","last_became_active":"2024-03-24T12:05:44.736383+0000","last_became_peered":"2024-03-24T12:05:44.736383+0000","last_unstale":"2024-03-24T12:07:16.890724+0000","last_undegraded":"2024-03-24T12:07:16.890724+0000","last_fullsized":"2024-03-24T12:07:16.890724+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.722733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.722733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:10.779416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":585,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.885408+0000","last_change":"2024-03-24T12:06:27.005514+0000","last_active":"2024-03-24T12:07:16.885408+0000","last_peered":"2024-03-24T12:07:16.885408+0000","last_clean":"2024-03-24T12:06:21.983027+0000","last_became_active":"2024-03-24T12:06:27.005514+0000","last_became_peered":"2024-03-24T12:06:27.005514+0000","last_unstale":"2024-03-24T12:07:16.885408+0000","last_undegraded":"2024-03-24T12:07:16.885408+0000","last_fullsized":"2024-03-24T12:06:26.999436+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:19.844413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.844413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":620,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.893877+0000","last_change":"2024-03-24T12:03:44.894651+0000","last_active":"2024-03-24T12:07:16.893877+0000","last_peered":"2024-03-24T12:07:16.893877+0000","last_clean":"2024-03-24T12:07:16.893877+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:07:16.893877+0000","last_undegraded":"2024-03-24T12:07:16.893877+0000","last_fullsized":"2024-03-24T12:07:16.893877+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:03:44.894523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:03:44.894523+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":612,"reported_epoch":334,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.088305+0000","last_change":"2024-03-24T12:06:27.009141+0000","last_active":"2024-03-24T12:07:16.088305+0000","last_peered":"2024-03-24T12:07:16.088305+0000","last_clean":"2024-03-24T12:06:22.145387+0000","last_became_active":"2024-03-24T12:06:27.009141+0000","last_became_peered":"2024-03-24T12:06:27.009141+0000","last_unstale":"2024-03-24T12:07:16.088305+0000","last_undegraded":"2024-03-24T12:07:16.088305+0000","last_fullsized":"2024-03-24T12:06:27.001371+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.177509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:21.099318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":42,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.890525+0000","last_change":"2024-03-24T12:06:27.009488+0000","last_active":"2024-03-24T12:07:16.890525+0000","last_peered":"2024-03-24T12:07:16.890525+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.009488+0000","last_became_peered":"2024-03-24T12:06:27.009488+0000","last_unstale":"2024-03-24T12:07:16.890525+0000","last_undegraded":"2024-03-24T12:07:16.890525+0000","last_fullsized":"2024-03-24T12:06:26.998029+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.099551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.099551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":580,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890963+0000","last_change":"2024-03-24T12:05:18.690563+0000","last_active":"2024-03-24T12:07:16.890963+0000","last_peered":"2024-03-24T12:07:16.890963+0000","last_clean":"2024-03-24T12:07:16.890963+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:07:16.890963+0000","last_undegraded":"2024-03-24T12:07:16.890963+0000","last_fullsized":"2024-03-24T12:07:16.890963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:18.690224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:05:18.690224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":618,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890880+0000","last_change":"2024-03-24T12:06:56.609759+0000","last_active":"2024-03-24T12:07:16.890880+0000","last_peered":"2024-03-24T12:07:16.890880+0000","last_clean":"2024-03-24T12:07:16.890880+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:07:16.890880+0000","last_undegraded":"2024-03-24T12:07:16.890880+0000","last_fullsized":"2024-03-24T12:07:16.890880+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:06:56.609621+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:06:56.609621+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:22.054130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":613,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.088350+0000","last_change":"2024-03-24T12:06:08.361300+0000","last_active":"2024-03-24T12:07:16.088350+0000","last_peered":"2024-03-24T12:07:16.088350+0000","last_clean":"2024-03-24T12:07:16.088350+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:07:16.088350+0000","last_undegraded":"2024-03-24T12:07:16.088350+0000","last_fullsized":"2024-03-24T12:07:16.088350+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:08.361148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:06:08.361148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:35.371222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":605,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.087816+0000","last_change":"2024-03-24T12:06:09.338475+0000","last_active":"2024-03-24T12:07:16.087816+0000","last_peered":"2024-03-24T12:07:16.087816+0000","last_clean":"2024-03-24T12:07:16.087816+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:07:16.087816+0000","last_undegraded":"2024-03-24T12:07:16.087816+0000","last_fullsized":"2024-03-24T12:07:16.087816+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:09.338261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T12:06:09.338261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:24.101055+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":39,"reported_epoch":334,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.084245+0000","last_change":"2024-03-24T12:06:27.013806+0000","last_active":"2024-03-24T12:07:16.084245+0000","last_peered":"2024-03-24T12:07:16.084245+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.013806+0000","last_became_peered":"2024-03-24T12:06:27.013806+0000","last_unstale":"2024-03-24T12:07:16.084245+0000","last_undegraded":"2024-03-24T12:07:16.084245+0000","last_fullsized":"2024-03-24T12:06:27.001624+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.117875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.117875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":624,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.891600+0000","last_change":"2024-03-24T12:05:06.830698+0000","last_active":"2024-03-24T12:07:16.891600+0000","last_peered":"2024-03-24T12:07:16.891600+0000","last_clean":"2024-03-24T12:07:16.891600+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:07:16.891600+0000","last_undegraded":"2024-03-24T12:07:16.891600+0000","last_fullsized":"2024-03-24T12:07:16.891600+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:05:06.830559+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.830559+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:18.481286+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":597,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.886533+0000","last_change":"2024-03-24T12:06:27.011373+0000","last_active":"2024-03-24T12:07:16.886533+0000","last_peered":"2024-03-24T12:07:16.886533+0000","last_clean":"2024-03-24T12:06:21.982614+0000","last_became_active":"2024-03-24T12:06:27.011373+0000","last_became_peered":"2024-03-24T12:06:27.011373+0000","last_unstale":"2024-03-24T12:07:16.886533+0000","last_undegraded":"2024-03-24T12:07:16.886533+0000","last_fullsized":"2024-03-24T12:06:26.998012+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:55.731711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:03:55.731711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":74,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.885153+0000","last_change":"2024-03-24T12:06:27.007261+0000","last_active":"2024-03-24T12:07:16.885153+0000","last_peered":"2024-03-24T12:07:16.885153+0000","last_clean":"2024-03-24T12:06:22.261369+0000","last_became_active":"2024-03-24T12:06:27.007261+0000","last_became_peered":"2024-03-24T12:06:27.007261+0000","last_unstale":"2024-03-24T12:07:16.885153+0000","last_undegraded":"2024-03-24T12:07:16.885153+0000","last_fullsized":"2024-03-24T12:06:26.999466+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.683700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.683700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":590,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-03-24T12:07:16.893780+0000","last_change":"2024-03-24T12:06:27.012256+0000","last_active":"2024-03-24T12:07:16.893780+0000","last_peered":"2024-03-24T12:07:16.893780+0000","last_clean":"2024-03-24T12:06:22.261185+0000","last_became_active":"2024-03-24T12:06:27.012256+0000","last_became_peered":"2024-03-24T12:06:27.012256+0000","last_unstale":"2024-03-24T12:07:16.893780+0000","last_undegraded":"2024-03-24T12:07:16.893780+0000","last_fullsized":"2024-03-24T12:06:26.999262+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.241824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.241824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:24.386144+0000","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":664,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:16.886371+0000","last_change":"2024-03-24T12:06:27.007085+0000","last_active":"2024-03-24T12:07:16.886371+0000","last_peered":"2024-03-24T12:07:16.886371+0000","last_clean":"2024-03-24T12:06:21.988852+0000","last_became_active":"2024-03-24T12:06:27.007085+0000","last_became_peered":"2024-03-24T12:06:27.007085+0000","last_unstale":"2024-03-24T12:07:16.886371+0000","last_undegraded":"2024-03-24T12:06:27.000133+0000","last_fullsized":"2024-03-24T12:06:26.999091+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:04:50.623172+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:04:50.623172+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":593,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.890424+0000","last_change":"2024-03-24T12:05:17.651948+0000","last_active":"2024-03-24T12:07:16.890424+0000","last_peered":"2024-03-24T12:07:16.890424+0000","last_clean":"2024-03-24T12:07:16.890424+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:07:16.890424+0000","last_undegraded":"2024-03-24T12:07:16.890424+0000","last_fullsized":"2024-03-24T12:07:16.890424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.651826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.651826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":63,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.883135+0000","last_change":"2024-03-24T12:05:56.863247+0000","last_active":"2024-03-24T12:07:16.883135+0000","last_peered":"2024-03-24T12:07:16.883135+0000","last_clean":"2024-03-24T12:07:16.883135+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:07:16.883135+0000","last_undegraded":"2024-03-24T12:07:16.883135+0000","last_fullsized":"2024-03-24T12:07:16.883135+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:05:12.732801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":580,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-24T12:07:16.884675+0000","last_change":"2024-03-24T12:04:54.805094+0000","last_active":"2024-03-24T12:07:16.884675+0000","last_peered":"2024-03-24T12:07:16.884675+0000","last_clean":"2024-03-24T12:07:16.884675+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:07:16.884675+0000","last_undegraded":"2024-03-24T12:07:16.884675+0000","last_fullsized":"2024-03-24T12:07:16.884675+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.804969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.804969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5118885888,"num_objects":78108,"num_object_clones":0,"num_object_copies":234324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78108,"num_write_kb":4998912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13347913728,"data_stored":13347913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71308,"ondisk_log_size":71308,"up":42,"acting":42,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":488689,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":80,"num_write":49,"num_write_kb":658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":483028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":52,"acting":52,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":98,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":58,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":783,"ondisk_log_size":783,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005240,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1780208,"kb_used_data":1181220,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91964944,"statfs":{"total":95995035648,"available":94172102656,"internally_reserved":0,"allocated":1209569280,"data_stored":1209118599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332288,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2710088,"kb_used_data":2227712,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91035064,"statfs":{"total":95995035648,"available":93219905536,"internally_reserved":0,"allocated":2281177088,"data_stored":2280705055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1510328,"kb_used_data":879404,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92234824,"statfs":{"total":95995035648,"available":94448459776,"internally_reserved":0,"allocated":900509696,"data_stored":900050220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738589,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3607112,"kb_used_data":2990072,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":90138040,"statfs":{"total":95995035648,"available":92301352960,"internally_reserved":0,"allocated":3061833728,"data_stored":3061368967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411539,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1375376,"kb_used_data":893000,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92369776,"statfs":{"total":95995035648,"available":94586650624,"internally_reserved":0,"allocated":914432000,"data_stored":914009262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117197,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2178588,"kb_used_data":1515724,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91566564,"statfs":{"total":95995035648,"available":93764161536,"internally_reserved":0,"allocated":1552101376,"data_stored":1551617560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":678754759},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920969,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3709508,"kb_used_data":3236860,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":90035644,"statfs":{"total":95995035648,"available":92196499456,"internally_reserved":0,"allocated":3314544640,"data_stored":3314075849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659320,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":573604,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93171548,"statfs":{"total":95995035648,"available":95407665152,"internally_reserved":0,"allocated":1007616,"data_stored":637309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3113811968,"data_stored":3113811968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":905838592,"data_stored":905838592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":916652032,"data_stored":916652032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1564606464,"data_stored":1564606464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3315531776,"data_stored":3315531776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2311192576,"data_stored":2311192576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1220280320,"data_stored":1220280320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:07:24.421 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-24T12:07:24.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:24.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:24.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:24.583 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:24.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:24.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:24.719 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:24.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:24.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 45 16 85564 85548 118.793 84.6875 0.00471678 0.00841338 2024-03-24T12:07:24.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:24.861 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:24.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:24.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:24.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:24.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:25.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:25.143 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:25.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:25.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:25.294 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:25.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:25.361 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:25 smithi064 ceph-mon[30223]: pgmap v691: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 30756/245349 objects degraded (12.536%) 2024-03-24T12:07:25.362 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:25 smithi064 ceph-mon[42243]: pgmap v691: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 30756/245349 objects degraded (12.536%) 2024-03-24T12:07:25.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:25.436 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:25.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:25.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:25.547 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:25 smithi193 ceph-mon[33593]: pgmap v691: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 30756/245349 objects degraded (12.536%) 2024-03-24T12:07:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:25.572 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:25.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:25.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:25.709 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:25.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:25.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 46 15 86939 86924 118.08 86 0.00547346 0.00846342 2024-03-24T12:07:25.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:25.853 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:25.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:25.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:25.991 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:25.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:26.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:07:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:26.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:26.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:26.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:26.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:26.410 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:26.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:26.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:26.544 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:26.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:26 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1170852135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:07:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:26 smithi064 ceph-mon[30223]: from='client.25603 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:07:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:26 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1170852135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:07:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:26 smithi064 ceph-mon[42243]: from='client.25603 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:07:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:26 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1170852135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:07:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:26 smithi193 ceph-mon[33593]: from='client.25603 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:07:26.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:26.691 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:26.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:26.743 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 47 16 88389 88373 117.495 90.5625 0.00994461 0.00850623 2024-03-24T12:07:26.793 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:26.825 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:26.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:26.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:26.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:26.961 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:26.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:27.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:27.103 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:27.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:27.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:27.237 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:27.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: pgmap v692: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 30756/245349 objects degraded (12.536%) 2024-03-24T12:07:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:07:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: from='client.25603 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:07:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: osdmap e336: 8 total, 7 up, 8 in 2024-03-24T12:07:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T12:07:27.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 30756/245349 objects degraded (12.536%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[30223]: osdmap e337: 8 total, 7 up, 8 in 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: pgmap v692: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 30756/245349 objects degraded (12.536%) 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: from='client.25603 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: osdmap e336: 8 total, 7 up, 8 in 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 30756/245349 objects degraded (12.536%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:27.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-mon[42243]: osdmap e337: 8 total, 7 up, 8 in 2024-03-24T12:07:27.338 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:27.372 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:27.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:27.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:27.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: pgmap v692: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 30756/245349 objects degraded (12.536%) 2024-03-24T12:07:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:07:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: from='client.25603 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:07:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: osdmap e336: 8 total, 7 up, 8 in 2024-03-24T12:07:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T12:07:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T12:07:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 30756/245349 objects degraded (12.536%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:07:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-mon[33593]: osdmap e337: 8 total, 7 up, 8 in 2024-03-24T12:07:27.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:27.662 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:27.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:27.715 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:07:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:07:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:27.744 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 48 16 89852 89836 116.949 91.4375 0.0221451 0.00854579 2024-03-24T12:07:27.764 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:27.799 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:27.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:27.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:27.935 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:27.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:27.946 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:07:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:07:28.030 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:07:28.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:28.065 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:07:28.066 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:28.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:28.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:07:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:07:28.103 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:07:28.144 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:07:28.168 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:28.183 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:07:28.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:07:28.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:07:28.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:07:28.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:07:28.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:07:28.183 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:07:28.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:07:28.184 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:07:28.212 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:28.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:28.245 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:28 smithi064 ceph-mon[30223]: 1.c scrub starts 2024-03-24T12:07:28.245 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:28 smithi064 ceph-mon[30223]: 1.c scrub ok 2024-03-24T12:07:28.245 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2376123712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:28.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:28 smithi064 ceph-mon[42243]: 1.c scrub starts 2024-03-24T12:07:28.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:28 smithi064 ceph-mon[42243]: 1.c scrub ok 2024-03-24T12:07:28.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:28 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2376123712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:28.314 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:28.382 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:28.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:28.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:28.519 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:28.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:28 smithi193 ceph-mon[33593]: 1.c scrub starts 2024-03-24T12:07:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:28 smithi193 ceph-mon[33593]: 1.c scrub ok 2024-03-24T12:07:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:28 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2376123712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:28.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:28.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:28.744 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 49 16 91238 91222 116.33 86.625 0.0308139 0.00859143 2024-03-24T12:07:28.764 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:28.767 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:07:28.767 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:07:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:28.796 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:28.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:28.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:28.935 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:28.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:29.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:29.086 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:29.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:29.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:29.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:29.224 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:29.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:07:29.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:07:29.326 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:29.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:29.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:29.422 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:07:29.422 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:07:29.422 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-03-24T12:07:29.422 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T12:07:29.422 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T12:07:29.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-03-24T12:07:29.456 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:29 smithi193 ceph-mon[33593]: pgmap v695: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 32397/260865 objects degraded (12.419%) 2024-03-24T12:07:29.456 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:29 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:29.456 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:29 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:07:29.456 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:29 smithi193 ceph-mon[33593]: 2.a scrub starts 2024-03-24T12:07:29.456 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:29 smithi193 ceph-mon[33593]: 2.a scrub ok 2024-03-24T12:07:29.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:29 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:07:29.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:29.508 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:29.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:29.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[30223]: pgmap v695: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 32397/260865 objects degraded (12.419%) 2024-03-24T12:07:29.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:29.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:07:29.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[30223]: 2.a scrub starts 2024-03-24T12:07:29.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[30223]: 2.a scrub ok 2024-03-24T12:07:29.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:07:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[42243]: pgmap v695: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 32397/260865 objects degraded (12.419%) 2024-03-24T12:07:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:07:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[42243]: 2.a scrub starts 2024-03-24T12:07:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[42243]: 2.a scrub ok 2024-03-24T12:07:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:29 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:07:29.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:29.647 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:29.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:29.745 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 50 16 92734 92718 115.874 93.5 0.0155652 0.00862576 2024-03-24T12:07:29.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:29.798 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:29.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:29.881 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:07:29.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:29.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:29.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:30.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:07:30.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:30.091 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:30.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:30.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:30.233 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:30.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:30.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:30 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T12:07:30.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:30 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T12:07:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:30 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T12:07:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:30 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T12:07:30.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:30.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:30 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T12:07:30.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:30 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T12:07:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:30.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:30.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:30.470 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:30.508 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:30.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:30.609 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:30.643 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:30.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:30.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:07:30.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:30.745 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 51 16 94144 94128 115.329 88.125 0.011318 0.00866521 2024-03-24T12:07:30.777 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:30.809 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:30.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:30.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:30.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:31.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:31.102 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:31.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:31.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[42243]: pgmap v696: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s; 33298/266250 objects degraded (12.506%) 2024-03-24T12:07:31.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T12:07:31.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T12:07:31.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T12:07:31.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T12:07:31.179 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[30223]: pgmap v696: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s; 33298/266250 objects degraded (12.506%) 2024-03-24T12:07:31.180 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T12:07:31.180 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T12:07:31.180 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T12:07:31.180 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:31 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T12:07:31.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:31.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:31.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:31.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:07:31.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:31.400 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:31.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:31.423 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:31 smithi193 ceph-mon[33593]: pgmap v696: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s; 33298/266250 objects degraded (12.506%) 2024-03-24T12:07:31.423 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:31 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T12:07:31.423 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:31 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T12:07:31.423 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:31 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T12:07:31.423 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:31 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T12:07:31.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:31.546 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:31.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:31.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:31.694 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:31.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:31.747 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 52 16 95459 95443 114.689 82.1875 0.00441304 0.00871351 2024-03-24T12:07:31.747 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:31.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:31.835 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:31.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:31.884 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:07:31.884 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T12:07:31.884 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:07:31.936 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:31.972 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:31.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:32.074 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:32.106 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:32.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:32.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:32.250 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:32.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:32.351 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:32.354 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:32.382 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:32.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 33298/266250 objects degraded (12.506%), 24 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 33298/266250 objects degraded (12.506%), 24 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T12:07:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:32 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T12:07:32.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:32.557 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:32.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:32.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:32 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T12:07:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:32 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T12:07:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:32 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 33298/266250 objects degraded (12.506%), 24 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:32 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T12:07:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:32 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T12:07:32.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:32.701 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:32.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:32.746 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 53 16 96858 96842 114.175 87.4375 0.00601498 0.00875407 2024-03-24T12:07:32.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:32.841 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:32.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:32.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:32.977 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:32.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:32.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:33.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:33.114 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:33.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:33.216 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:33.248 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:33.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:33.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:33 smithi064 ceph-mon[30223]: pgmap v697: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 34677/277113 objects degraded (12.514%) 2024-03-24T12:07:33.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:33 smithi064 ceph-mon[42243]: pgmap v697: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 34677/277113 objects degraded (12.514%) 2024-03-24T12:07:33.350 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:33.381 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:33.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:33.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:33.523 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:33.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:33 smithi193 ceph-mon[33593]: pgmap v697: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 34677/277113 objects degraded (12.514%) 2024-03-24T12:07:33.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:33.654 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:33.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:33.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:33.746 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 54 16 98262 98246 113.685 87.75 0.0078534 0.00879192 2024-03-24T12:07:33.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:33.798 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:33.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:33.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:33.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:33.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:33.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:34.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:34.095 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:34.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:34.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:34.237 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:34.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:34.339 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:34.373 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:34.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:34.475 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:34.511 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:34.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:34.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[30223]: 1.7 scrub starts 2024-03-24T12:07:34.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[30223]: 1.7 scrub ok 2024-03-24T12:07:34.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[30223]: 10.5 scrub starts 2024-03-24T12:07:34.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T12:07:34.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T12:07:34.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[42243]: 1.7 scrub starts 2024-03-24T12:07:34.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[42243]: 1.7 scrub ok 2024-03-24T12:07:34.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[42243]: 10.5 scrub starts 2024-03-24T12:07:34.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T12:07:34.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:34 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T12:07:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:34 smithi193 ceph-mon[33593]: 1.7 scrub starts 2024-03-24T12:07:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:34 smithi193 ceph-mon[33593]: 1.7 scrub ok 2024-03-24T12:07:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:34 smithi193 ceph-mon[33593]: 10.5 scrub starts 2024-03-24T12:07:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:34 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T12:07:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:34 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T12:07:34.615 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:34.647 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:34.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:34.746 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 55 15 99650 99635 113.196 86.8125 0.00651394 0.00882963 2024-03-24T12:07:34.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:34.784 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:34.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:34.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:34.924 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:34.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:35.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:35.064 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:35.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:35.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:35.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:35.207 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:35.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:35.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:35.350 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:35.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:35.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:35.485 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:35.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[30223]: pgmap v698: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 35930/287631 objects degraded (12.492%) 2024-03-24T12:07:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T12:07:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T12:07:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T12:07:35.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T12:07:35.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[42243]: pgmap v698: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 35930/287631 objects degraded (12.492%) 2024-03-24T12:07:35.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T12:07:35.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T12:07:35.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T12:07:35.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:35 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T12:07:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:35 smithi193 ceph-mon[33593]: pgmap v698: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 35930/287631 objects degraded (12.492%) 2024-03-24T12:07:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:35 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T12:07:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:35 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T12:07:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:35 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T12:07:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:35 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T12:07:35.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:35.628 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:35.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:35.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:35.746 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 56 16 101024 101008 112.707 85.8125 0.00296502 0.00886723 2024-03-24T12:07:35.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:35.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:35.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:35.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:35.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:35.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:35.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:36.037 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:36.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:36.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:36.174 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:36.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:36.275 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:36.311 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:36.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:36.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:36.457 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:36.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:36.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:36.595 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:36.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:36.697 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:36.729 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:36.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:36.747 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 57 16 102501 102485 112.349 92.3125 0.0104698 0.00889631 2024-03-24T12:07:36.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:36.863 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:36.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:36.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:37.022 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:37.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:37.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:37.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:37.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[30223]: pgmap v699: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 35930/287631 objects degraded (12.492%) 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[30223]: 10.5 scrub ok 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[30223]: 5.1e scrub starts 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[30223]: 5.1e scrub ok 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 35930/287631 objects degraded (12.492%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[42243]: pgmap v699: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 35930/287631 objects degraded (12.492%) 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[42243]: 10.5 scrub ok 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T12:07:37.267 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T12:07:37.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[42243]: 5.1e scrub starts 2024-03-24T12:07:37.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[42243]: 5.1e scrub ok 2024-03-24T12:07:37.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 35930/287631 objects degraded (12.492%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:37.279 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:37.314 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:37.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:37.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:37.460 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:37.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:37.523 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:07:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:07:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:37.561 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:37 smithi193 ceph-mon[33593]: pgmap v699: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 35930/287631 objects degraded (12.492%) 2024-03-24T12:07:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:37 smithi193 ceph-mon[33593]: 10.5 scrub ok 2024-03-24T12:07:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:37 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T12:07:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:37 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T12:07:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:37 smithi193 ceph-mon[33593]: 5.1e scrub starts 2024-03-24T12:07:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:37 smithi193 ceph-mon[33593]: 5.1e scrub ok 2024-03-24T12:07:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:37 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 35930/287631 objects degraded (12.492%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:37.593 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:37.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:37.612 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:07:37.613 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:07:37.613 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-24T12:07:37.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:07:37.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:37.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:37.747 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 58 16 103882 103866 111.9 86.3125 0.00508993 0.00893212 2024-03-24T12:07:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:37.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:37.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:37.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:37.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:38.008 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:38.037 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:38.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:38.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:07:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:07:38.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:38.172 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:07:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:38.195 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:38.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:38.195 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:07:38.196 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:07:38.258 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:07:38.296 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:38.308 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:07:38.336 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:38.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:38.347 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:07:38.393 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:07:38.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:38.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:38 smithi064 ceph-mon[30223]: 1.d scrub starts 2024-03-24T12:07:38.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:38 smithi064 ceph-mon[30223]: 1.d scrub ok 2024-03-24T12:07:38.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3310929111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:38.440 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:38 smithi064 ceph-mon[42243]: 1.d scrub starts 2024-03-24T12:07:38.440 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:38 smithi064 ceph-mon[42243]: 1.d scrub ok 2024-03-24T12:07:38.440 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3310929111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:38.440 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:07:38.440 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:07:38.440 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:07:38.440 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:07:38.441 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:07:38.441 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:07:38.441 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:07:38.441 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:07:38.441 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:07:38.480 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:38.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:38.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:38 smithi193 ceph-mon[33593]: 1.d scrub starts 2024-03-24T12:07:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:38 smithi193 ceph-mon[33593]: 1.d scrub ok 2024-03-24T12:07:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3310929111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:38.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:38.622 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:38.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:38.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:38.747 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 59 16 105110 105094 111.304 76.75 0.00270705 0.0089786 2024-03-24T12:07:38.752 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:38.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:38.853 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:38.883 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:38.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:38.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:39.020 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:39.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:39.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:39.159 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:39.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:39.221 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:07:39.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:39.304 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:39.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:39.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:39.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:39.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:39.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[30223]: pgmap v700: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 971 B/s rd, 87 MiB/s wr, 1.39k op/s; 37667/303627 objects degraded (12.406%) 2024-03-24T12:07:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T12:07:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T12:07:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2105518775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[30223]: from='client.25615 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:39.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[42243]: pgmap v700: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 971 B/s rd, 87 MiB/s wr, 1.39k op/s; 37667/303627 objects degraded (12.406%) 2024-03-24T12:07:39.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T12:07:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T12:07:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2105518775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:39.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:39 smithi064 ceph-mon[42243]: from='client.25615 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:39 smithi193 ceph-mon[33593]: pgmap v700: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 971 B/s rd, 87 MiB/s wr, 1.39k op/s; 37667/303627 objects degraded (12.406%) 2024-03-24T12:07:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:39 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:39 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T12:07:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:39 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T12:07:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:39 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2105518775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:39 smithi193 ceph-mon[33593]: from='client.25615 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:07:39.587 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:39.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:39.688 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:39.725 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:39.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:39.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:39.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:07:39.747 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:07:39.749310+0000 min lat: 0.00129496 max lat: 0.125534 avg lat: 0.00900856 2024-03-24T12:07:39.747 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:07:39.747 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 60 16 106538 106522 110.937 89.25 0.00662452 0.00900856 2024-03-24T12:07:39.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:39.871 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:39.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:39.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:40.011 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:40.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:40.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:40.159 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:40.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:40.259 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:40.291 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:40.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:40.303 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:07:40.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:07:40.393 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:40.426 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:40.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:40.528 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:40.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T12:07:40.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T12:07:40.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T12:07:40.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T12:07:40.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: from='client.25615 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: osdmap e338: 8 total, 7 up, 8 in 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[30223]: pgmap v702: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s; 38453/308634 objects degraded (12.459%) 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: from='client.25615 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: osdmap e338: 8 total, 7 up, 8 in 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T12:07:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T12:07:40.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:40 smithi064 ceph-mon[42243]: pgmap v702: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s; 38453/308634 objects degraded (12.459%) 2024-03-24T12:07:40.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:40.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T12:07:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T12:07:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T12:07:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T12:07:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:07:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: from='client.25615 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:07:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: osdmap e338: 8 total, 7 up, 8 in 2024-03-24T12:07:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T12:07:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T12:07:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:40 smithi193 ceph-mon[33593]: pgmap v702: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s; 38453/308634 objects degraded (12.459%) 2024-03-24T12:07:40.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:40.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:40.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:40.747 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 61 16 107892 107876 110.505 84.625 0.00565102 0.00904469 2024-03-24T12:07:40.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:40.846 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:40.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:40.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:41.024 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:41.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:41.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:41.229 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:41.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:41.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:41 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T12:07:41.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:41 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T12:07:41.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:41 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T12:07:41.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:41 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T12:07:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:41 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T12:07:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:41 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T12:07:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:41 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T12:07:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:41 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T12:07:41.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:41.367 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:41.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:41.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:41.508 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:41.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:41 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T12:07:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:41 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T12:07:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:41 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T12:07:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:41 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T12:07:41.610 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:41.642 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:41.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:41.744 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:41.748 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 62 16 109265 109249 110.107 85.8125 0.0226439 0.00907763 2024-03-24T12:07:41.778 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:41.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:41.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:41.924 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:41.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:42.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:42.064 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:42.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:42.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:42.204 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:42.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:42.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:07:42.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:42.349 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:42.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:42.450 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:42.483 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:42.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: 4.1d scrub starts 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: 4.d scrub starts 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: 4.1d scrub ok 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: 4.d scrub ok 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: pgmap v703: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 39719/318741 objects degraded (12.461%) 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 38453/308634 objects degraded (12.459%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: osdmap e339: 8 total, 7 up, 8 in 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T12:07:42.487 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: 4.1d scrub starts 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: 4.d scrub starts 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: 4.1d scrub ok 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: 4.d scrub ok 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: pgmap v703: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 39719/318741 objects degraded (12.461%) 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 38453/308634 objects degraded (12.459%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: osdmap e339: 8 total, 7 up, 8 in 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:42.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:42.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T12:07:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T12:07:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: 4.1d scrub starts 2024-03-24T12:07:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: 4.d scrub starts 2024-03-24T12:07:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: 4.1d scrub ok 2024-03-24T12:07:42.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: 4.d scrub ok 2024-03-24T12:07:42.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: pgmap v703: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 39719/318741 objects degraded (12.461%) 2024-03-24T12:07:42.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 38453/308634 objects degraded (12.459%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:42.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: osdmap e339: 8 total, 7 up, 8 in 2024-03-24T12:07:42.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:07:42.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:42.623 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:42.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:42.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:42.754 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 63 16 110484 110468 109.557 76.1875 0.00952731 0.00912341 2024-03-24T12:07:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:42.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:42.863 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:42.889 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:42.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:42.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:43.029 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:43.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:43.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:07:43.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:43.172 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:43.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:43.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:43.273 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:43.306 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:43.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:43.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:43 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T12:07:43.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:43 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T12:07:43.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:43 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T12:07:43.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:43 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T12:07:43.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:43.457 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:43.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:43.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:43.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:43 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T12:07:43.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:43 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T12:07:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:43.593 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:43.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:43.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:43.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:43.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:43.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:43.754 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 64 16 111861 111845 109.19 86.0625 0.0161912 0.00915405 2024-03-24T12:07:43.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:43.865 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:43.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:43.944 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:43.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:44.044 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:44.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:44.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:44.187 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:44.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:44.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:44.324 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:44.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:44.356 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:44 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T12:07:44.356 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:44 smithi064 ceph-mon[30223]: pgmap v705: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 40942/328893 objects degraded (12.448%) 2024-03-24T12:07:44.356 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:44 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T12:07:44.356 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:44 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T12:07:44.357 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:44 smithi064 ceph-mon[42243]: pgmap v705: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 40942/328893 objects degraded (12.448%) 2024-03-24T12:07:44.357 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:44 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T12:07:44.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:44.465 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:44.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:44.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:44 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T12:07:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:44 smithi193 ceph-mon[33593]: pgmap v705: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 40942/328893 objects degraded (12.448%) 2024-03-24T12:07:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:44 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T12:07:44.602 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:44.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:44.704 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:44.740 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:44.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:44.754 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 65 16 113198 113182 108.796 83.5625 0.0117669 0.00918683 2024-03-24T12:07:44.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:44.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:44.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:44.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:45.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:45.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:45.156 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:45.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:45.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:45.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:45.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:45.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:45.440 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:45.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:45.515 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:45 smithi064 ceph-mon[42243]: 10.0 scrub starts 2024-03-24T12:07:45.515 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:45 smithi064 ceph-mon[42243]: 1.14 scrub starts 2024-03-24T12:07:45.515 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:45 smithi064 ceph-mon[42243]: 1.14 scrub ok 2024-03-24T12:07:45.515 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:45 smithi064 ceph-mon[30223]: 10.0 scrub starts 2024-03-24T12:07:45.515 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:45 smithi064 ceph-mon[30223]: 1.14 scrub starts 2024-03-24T12:07:45.515 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:45 smithi064 ceph-mon[30223]: 1.14 scrub ok 2024-03-24T12:07:45.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:45.577 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:45.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:45.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:45 smithi193 ceph-mon[33593]: 10.0 scrub starts 2024-03-24T12:07:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:45 smithi193 ceph-mon[33593]: 1.14 scrub starts 2024-03-24T12:07:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:45 smithi193 ceph-mon[33593]: 1.14 scrub ok 2024-03-24T12:07:45.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:45.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:45.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:45.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:45.754 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 66 16 114516 114500 108.396 82.375 0.0300355 0.00922093 2024-03-24T12:07:45.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:45.858 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:45.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:45.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:46.005 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:46.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:46.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:46.155 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:46.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:46.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:46.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:46.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:46.402 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:46.437 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:46.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:46.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: 10.d scrub starts 2024-03-24T12:07:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:07:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:07:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T12:07:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T12:07:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: pgmap v706: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s; 40942/328893 objects degraded (12.448%) 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[30223]: 10.0 scrub ok 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: 10.d scrub starts 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: pgmap v706: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s; 40942/328893 objects degraded (12.448%) 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T12:07:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:46 smithi064 ceph-mon[42243]: 10.0 scrub ok 2024-03-24T12:07:46.578 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:46.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: 10.d scrub starts 2024-03-24T12:07:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:07:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:07:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T12:07:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T12:07:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T12:07:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: pgmap v706: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s; 40942/328893 objects degraded (12.448%) 2024-03-24T12:07:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T12:07:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:46 smithi193 ceph-mon[33593]: 10.0 scrub ok 2024-03-24T12:07:46.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:46.715 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:46.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:46.754 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 67 15 115896 115881 108.066 86.3125 0.00416654 0.00924891 2024-03-24T12:07:46.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:46.852 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:46.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:46.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:46.998 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:46.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:47.100 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:47.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:47.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:47.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:47.270 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:47.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:47.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:47.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:47.418 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:47.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:47.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:47.543 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:47 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T12:07:47.543 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:47 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T12:07:47.543 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:47 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 40942/328893 objects degraded (12.448%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:47.543 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:07:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:07:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:47.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:47 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T12:07:47.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:47 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T12:07:47.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:47 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 40942/328893 objects degraded (12.448%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:47.566 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:47.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:47 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T12:07:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:47 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T12:07:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:47 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 40942/328893 objects degraded (12.448%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:47.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:47.670 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:47.674 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:47.692 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:47.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:47.754 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 68 16 117102 117086 107.584 75.3125 0.0226241 0.0092892 2024-03-24T12:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:117105 10.f e339/320 10:fa75f185:::benchmark_data_smithi064_174908_object117104:head v 339'7300, mlcod=339'7298)", 2024-03-24T12:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.761744+0000", 2024-03-24T12:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0065589050000000003, 2024-03-24T12:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065913170000000002, 2024-03-24T12:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761744+0000", 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761744+0000", 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761747+0000", 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4210000000000002e-06 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761792+0000", 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5330000000000001e-05 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761813+0000", 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1387999999999999e-05 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761824+0000", 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0812000000000001e-05 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.762118+0000", 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029339699999999998 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.762136+0000", 2024-03-24T12:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8246000000000001e-05 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:117107 10.8 e339/320 10:13fd4f3f:::benchmark_data_smithi064_174908_object117106:head v 339'7264, mlcod=339'7262)", 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.761858+0000", 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0064455509999999999, 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065505520000000003, 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761858+0000", 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761858+0000", 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761859+0000", 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.206e-06 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761890+0000", 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0951999999999998e-05 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761898+0000", 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9799999999999998e-06 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761904+0000", 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8579999999999996e-06 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761976+0000", 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2168000000000004e-05 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761998+0000", 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2070999999999999e-05 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:117110 10.5 10:a1255d15:::benchmark_data_smithi064_174908_object117109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.761894+0000", 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.006409634, 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065824619999999999, 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117110 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761894+0000", 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761894+0000", 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761896+0000", 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3760000000000002e-06 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761923+0000", 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7013e-05 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761938+0000", 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4523e-05 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.761947+0000", 2024-03-24T12:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7459999999999995e-06 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.762015+0000", 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7935999999999995e-05 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.762176+0000", 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016059000000000001 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.762325+0000", 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014967499999999999 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 3 2024-03-24T12:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:07:47.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:47.844 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:47.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:47.891 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:07:47.894 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:07:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 339, 2024-03-24T12:07:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T12:07:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-03-24T11:46:40.657065+0000", 2024-03-24T12:07:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-03-24T12:07:41.886036+0000", 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-03-24T12:06:25.982459+0000", 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-03-24T11:51:43.667453+0000", 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5799936, 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 18, 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:07:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 10, 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:49:56.369674+0000", 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:07:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 21, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 21, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 21, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 21, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 21, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:07:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "260", 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "258", 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:07:47.898 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:07:47.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.900 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.8999999761581421, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.8999999761581421, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.90672111511230469, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".rgw.root", 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:13.871974+0000", 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:07:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "68", 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:07:47.903 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:07:47.904 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:07:47.905 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.7400000095367432, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.7400000095367432, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.90672111511230469, 2024-03-24T12:07:47.906 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "datapool", 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:14.117018+0000", 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:07:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 3, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 3, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 3, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 3, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 3, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:07:47.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "71", 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 2, 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 71, 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:07:47.909 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.910 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.911 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.6500000953674316, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.6500000953674316, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.62999999523162842, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.90672111511230469, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 4, 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:15.453744+0000", 2024-03-24T12:07:47.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:07:47.913 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "70", 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:07:47.914 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:07:47.915 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:07:47.916 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.2400000095367432, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.2400000095367432, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.90672111511230469, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 5, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:17.611004+0000", 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:07:47.918 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "72", 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:07:47.919 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:07:47.920 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:07:47.921 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.9900000095367432, 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.9900000095367432, 2024-03-24T12:07:47.922 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.90672111511230469, 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 6, 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:07:47.923 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:19.701416+0000", 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:07:47.924 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "74", 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:07:47.925 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:07:47.926 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:07:47.927 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_bias": 4 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.2400000095367432, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.2400000095367432, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.90672111511230469, 2024-03-24T12:07:47.928 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 10, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "unique_pool_3", 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T12:06:28.568197+0000", 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:07:47.929 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 16, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 16, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 16, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 16, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 16, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:07:47.930 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "323", 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:07:47.931 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.932 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "rados": {} 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:07:47.933 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.9900000095367432, 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.9900000095367432, 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.630000114440918, 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.630000114440918, 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.90672111511230469, 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d", 2024-03-24T12:07:47.934 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 8, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 320, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-03-24T12:07:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.936 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.937 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/2675992571", 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6802/2675992571", 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6804/2675992571", 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6803/2675992571", 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:07:47.938 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9", 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.2537689208984375, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 320, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:07:47.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.941 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6805/841667271", 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6806/841667271", 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6808/841667271", 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6807/841667271", 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b", 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.3165130615234375, 2024-03-24T12:07:47.942 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 18, 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 320, 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.943 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/435147390", 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6810/435147390", 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6812/435147390", 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6811/435147390", 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:07:47.945 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479", 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 27, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 320, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:07:47.946 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-03-24T12:07:47.947 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.948 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6813/3355318038", 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6814/3355318038", 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6816/3355318038", 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6815/3355318038", 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b", 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 33, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 320, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6800", 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6801", 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6803", 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6802", 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:07:47.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6800/3851339176", 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6801/3851339176", 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3851339176", 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3851339176", 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6", 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 0, 2024-03-24T12:07:47.953 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 41, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 314, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 318, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6804", 2024-03-24T12:07:47.954 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1625306337 2024-03-24T12:07:47.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:47.955 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6805", 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1625306337 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.956 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6807", 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1625306337 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6806", 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1625306337 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6804/1625306337", 2024-03-24T12:07:47.957 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6805/1625306337", 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6807/1625306337", 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6806/1625306337", 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "exists" 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996", 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:07:47.958 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 51, 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 320, 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6808", 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:07:47.959 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6809", 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6811", 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6810", 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6808/2408237278", 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6809/2408237278", 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6811/2408237278", 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6810/2408237278", 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:07:47.961 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015", 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 61, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 320, 2024-03-24T12:07:47.962 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6812", 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6813", 2024-03-24T12:07:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6815", 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6814", 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6812/2649989609", 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6813/2649989609", 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6815/2649989609", 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6814/2649989609", 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:07:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:07:06.034256+0000", 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:07:47.966 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:07:03.859914+0000", 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:07:15.648859+0000", 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:06:54.746946+0000", 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:07:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:06:55.911060+0000", 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:05:51.732984+0000", 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 317 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:07:47.969 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:07:13.421605+0000", 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:07:10.371035+0000", 2024-03-24T12:07:47.970 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.0", 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [ 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.b", 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:07:47.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.4", 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 5 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.9", 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.1d", 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.6", 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.13", 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:07:47.974 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.18", 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 4, 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.19", 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:07:47.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 6 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": { 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/3538503390": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1277626303": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3407451558": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3753895773": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/842059688": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:07:47.976 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1610565583": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/2730170969": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3257265142": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1459540658": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/629947599": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/520335082": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3351953651": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/4027721305": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/4028269665": "2024-03-25T11:47:36.035658+0000" 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-03-24T12:07:47.977 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:07:47.978 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:07:48.018 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:48.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:48.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:48.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:07:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:07:48.120 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:48.153 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:48.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:48.254 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:48.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:48.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:48.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:48.423 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:48.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:48.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[30223]: 10.d scrub ok 2024-03-24T12:07:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[30223]: 6.b scrub starts 2024-03-24T12:07:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[30223]: 6.b scrub ok 2024-03-24T12:07:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[30223]: pgmap v707: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 42522/343875 objects degraded (12.366%) 2024-03-24T12:07:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:48.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1564032516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:07:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[42243]: 10.d scrub ok 2024-03-24T12:07:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[42243]: 6.b scrub starts 2024-03-24T12:07:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[42243]: 6.b scrub ok 2024-03-24T12:07:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[42243]: pgmap v707: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 42522/343875 objects degraded (12.366%) 2024-03-24T12:07:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:48 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1564032516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:07:48.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:48.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:48 smithi193 ceph-mon[33593]: 10.d scrub ok 2024-03-24T12:07:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:48 smithi193 ceph-mon[33593]: 6.b scrub starts 2024-03-24T12:07:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:48 smithi193 ceph-mon[33593]: 6.b scrub ok 2024-03-24T12:07:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:48 smithi193 ceph-mon[33593]: pgmap v707: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 42522/343875 objects degraded (12.366%) 2024-03-24T12:07:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:48 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:48.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:48 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1564032516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:07:48.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:48.717 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:48.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:48.754 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 69 16 118436 118420 107.233 83.375 0.010719 0.00932122 2024-03-24T12:07:48.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:48.853 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:48.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:48.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:48.989 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:48.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:49.091 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:49.125 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:49.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:49.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:49.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:49.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:49.378 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:49 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T12:07:49.378 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:49 smithi064 ceph-mon[30223]: 5.c deep-scrub starts 2024-03-24T12:07:49.378 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:49 smithi064 ceph-mon[30223]: 5.c deep-scrub ok 2024-03-24T12:07:49.378 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:49 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T12:07:49.379 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:49 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T12:07:49.379 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:49 smithi064 ceph-mon[42243]: 5.c deep-scrub starts 2024-03-24T12:07:49.379 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:49 smithi064 ceph-mon[42243]: 5.c deep-scrub ok 2024-03-24T12:07:49.379 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:49 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T12:07:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:49.394 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:49.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:49.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:49.533 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:49.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:49 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T12:07:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:49 smithi193 ceph-mon[33593]: 5.c deep-scrub starts 2024-03-24T12:07:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:49 smithi193 ceph-mon[33593]: 5.c deep-scrub ok 2024-03-24T12:07:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:49 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T12:07:49.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:49.671 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:49.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:49.754 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 70 16 119797 119781 106.916 85.0625 0.00909828 0.00934874 2024-03-24T12:07:49.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:49.811 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:49.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:49.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:49.950 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:49.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:50.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:50.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:50.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:50.229 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:50.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:50.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:50.365 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:50.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:50.467 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:50.515 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:50.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:50.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:50 smithi064 ceph-mon[30223]: 5.a deep-scrub starts 2024-03-24T12:07:50.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:50 smithi064 ceph-mon[30223]: 5.a deep-scrub ok 2024-03-24T12:07:50.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:50 smithi064 ceph-mon[30223]: pgmap v708: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s; 43328/348732 objects degraded (12.424%) 2024-03-24T12:07:50.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:50 smithi064 ceph-mon[42243]: 5.a deep-scrub starts 2024-03-24T12:07:50.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:50 smithi064 ceph-mon[42243]: 5.a deep-scrub ok 2024-03-24T12:07:50.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:50 smithi064 ceph-mon[42243]: pgmap v708: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s; 43328/348732 objects degraded (12.424%) 2024-03-24T12:07:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:50 smithi193 ceph-mon[33593]: 5.a deep-scrub starts 2024-03-24T12:07:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:50 smithi193 ceph-mon[33593]: 5.a deep-scrub ok 2024-03-24T12:07:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:50 smithi193 ceph-mon[33593]: pgmap v708: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s; 43328/348732 objects degraded (12.424%) 2024-03-24T12:07:50.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:50.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:50.755 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 71 16 121206 121190 106.651 88.0625 0.00578684 0.00937204 2024-03-24T12:07:50.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:50.801 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:50.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:50.903 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:50.938 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:50.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:51.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:51.076 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:51.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:51.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:51.215 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:51.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:51.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:51.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:51.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:07:51.458 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:51.491 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:51.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:51 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T12:07:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:51 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T12:07:51.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:51 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T12:07:51.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:51 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T12:07:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:51 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T12:07:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:51 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T12:07:51.594 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:51.625 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:51.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:51.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:51.756 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 72 16 122594 122578 106.373 86.75 0.00529751 0.00939636 2024-03-24T12:07:51.762 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:51.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:51.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:51.894 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:51.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:51.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:52.040 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:52.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:52.141 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:52.176 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:52.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:52.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:52.305 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:52.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:52.407 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:52.440 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:52.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:52.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:52 smithi193 ceph-mon[33593]: 4.1e scrub starts 2024-03-24T12:07:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:52 smithi193 ceph-mon[33593]: 4.1e scrub ok 2024-03-24T12:07:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:52 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T12:07:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:52 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T12:07:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:52 smithi193 ceph-mon[33593]: pgmap v709: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 68 MiB/s wr, 1.08k op/s; 43746/351210 objects degraded (12.456%) 2024-03-24T12:07:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:52 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 43328/348732 objects degraded (12.424%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:52.640 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:52.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:52.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:52.666 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:07:52.666 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[42243]: 4.1e scrub starts 2024-03-24T12:07:52.666 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[42243]: 4.1e scrub ok 2024-03-24T12:07:52.666 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T12:07:52.666 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T12:07:52.666 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[42243]: pgmap v709: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 68 MiB/s wr, 1.08k op/s; 43746/351210 objects degraded (12.456%) 2024-03-24T12:07:52.666 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 43328/348732 objects degraded (12.424%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:52.667 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[30223]: 4.1e scrub starts 2024-03-24T12:07:52.667 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[30223]: 4.1e scrub ok 2024-03-24T12:07:52.667 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T12:07:52.667 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T12:07:52.667 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[30223]: pgmap v709: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 68 MiB/s wr, 1.08k op/s; 43746/351210 objects degraded (12.456%) 2024-03-24T12:07:52.667 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:52 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 43328/348732 objects degraded (12.424%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:52.710 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:07:52.742 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:52.750 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:07:52.756 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 73 16 123967 123951 106.091 85.8125 0.0184033 0.00942141 2024-03-24T12:07:52.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:52.775 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:52.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:07:52.823 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:07:52.865 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:07:52.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:52.905 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:07:52.905 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:07:52.905 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:07:52.905 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:07:52.905 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:07:52.905 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:07:52.905 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:07:52.906 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:07:52.906 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:07:52.916 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:52.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:53.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:53.059 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:53.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:53.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:53.197 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:53.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:53.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:53.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:53.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:53.480 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:53.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:53.581 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:53 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T12:07:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:53 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T12:07:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:53 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T12:07:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:53 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T12:07:53.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/866247336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:53.618 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:53.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:53.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:53.756 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 74 16 125371 125355 105.843 87.75 0.00629134 0.00944351 2024-03-24T12:07:53.761 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:53.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:53.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T12:07:53.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T12:07:53.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T12:07:53.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T12:07:53.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/866247336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:53.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T12:07:53.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T12:07:53.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T12:07:53.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T12:07:53.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/866247336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:07:53.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:53.901 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:53.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:54.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:54.042 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:54.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:54.144 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:54.180 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:54.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:54.282 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:54.317 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:54.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:54.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:54.458 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:54.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:54.493 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:54 smithi193 ceph-mon[33593]: 5.1d deep-scrub starts 2024-03-24T12:07:54.493 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:54 smithi193 ceph-mon[33593]: 5.1d deep-scrub ok 2024-03-24T12:07:54.493 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:54 smithi193 ceph-mon[33593]: pgmap v710: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 886 B/s rd, 84 MiB/s wr, 1.34k op/s; 45804/369117 objects degraded (12.409%) 2024-03-24T12:07:54.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:54.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:54.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:54.742 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:54.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:54.756 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 75 16 126789 126773 105.613 88.625 0.0122232 0.00946411 2024-03-24T12:07:54.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:54 smithi064 ceph-mon[30223]: 5.1d deep-scrub starts 2024-03-24T12:07:54.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:54 smithi064 ceph-mon[30223]: 5.1d deep-scrub ok 2024-03-24T12:07:54.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:54 smithi064 ceph-mon[30223]: pgmap v710: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 886 B/s rd, 84 MiB/s wr, 1.34k op/s; 45804/369117 objects degraded (12.409%) 2024-03-24T12:07:54.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:54 smithi064 ceph-mon[42243]: 5.1d deep-scrub starts 2024-03-24T12:07:54.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:54 smithi064 ceph-mon[42243]: 5.1d deep-scrub ok 2024-03-24T12:07:54.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:54 smithi064 ceph-mon[42243]: pgmap v710: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 886 B/s rd, 84 MiB/s wr, 1.34k op/s; 45804/369117 objects degraded (12.409%) 2024-03-24T12:07:54.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:54.880 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:54.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:54.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:55.006 INFO:tasks.thrashosds.thrasher:j is {'epoch': 339, 'fsid': '06f23456-e9d4-11ee-95cd-87774f69a715', 'created': '2024-03-24T11:46:40.657065+0000', 'modified': '2024-03-24T12:07:41.886036+0000', 'last_up_change': '2024-03-24T12:06:25.982459+0000', 'last_in_change': '2024-03-24T11:51:43.667453+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:49:56.369674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '260', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '258', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9067211151123047, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T11:52:13.871974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9067211151123047, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-24T11:52:14.117018+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9067211151123047, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T11:52:15.453744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9067211151123047, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T11:52:17.611004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9067211151123047, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T11:52:19.701416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9067211151123047, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-24T12:06:28.568197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '323', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.9067211151123047, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}], 'osds': [{'osd': 0, 'uuid': 'bc14a64f-9e6c-41f1-901e-c49fabffb53d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 320, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 2675992571}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6802', 'nonce': 2675992571}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6804', 'nonce': 2675992571}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 2675992571}]}, 'public_addr': '172.21.15.64:6801/2675992571', 'cluster_addr': '172.21.15.64:6802/2675992571', 'heartbeat_back_addr': '172.21.15.64:6804/2675992571', 'heartbeat_front_addr': '172.21.15.64:6803/2675992571', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '74587332-f0b2-4905-9888-7bdb604cc0e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2537689208984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 320, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 841667271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6806', 'nonce': 841667271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6808', 'nonce': 841667271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 841667271}]}, 'public_addr': '172.21.15.64:6805/841667271', 'cluster_addr': '172.21.15.64:6806/841667271', 'heartbeat_back_addr': '172.21.15.64:6808/841667271', 'heartbeat_front_addr': '172.21.15.64:6807/841667271', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a0c0ed85-9f33-4198-a1f2-447eb17c364b', 'up': 1, 'in': 1, 'weight': 0.3165130615234375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 320, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 435147390}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6810', 'nonce': 435147390}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6812', 'nonce': 435147390}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 435147390}]}, 'public_addr': '172.21.15.64:6809/435147390', 'cluster_addr': '172.21.15.64:6810/435147390', 'heartbeat_back_addr': '172.21.15.64:6812/435147390', 'heartbeat_front_addr': '172.21.15.64:6811/435147390', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1396f66c-3e0c-4a01-b0a7-6f763989a479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 320, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3355318038}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6814', 'nonce': 3355318038}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6816', 'nonce': 3355318038}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3355318038}]}, 'public_addr': '172.21.15.64:6813/3355318038', 'cluster_addr': '172.21.15.64:6814/3355318038', 'heartbeat_back_addr': '172.21.15.64:6816/3355318038', 'heartbeat_front_addr': '172.21.15.64:6815/3355318038', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50d65d8b-3e90-46a4-b19d-50c3015af40b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 320, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3851339176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3851339176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3851339176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3851339176}]}, 'public_addr': '172.21.15.193:6800/3851339176', 'cluster_addr': '172.21.15.193:6801/3851339176', 'heartbeat_back_addr': '172.21.15.193:6803/3851339176', 'heartbeat_front_addr': '172.21.15.193:6802/3851339176', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '50c5d50e-7e68-487c-9f68-1abb8c927ff6', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 314, 'down_at': 318, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 1625306337}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 1625306337}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 1625306337}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 1625306337}]}, 'public_addr': '172.21.15.193:6804/1625306337', 'cluster_addr': '172.21.15.193:6805/1625306337', 'heartbeat_back_addr': '172.21.15.193:6807/1625306337', 'heartbeat_front_addr': '172.21.15.193:6806/1625306337', 'state': ['exists']}, {'osd': 6, 'uuid': '7e74fe39-5707-4c90-ad38-1083cd7a6996', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 320, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 2408237278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 2408237278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 2408237278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 2408237278}]}, 'public_addr': '172.21.15.193:6808/2408237278', 'cluster_addr': '172.21.15.193:6809/2408237278', 'heartbeat_back_addr': '172.21.15.193:6811/2408237278', 'heartbeat_front_addr': '172.21.15.193:6810/2408237278', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e568df64-cc9a-43f8-a432-8628d1da1015', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 320, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 2649989609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 2649989609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 2649989609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 2649989609}]}, 'public_addr': '172.21.15.193:6812/2649989609', 'cluster_addr': '172.21.15.193:6813/2649989609', 'heartbeat_back_addr': '172.21.15.193:6815/2649989609', 'heartbeat_front_addr': '172.21.15.193:6814/2649989609', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:07:06.034256+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:07:03.859914+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:07:15.648859+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:06:54.746946+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:06:55.911060+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:05:51.732984+0000', 'dead_epoch': 317}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:07:13.421605+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:07:10.371035+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [0, 1, 7]}], 'pg_upmap_items': [{'pgid': '2.b', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.6', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.64:6800/3538503390': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/1277626303': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3407451558': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3753895773': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/842059688': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/1610565583': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/2730170969': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3257265142': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/1459540658': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/629947599': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:6800/520335082': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3351953651': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:6800/4027721305': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/4028269665': '2024-03-25T11:47:36.035658+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:07:55.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:07:55.020 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:55.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:55.123 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:55.151 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:55.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:55.253 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:55.287 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:55.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:55.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:55.423 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:55.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:55.524 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:07:55.526 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:07:55.555 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:55.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:55.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:55.691 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:55.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:55.756 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 76 16 128066 128050 105.274 79.8125 0.0100359 0.00949494 2024-03-24T12:07:55.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:55.830 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:55.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:55.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:55.959 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:55.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:56.061 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:56.087 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:56.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:56.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:56.228 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:56.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:56.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:56.367 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:56.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:56.468 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:56.501 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:56.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:56.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:56.614 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:56.670 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:56.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:56.756 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 77 16 129318 129302 104.923 78.25 0.00566574 0.00952646 2024-03-24T12:07:56.766 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:07:56.766 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:07:56.766 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[42243]: pgmap v711: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 45804/369117 objects degraded (12.409%) 2024-03-24T12:07:56.766 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T12:07:56.767 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[42243]: 4.c scrub starts 2024-03-24T12:07:56.767 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[42243]: 4.c scrub ok 2024-03-24T12:07:56.767 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:07:56.767 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:07:56.767 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[30223]: pgmap v711: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 45804/369117 objects degraded (12.409%) 2024-03-24T12:07:56.767 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T12:07:56.767 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[30223]: 4.c scrub starts 2024-03-24T12:07:56.767 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:56 smithi064 ceph-mon[30223]: 4.c scrub ok 2024-03-24T12:07:56.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:56.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:56.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:56 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:07:56.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:56 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:07:56.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:56 smithi193 ceph-mon[33593]: pgmap v711: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 45804/369117 objects degraded (12.409%) 2024-03-24T12:07:56.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:56 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T12:07:56.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:56 smithi193 ceph-mon[33593]: 4.c scrub starts 2024-03-24T12:07:56.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:56 smithi193 ceph-mon[33593]: 4.c scrub ok 2024-03-24T12:07:56.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:56.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:57.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:57.095 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:57.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:57.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:57.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:57.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:57.284 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:57.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:57.373 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:57.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:57.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:57.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:57.513 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:57.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:57.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:07:57.614 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:57.648 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:57.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:57.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-mon[30223]: 10.f scrub starts 2024-03-24T12:07:57.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T12:07:57.720 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 45804/369117 objects degraded (12.409%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:57.720 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:57.720 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:07:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:57.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-mon[42243]: 10.f scrub starts 2024-03-24T12:07:57.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T12:07:57.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 45804/369117 objects degraded (12.409%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:57.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:57.750 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:57.757 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 78 16 130541 130525 104.557 76.4375 0.0193334 0.00955932 2024-03-24T12:07:57.789 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:57.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:57.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:07:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:07:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:57 smithi193 ceph-mon[33593]: 10.f scrub starts 2024-03-24T12:07:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:57 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T12:07:57.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:57 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 45804/369117 objects degraded (12.409%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:07:57.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:57.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:57.926 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:57.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:58.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:58.062 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:58.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:58.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:58.216 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:58.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:58.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:58.352 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:58.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:58.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:07:58.494 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:58.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:07:58.555 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:07:58.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:58.626 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:58.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:58.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:58.757 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 79 16 131784 131768 104.217 77.6875 0.00949435 0.00958913 2024-03-24T12:07:58.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[30223]: pgmap v712: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 47749/385065 objects degraded (12.400%) 2024-03-24T12:07:58.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:58.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3821296893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:07:58.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[30223]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:07:58.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[30223]: 10.2 deep-scrub starts 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[30223]: 1.e scrub starts 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[30223]: 1.e scrub ok 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[42243]: pgmap v712: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 47749/385065 objects degraded (12.400%) 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3821296893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[42243]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[42243]: 10.2 deep-scrub starts 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[42243]: 1.e scrub starts 2024-03-24T12:07:58.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:58 smithi064 ceph-mon[42243]: 1.e scrub ok 2024-03-24T12:07:58.763 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:58.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:58.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:58.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:58 smithi193 ceph-mon[33593]: pgmap v712: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 47749/385065 objects degraded (12.400%) 2024-03-24T12:07:58.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:58 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:58.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:58 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3821296893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:07:58.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:58 smithi193 ceph-mon[33593]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:07:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:58 smithi193 ceph-mon[33593]: 10.2 deep-scrub starts 2024-03-24T12:07:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:58 smithi193 ceph-mon[33593]: 1.e scrub starts 2024-03-24T12:07:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:58 smithi193 ceph-mon[33593]: 1.e scrub ok 2024-03-24T12:07:58.865 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:07:58.896 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:58.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:07:58.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:59.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:59.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:59.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:59.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:07:59.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:59.314 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:59.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:07:59.341 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:07:59.341 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:07:59.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:07:59.446 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:59.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:07:59.548 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:07:59.582 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:07:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:59.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:07:59.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:07:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:59.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:07:59.757 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:07:59.758875+0000 min lat: 0.00129496 max lat: 0.147148 avg lat: 0.00962361 2024-03-24T12:07:59.757 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:07:59.757 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 80 16 132995 132979 103.861 75.6875 0.0045628 0.00962361 2024-03-24T12:07:59.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:07:59.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:59 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:07:59.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:59 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:07:59.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:59 smithi193 ceph-mon[33593]: 10.f scrub ok 2024-03-24T12:07:59.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:59 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T12:07:59.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:59 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T12:07:59.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:59 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:07:59.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:59 smithi193 ceph-mon[33593]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:07:59.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:07:59 smithi193 ceph-mon[33593]: osdmap e340: 8 total, 7 up, 8 in 2024-03-24T12:07:59.871 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:07:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:07:59.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:07:59.936 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:07:59.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:00.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:00.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:08:00.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:08:00.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[30223]: 10.f scrub ok 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[30223]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[30223]: osdmap e340: 8 total, 7 up, 8 in 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[42243]: 10.f scrub ok 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:00.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[42243]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:08:00.053 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:07:59 smithi064 ceph-mon[42243]: osdmap e340: 8 total, 7 up, 8 in 2024-03-24T12:08:00.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:00.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:00.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:00.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:00.289 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:00.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:00.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:00.422 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:00.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:00.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:00.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:00.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:00.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:00.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:00.757 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 81 15 134203 134188 103.511 75.5625 0.0223852 0.00965566 2024-03-24T12:08:00.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:00 smithi064 ceph-mon[30223]: pgmap v714: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s; 48173/388785 objects degraded (12.391%) 2024-03-24T12:08:00.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:00 smithi064 ceph-mon[42243]: pgmap v714: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s; 48173/388785 objects degraded (12.391%) 2024-03-24T12:08:00.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:00.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:00 smithi193 ceph-mon[33593]: pgmap v714: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s; 48173/388785 objects degraded (12.391%) 2024-03-24T12:08:00.839 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:00.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:00.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:00.977 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:00.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:01.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:08:01.078 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:01.108 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:01.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:01.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:01.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:01.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:01.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:01.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:01.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:01.551 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:01.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:01.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:01.653 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:01.683 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:01.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:01.757 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 82 16 135361 135345 103.131 72.3125 0.00793896 0.00969188 2024-03-24T12:08:01.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:01.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:01.925 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:01.954 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:01.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:02.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:02.095 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:02.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:02.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:02.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:02.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:02.377 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:02.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:02.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:02.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:02.615 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:02.648 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:02.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:02.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:02.757 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 83 16 136475 136459 102.727 69.625 0.0047755 0.00972967 2024-03-24T12:08:02.787 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:02.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:02.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:02 smithi193 ceph-mon[33593]: pgmap v715: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s; 48962/392301 objects degraded (12.481%) 2024-03-24T12:08:02.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:02 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 48962/392301 objects degraded (12.481%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:02.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:02 smithi193 ceph-mon[33593]: osdmap e341: 8 total, 7 up, 8 in 2024-03-24T12:08:02.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:02.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:03.034 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:03.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:02 smithi064 ceph-mon[30223]: pgmap v715: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s; 48962/392301 objects degraded (12.481%) 2024-03-24T12:08:03.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:02 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 48962/392301 objects degraded (12.481%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:03.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:02 smithi064 ceph-mon[30223]: osdmap e341: 8 total, 7 up, 8 in 2024-03-24T12:08:03.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:02 smithi064 ceph-mon[42243]: pgmap v715: 168 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s; 48962/392301 objects degraded (12.481%) 2024-03-24T12:08:03.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:02 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 48962/392301 objects degraded (12.481%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:03.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:02 smithi064 ceph-mon[42243]: osdmap e341: 8 total, 7 up, 8 in 2024-03-24T12:08:03.064 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:03.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:03.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:03.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:03.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:03.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:03.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:03.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:03.501 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:03.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:03.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:03.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:03.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:03.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:08:03.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:03.758 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:08:03.761 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 84 16 137606 137590 102.344 70.6875 0.0172867 0.00976556 2024-03-24T12:08:03.762 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:08:03.783 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:03.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:03.885 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:03.913 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:03.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:04.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:04.056 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:04.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:04.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:04.198 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:04.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:04.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:04.334 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:04.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:04.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:04.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:04.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:04.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:04.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:04.600 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-03-24T12:08:03.457595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8875772975,"num_objects":135624,"num_object_clones":0,"num_object_copies":406872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135624,"num_whiteouts":0,"num_read":1336,"num_read_kb":1114,"num_write":136197,"num_write_kb":8667967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":484956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79020,"ondisk_log_size":79020,"up":435,"acting":435,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":507,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":27552204,"kb_used_data":22701956,"kb_used_omap":12,"kb_used_meta":4850163,"kb_avail":722409012,"statfs":{"total":767960285184,"available":739746828288,"internally_reserved":0,"allocated":23246802944,"data_stored":23243005435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":4966567472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":824770560,"num_objects":12585,"num_object_clones":0,"num_object_copies":37755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12585,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12585,"num_write_kb":805440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004251"},"pg_stats":[{"pgid":"10.6","version":"341'8387","reported_seq":8436,"reported_epoch":341,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:03.178725+0000","last_change":"2024-03-24T12:06:39.744919+0000","last_active":"2024-03-24T12:08:03.178725+0000","last_peered":"2024-03-24T12:08:03.178725+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.019918+0000","last_became_peered":"2024-03-24T12:06:30.019918+0000","last_unstale":"2024-03-24T12:08:03.178725+0000","last_undegraded":"2024-03-24T12:06:39.740916+0000","last_fullsized":"2024-03-24T12:06:30.013919+0000","mapping_epoch":320,"log_start":"329'3900","ondisk_log_start":"329'3900","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.272761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549650432,"num_objects":8387,"num_object_clones":0,"num_object_copies":25161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8387,"num_write_kb":536768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":8387}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"83'130","reported_seq":490,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.530336+0000","last_change":"2024-03-24T12:07:37.002106+0000","last_active":"2024-03-24T12:07:58.530336+0000","last_peered":"2024-03-24T12:07:58.530336+0000","last_clean":"2024-03-24T12:07:58.530336+0000","last_became_active":"2024-03-24T12:03:19.822426+0000","last_became_peered":"2024-03-24T12:03:19.822426+0000","last_unstale":"2024-03-24T12:07:58.530336+0000","last_undegraded":"2024-03-24T12:07:58.530336+0000","last_fullsized":"2024-03-24T12:07:58.530336+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:37.001978+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:37.001978+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:55.048255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":818,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.531538+0000","last_change":"2024-03-24T12:06:50.162950+0000","last_active":"2024-03-24T12:07:58.531538+0000","last_peered":"2024-03-24T12:07:58.531538+0000","last_clean":"2024-03-24T12:07:58.531538+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:07:58.531538+0000","last_undegraded":"2024-03-24T12:07:58.531538+0000","last_fullsized":"2024-03-24T12:07:58.531538+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:06:50.162823+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:06:50.162823+0000","last_clean_scrub_stamp":"2024-03-24T12:06:50.162823+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":481,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.529931+0000","last_change":"2024-03-24T12:06:27.010354+0000","last_active":"2024-03-24T12:07:58.529931+0000","last_peered":"2024-03-24T12:07:58.529931+0000","last_clean":"2024-03-24T12:02:22.958798+0000","last_became_active":"2024-03-24T12:06:27.010354+0000","last_became_peered":"2024-03-24T12:06:27.010354+0000","last_unstale":"2024-03-24T12:07:58.529931+0000","last_undegraded":"2024-03-24T12:06:27.003330+0000","last_fullsized":"2024-03-24T12:06:27.003218+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:05:17.253630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.253630+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.043271+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":91,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.566671+0000","last_change":"2024-03-24T12:05:44.732301+0000","last_active":"2024-03-24T12:07:58.566671+0000","last_peered":"2024-03-24T12:07:58.566671+0000","last_clean":"2024-03-24T12:07:58.566671+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:07:58.566671+0000","last_undegraded":"2024-03-24T12:07:58.566671+0000","last_fullsized":"2024-03-24T12:07:58.566671+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:11.505956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.505956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.543276+0000","last_change":"2024-03-24T12:06:27.009659+0000","last_active":"2024-03-24T12:07:58.543276+0000","last_peered":"2024-03-24T12:07:58.543276+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.009659+0000","last_became_peered":"2024-03-24T12:06:27.009659+0000","last_unstale":"2024-03-24T12:07:58.543276+0000","last_undegraded":"2024-03-24T12:07:58.543276+0000","last_fullsized":"2024-03-24T12:06:26.998639+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.095464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.095464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"341'8683","reported_seq":8809,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:02.050851+0000","last_change":"2024-03-24T12:07:35.723831+0000","last_active":"2024-03-24T12:08:02.050851+0000","last_peered":"2024-03-24T12:08:02.050851+0000","last_clean":"2024-03-24T12:08:02.050851+0000","last_became_active":"2024-03-24T12:06:30.016602+0000","last_became_peered":"2024-03-24T12:06:30.016602+0000","last_unstale":"2024-03-24T12:08:02.050851+0000","last_undegraded":"2024-03-24T12:08:02.050851+0000","last_fullsized":"2024-03-24T12:08:02.050851+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'6476","last_scrub_stamp":"2024-03-24T12:07:35.723542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:35.723542+0000","objects_scrubbed":6342,"log_size":8683,"log_dups_size":0,"ondisk_log_size":8683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:37.812028+0000","scrub_duration":2794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569049088,"num_objects":8683,"num_object_clones":0,"num_object_copies":26049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8683,"num_write_kb":555712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'130","reported_seq":489,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534396+0000","last_change":"2024-03-24T12:07:58.303998+0000","last_active":"2024-03-24T12:07:58.534396+0000","last_peered":"2024-03-24T12:07:58.534396+0000","last_clean":"2024-03-24T12:07:58.534396+0000","last_became_active":"2024-03-24T12:03:19.822938+0000","last_became_peered":"2024-03-24T12:03:19.822938+0000","last_unstale":"2024-03-24T12:07:58.534396+0000","last_undegraded":"2024-03-24T12:07:58.534396+0000","last_fullsized":"2024-03-24T12:07:58.534396+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:58.303527+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.303527+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:59.650037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":642,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.913458+0000","last_change":"2024-03-24T12:07:52.313886+0000","last_active":"2024-03-24T12:08:01.913458+0000","last_peered":"2024-03-24T12:08:01.913458+0000","last_clean":"2024-03-24T12:08:01.913458+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:08:01.913458+0000","last_undegraded":"2024-03-24T12:08:01.913458+0000","last_fullsized":"2024-03-24T12:08:01.913458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:52.313429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:52.313429+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:20.009793+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":634,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.537636+0000","last_change":"2024-03-24T12:06:54.372697+0000","last_active":"2024-03-24T12:07:58.537636+0000","last_peered":"2024-03-24T12:07:58.537636+0000","last_clean":"2024-03-24T12:07:58.537636+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:07:58.537636+0000","last_undegraded":"2024-03-24T12:07:58.537636+0000","last_fullsized":"2024-03-24T12:07:58.537636+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:54.372598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.372598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:21.803855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":99,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.533950+0000","last_change":"2024-03-24T12:07:49.261571+0000","last_active":"2024-03-24T12:07:58.533950+0000","last_peered":"2024-03-24T12:07:58.533950+0000","last_clean":"2024-03-24T12:07:58.533950+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:07:58.533950+0000","last_undegraded":"2024-03-24T12:07:58.533950+0000","last_fullsized":"2024-03-24T12:07:58.533950+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:07:49.261483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:50.112434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":607,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.913597+0000","last_change":"2024-03-24T12:07:43.508366+0000","last_active":"2024-03-24T12:08:01.913597+0000","last_peered":"2024-03-24T12:08:01.913597+0000","last_clean":"2024-03-24T12:08:01.913597+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:08:01.913597+0000","last_undegraded":"2024-03-24T12:08:01.913597+0000","last_fullsized":"2024-03-24T12:08:01.913597+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:43.508257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:43.508257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:54.454697+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"341'8661","reported_seq":8713,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:03.201471+0000","last_change":"2024-03-24T12:06:30.020617+0000","last_active":"2024-03-24T12:08:03.201471+0000","last_peered":"2024-03-24T12:08:03.201471+0000","last_clean":"2024-03-24T12:08:03.201471+0000","last_became_active":"2024-03-24T12:06:30.020223+0000","last_became_peered":"2024-03-24T12:06:30.020223+0000","last_unstale":"2024-03-24T12:08:03.201471+0000","last_undegraded":"2024-03-24T12:08:03.201471+0000","last_fullsized":"2024-03-24T12:08:03.201471+0000","mapping_epoch":320,"log_start":"330'4100","ondisk_log_start":"330'4100","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.272828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567607296,"num_objects":8661,"num_object_clones":0,"num_object_copies":25983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":554304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'130","reported_seq":670,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.543812+0000","last_change":"2024-03-24T12:06:27.005543+0000","last_active":"2024-03-24T12:07:58.543812+0000","last_peered":"2024-03-24T12:07:58.543812+0000","last_clean":"2024-03-24T12:06:22.267535+0000","last_became_active":"2024-03-24T12:06:27.005543+0000","last_became_peered":"2024-03-24T12:06:27.005543+0000","last_unstale":"2024-03-24T12:07:58.543812+0000","last_undegraded":"2024-03-24T12:07:58.543812+0000","last_fullsized":"2024-03-24T12:06:27.001250+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:02.370579+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.370579+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"296'44","reported_seq":685,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.544015+0000","last_change":"2024-03-24T12:07:37.639999+0000","last_active":"2024-03-24T12:07:58.544015+0000","last_peered":"2024-03-24T12:07:58.544015+0000","last_clean":"2024-03-24T12:07:58.544015+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:07:58.544015+0000","last_undegraded":"2024-03-24T12:07:58.544015+0000","last_fullsized":"2024-03-24T12:07:58.544015+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:07:37.639868+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:07:37.639868+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:54.792271+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":56,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.912982+0000","last_change":"2024-03-24T12:06:27.009675+0000","last_active":"2024-03-24T12:08:01.912982+0000","last_peered":"2024-03-24T12:08:01.912982+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.009675+0000","last_became_peered":"2024-03-24T12:06:27.009675+0000","last_unstale":"2024-03-24T12:08:01.912982+0000","last_undegraded":"2024-03-24T12:08:01.912982+0000","last_fullsized":"2024-03-24T12:06:26.998070+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:00.489543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:05:00.489543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.312956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":96,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.912368+0000","last_change":"2024-03-24T12:06:27.009828+0000","last_active":"2024-03-24T12:08:01.912368+0000","last_peered":"2024-03-24T12:08:01.912368+0000","last_clean":"2024-03-24T12:06:22.261311+0000","last_became_active":"2024-03-24T12:06:27.009828+0000","last_became_peered":"2024-03-24T12:06:27.009828+0000","last_unstale":"2024-03-24T12:08:01.912368+0000","last_undegraded":"2024-03-24T12:08:01.912368+0000","last_fullsized":"2024-03-24T12:06:26.998323+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:48.399346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:48.399346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.312942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":621,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.529650+0000","last_change":"2024-03-24T12:07:35.957089+0000","last_active":"2024-03-24T12:07:58.529650+0000","last_peered":"2024-03-24T12:07:58.529650+0000","last_clean":"2024-03-24T12:07:58.529650+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:07:58.529650+0000","last_undegraded":"2024-03-24T12:07:58.529650+0000","last_fullsized":"2024-03-24T12:07:58.529650+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:35.956974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:07:35.956974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:52.587297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"340'8373","reported_seq":8420,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.079621+0000","last_change":"2024-03-24T12:06:39.741733+0000","last_active":"2024-03-24T12:08:01.079621+0000","last_peered":"2024-03-24T12:08:01.079621+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.019248+0000","last_became_peered":"2024-03-24T12:06:30.019248+0000","last_unstale":"2024-03-24T12:08:01.079621+0000","last_undegraded":"2024-03-24T12:06:39.741580+0000","last_fullsized":"2024-03-24T12:06:30.008982+0000","mapping_epoch":320,"log_start":"330'4200","ondisk_log_start":"330'4200","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548732928,"num_objects":8373,"num_object_clones":0,"num_object_copies":25119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8373,"num_write_kb":535872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":8373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":694,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534233+0000","last_change":"2024-03-24T12:07:41.143174+0000","last_active":"2024-03-24T12:07:58.534233+0000","last_peered":"2024-03-24T12:07:58.534233+0000","last_clean":"2024-03-24T12:07:58.534233+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:07:58.534233+0000","last_undegraded":"2024-03-24T12:07:58.534233+0000","last_fullsized":"2024-03-24T12:07:58.534233+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:07:41.143037+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.143037+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:08.579732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":615,"reported_epoch":341,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.914793+0000","last_change":"2024-03-24T12:06:27.004815+0000","last_active":"2024-03-24T12:08:01.914793+0000","last_peered":"2024-03-24T12:08:01.914793+0000","last_clean":"2024-03-24T12:06:21.980599+0000","last_became_active":"2024-03-24T12:06:27.004815+0000","last_became_peered":"2024-03-24T12:06:27.004815+0000","last_unstale":"2024-03-24T12:08:01.914793+0000","last_undegraded":"2024-03-24T12:06:27.001638+0000","last_fullsized":"2024-03-24T12:06:27.001523+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:05:05.492318+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.492318+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":362,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534900+0000","last_change":"2024-03-24T12:07:48.250602+0000","last_active":"2024-03-24T12:07:58.534900+0000","last_peered":"2024-03-24T12:07:58.534900+0000","last_clean":"2024-03-24T12:07:58.534900+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:07:58.534900+0000","last_undegraded":"2024-03-24T12:07:58.534900+0000","last_fullsized":"2024-03-24T12:07:58.534900+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_clean_scrub_stamp":"2024-03-24T12:07:48.250404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:51.688996+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":90,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.556476+0000","last_change":"2024-03-24T12:05:44.734066+0000","last_active":"2024-03-24T12:07:58.556476+0000","last_peered":"2024-03-24T12:07:58.556476+0000","last_clean":"2024-03-24T12:07:58.556476+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:07:58.556476+0000","last_undegraded":"2024-03-24T12:07:58.556476+0000","last_fullsized":"2024-03-24T12:07:58.556476+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:12.726872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:12.726872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":350,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.914932+0000","last_change":"2024-03-24T12:06:27.007623+0000","last_active":"2024-03-24T12:08:01.914932+0000","last_peered":"2024-03-24T12:08:01.914932+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T12:06:27.007623+0000","last_became_peered":"2024-03-24T12:06:27.007623+0000","last_unstale":"2024-03-24T12:08:01.914932+0000","last_undegraded":"2024-03-24T12:08:01.914932+0000","last_fullsized":"2024-03-24T12:06:27.000534+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:19.243242+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.243242+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.824159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"340'8846","reported_seq":8925,"reported_epoch":340,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T12:08:01.747383+0000","last_change":"2024-03-24T12:07:58.286295+0000","last_active":"2024-03-24T12:08:01.747383+0000","last_peered":"2024-03-24T12:08:01.747383+0000","last_clean":"2024-03-24T12:08:01.747383+0000","last_became_active":"2024-03-24T12:06:30.018755+0000","last_became_peered":"2024-03-24T12:06:30.018755+0000","last_unstale":"2024-03-24T12:08:01.747383+0000","last_undegraded":"2024-03-24T12:08:01.747383+0000","last_fullsized":"2024-03-24T12:08:01.747383+0000","mapping_epoch":320,"log_start":"331'4201","ondisk_log_start":"331'4201","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":699,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547356672,"num_objects":8352,"num_object_clones":0,"num_object_copies":25056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":534528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":649,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534925+0000","last_change":"2024-03-24T12:07:56.328634+0000","last_active":"2024-03-24T12:07:58.534925+0000","last_peered":"2024-03-24T12:07:58.534925+0000","last_clean":"2024-03-24T12:07:58.534925+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:07:58.534925+0000","last_undegraded":"2024-03-24T12:07:58.534925+0000","last_fullsized":"2024-03-24T12:07:58.534925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:56.328393+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.328393+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:14.542386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":361,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.562301+0000","last_change":"2024-03-24T12:07:28.052211+0000","last_active":"2024-03-24T12:07:58.562301+0000","last_peered":"2024-03-24T12:07:58.562301+0000","last_clean":"2024-03-24T12:07:58.562301+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:07:58.562301+0000","last_undegraded":"2024-03-24T12:07:58.562301+0000","last_fullsized":"2024-03-24T12:07:58.562301+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:28.052079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:07:28.052079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:33.049940+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"319'9","reported_seq":815,"reported_epoch":341,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.915158+0000","last_change":"2024-03-24T12:06:27.007547+0000","last_active":"2024-03-24T12:08:01.915158+0000","last_peered":"2024-03-24T12:08:01.915158+0000","last_clean":"2024-03-24T12:06:25.366499+0000","last_became_active":"2024-03-24T12:06:27.007547+0000","last_became_peered":"2024-03-24T12:06:27.007547+0000","last_unstale":"2024-03-24T12:08:01.915158+0000","last_undegraded":"2024-03-24T12:06:27.001189+0000","last_fullsized":"2024-03-24T12:06:27.001079+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:06.478884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.478884+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":611,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.535022+0000","last_change":"2024-03-24T12:06:27.012210+0000","last_active":"2024-03-24T12:07:58.535022+0000","last_peered":"2024-03-24T12:07:58.535022+0000","last_clean":"2024-03-24T12:06:22.267578+0000","last_became_active":"2024-03-24T12:06:27.012210+0000","last_became_peered":"2024-03-24T12:06:27.012210+0000","last_unstale":"2024-03-24T12:07:58.535022+0000","last_undegraded":"2024-03-24T12:06:27.002891+0000","last_fullsized":"2024-03-24T12:06:27.002772+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:03:47.909970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:47.909970+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":905,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.531152+0000","last_change":"2024-03-24T12:06:27.013805+0000","last_active":"2024-03-24T12:07:58.531152+0000","last_peered":"2024-03-24T12:07:58.531152+0000","last_clean":"2024-03-24T12:06:22.145375+0000","last_became_active":"2024-03-24T12:06:27.013805+0000","last_became_peered":"2024-03-24T12:06:27.013805+0000","last_unstale":"2024-03-24T12:07:58.531152+0000","last_undegraded":"2024-03-24T12:07:58.531152+0000","last_fullsized":"2024-03-24T12:06:27.003965+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.142552+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.043246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"340'8586","reported_seq":8636,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.376199+0000","last_change":"2024-03-24T12:06:30.019783+0000","last_active":"2024-03-24T12:08:01.376199+0000","last_peered":"2024-03-24T12:08:01.376199+0000","last_clean":"2024-03-24T12:08:01.376199+0000","last_became_active":"2024-03-24T12:06:30.019501+0000","last_became_peered":"2024-03-24T12:06:30.019501+0000","last_unstale":"2024-03-24T12:08:01.376199+0000","last_undegraded":"2024-03-24T12:08:01.376199+0000","last_fullsized":"2024-03-24T12:08:01.376199+0000","mapping_epoch":320,"log_start":"331'4300","ondisk_log_start":"331'4300","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.337460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562692096,"num_objects":8586,"num_object_clones":0,"num_object_copies":25758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8586,"num_write_kb":549504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":672,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.537388+0000","last_change":"2024-03-24T12:06:48.430894+0000","last_active":"2024-03-24T12:07:58.537388+0000","last_peered":"2024-03-24T12:07:58.537388+0000","last_clean":"2024-03-24T12:07:58.537388+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:07:58.537388+0000","last_undegraded":"2024-03-24T12:07:58.537388+0000","last_fullsized":"2024-03-24T12:07:58.537388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:06:48.430734+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:06:48.430734+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.923096+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":627,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.537456+0000","last_change":"2024-03-24T12:06:38.413964+0000","last_active":"2024-03-24T12:07:58.537456+0000","last_peered":"2024-03-24T12:07:58.537456+0000","last_clean":"2024-03-24T12:07:58.537456+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:07:58.537456+0000","last_undegraded":"2024-03-24T12:07:58.537456+0000","last_fullsized":"2024-03-24T12:07:58.537456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:38.413878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:06:38.413878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.670119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"319'9","reported_seq":806,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.535102+0000","last_change":"2024-03-24T12:06:27.011265+0000","last_active":"2024-03-24T12:07:58.535102+0000","last_peered":"2024-03-24T12:07:58.535102+0000","last_clean":"2024-03-24T12:06:25.366490+0000","last_became_active":"2024-03-24T12:06:27.011265+0000","last_became_peered":"2024-03-24T12:06:27.011265+0000","last_unstale":"2024-03-24T12:07:58.535102+0000","last_undegraded":"2024-03-24T12:06:26.998885+0000","last_fullsized":"2024-03-24T12:06:26.998748+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:05.860814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.860814+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.322540+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.910788+0000","last_change":"2024-03-24T12:06:27.012535+0000","last_active":"2024-03-24T12:08:01.910788+0000","last_peered":"2024-03-24T12:08:01.910788+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.012535+0000","last_became_peered":"2024-03-24T12:06:27.012535+0000","last_unstale":"2024-03-24T12:08:01.910788+0000","last_undegraded":"2024-03-24T12:08:01.910788+0000","last_fullsized":"2024-03-24T12:06:26.998792+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.219541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.219541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.312895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":350,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.914614+0000","last_change":"2024-03-24T12:06:27.005493+0000","last_active":"2024-03-24T12:08:01.914614+0000","last_peered":"2024-03-24T12:08:01.914614+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T12:06:27.005493+0000","last_became_peered":"2024-03-24T12:06:27.005493+0000","last_unstale":"2024-03-24T12:08:01.914614+0000","last_undegraded":"2024-03-24T12:08:01.914614+0000","last_fullsized":"2024-03-24T12:06:27.001315+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:11.242568+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.242568+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.824140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"340'8540","reported_seq":8670,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.455770+0000","last_change":"2024-03-24T12:07:45.462425+0000","last_active":"2024-03-24T12:08:01.455770+0000","last_peered":"2024-03-24T12:08:01.455770+0000","last_clean":"2024-03-24T12:08:01.455770+0000","last_became_active":"2024-03-24T12:06:30.016735+0000","last_became_peered":"2024-03-24T12:06:30.016735+0000","last_unstale":"2024-03-24T12:08:01.455770+0000","last_undegraded":"2024-03-24T12:08:01.455770+0000","last_fullsized":"2024-03-24T12:08:01.455770+0000","mapping_epoch":320,"log_start":"329'4000","ondisk_log_start":"329'4000","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7238","last_scrub_stamp":"2024-03-24T12:07:45.462060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.462060+0000","objects_scrubbed":7162,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:48.854132+0000","scrub_duration":1839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":559677440,"num_objects":8540,"num_object_clones":0,"num_object_copies":25620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":546560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"83'130","reported_seq":488,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.911322+0000","last_change":"2024-03-24T12:06:27.010346+0000","last_active":"2024-03-24T12:08:01.911322+0000","last_peered":"2024-03-24T12:08:01.911322+0000","last_clean":"2024-03-24T12:06:22.261133+0000","last_became_active":"2024-03-24T12:06:27.010346+0000","last_became_peered":"2024-03-24T12:06:27.010346+0000","last_unstale":"2024-03-24T12:08:01.911322+0000","last_undegraded":"2024-03-24T12:08:01.911322+0000","last_fullsized":"2024-03-24T12:06:26.997700+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:20.649114+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:20.649114+0000","last_clean_scrub_stamp":"2024-03-24T12:05:20.649114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.313329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":641,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534502+0000","last_change":"2024-03-24T12:06:46.690666+0000","last_active":"2024-03-24T12:07:58.534502+0000","last_peered":"2024-03-24T12:07:58.534502+0000","last_clean":"2024-03-24T12:07:58.534502+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:07:58.534502+0000","last_undegraded":"2024-03-24T12:07:58.534502+0000","last_fullsized":"2024-03-24T12:07:58.534502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:06:46.690548+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:06:46.690548+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":627,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.529844+0000","last_change":"2024-03-24T12:06:55.146672+0000","last_active":"2024-03-24T12:07:58.529844+0000","last_peered":"2024-03-24T12:07:58.529844+0000","last_clean":"2024-03-24T12:07:58.529844+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:07:58.529844+0000","last_undegraded":"2024-03-24T12:07:58.529844+0000","last_fullsized":"2024-03-24T12:07:58.529844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:55.146558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:06:55.146558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.951968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":52,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.534072+0000","last_change":"2024-03-24T12:06:27.008428+0000","last_active":"2024-03-24T12:07:58.534072+0000","last_peered":"2024-03-24T12:07:58.534072+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.008428+0000","last_became_peered":"2024-03-24T12:06:27.008428+0000","last_unstale":"2024-03-24T12:07:58.534072+0000","last_undegraded":"2024-03-24T12:07:58.534072+0000","last_fullsized":"2024-03-24T12:06:27.000650+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:15.334211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:15.334211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":602,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.556685+0000","last_change":"2024-03-24T12:06:27.006596+0000","last_active":"2024-03-24T12:07:58.556685+0000","last_peered":"2024-03-24T12:07:58.556685+0000","last_clean":"2024-03-24T12:06:21.985308+0000","last_became_active":"2024-03-24T12:06:27.006596+0000","last_became_peered":"2024-03-24T12:06:27.006596+0000","last_unstale":"2024-03-24T12:07:58.556685+0000","last_undegraded":"2024-03-24T12:07:58.556685+0000","last_fullsized":"2024-03-24T12:06:27.000436+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:06.995768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:06.995768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"340'8427","reported_seq":8568,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.439799+0000","last_change":"2024-03-24T12:07:58.231671+0000","last_active":"2024-03-24T12:08:01.439799+0000","last_peered":"2024-03-24T12:08:01.439799+0000","last_clean":"2024-03-24T12:08:01.439799+0000","last_became_active":"2024-03-24T12:06:30.015853+0000","last_became_peered":"2024-03-24T12:06:30.015853+0000","last_unstale":"2024-03-24T12:08:01.439799+0000","last_undegraded":"2024-03-24T12:08:01.439799+0000","last_fullsized":"2024-03-24T12:08:01.439799+0000","mapping_epoch":320,"log_start":"329'3900","ondisk_log_start":"329'3900","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'8176","last_scrub_stamp":"2024-03-24T12:07:58.231548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.231548+0000","objects_scrubbed":8101,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:26.835989+0000","scrub_duration":2379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":552271872,"num_objects":8427,"num_object_clones":0,"num_object_copies":25281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8427,"num_write_kb":539328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":802,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.553032+0000","last_change":"2024-03-24T12:07:34.996457+0000","last_active":"2024-03-24T12:07:58.553032+0000","last_peered":"2024-03-24T12:07:58.553032+0000","last_clean":"2024-03-24T12:07:58.553032+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:07:58.553032+0000","last_undegraded":"2024-03-24T12:07:58.553032+0000","last_fullsized":"2024-03-24T12:07:58.553032+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:07:34.996314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:07:34.996314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:57.285026+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":658,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.533940+0000","last_change":"2024-03-24T12:06:27.007808+0000","last_active":"2024-03-24T12:07:58.533940+0000","last_peered":"2024-03-24T12:07:58.533940+0000","last_clean":"2024-03-24T12:06:22.260785+0000","last_became_active":"2024-03-24T12:06:27.007808+0000","last_became_peered":"2024-03-24T12:06:27.007808+0000","last_unstale":"2024-03-24T12:07:58.533940+0000","last_undegraded":"2024-03-24T12:06:27.001358+0000","last_fullsized":"2024-03-24T12:06:27.001242+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:05:29.699282+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:05:29.699282+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.301143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":625,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.544144+0000","last_change":"2024-03-24T12:07:30.557017+0000","last_active":"2024-03-24T12:07:58.544144+0000","last_peered":"2024-03-24T12:07:58.544144+0000","last_clean":"2024-03-24T12:07:58.544144+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:07:58.544144+0000","last_undegraded":"2024-03-24T12:07:58.544144+0000","last_fullsized":"2024-03-24T12:07:58.544144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.556903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.556903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:34.495382+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":629,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.530566+0000","last_change":"2024-03-24T12:07:45.151754+0000","last_active":"2024-03-24T12:07:58.530566+0000","last_peered":"2024-03-24T12:07:58.530566+0000","last_clean":"2024-03-24T12:07:58.530566+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:07:58.530566+0000","last_undegraded":"2024-03-24T12:07:58.530566+0000","last_fullsized":"2024-03-24T12:07:58.530566+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:45.151622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.151622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:55.111128+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":640,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.552733+0000","last_change":"2024-03-24T12:06:27.013509+0000","last_active":"2024-03-24T12:07:58.552733+0000","last_peered":"2024-03-24T12:07:58.552733+0000","last_clean":"2024-03-24T12:06:21.993056+0000","last_became_active":"2024-03-24T12:06:27.013509+0000","last_became_peered":"2024-03-24T12:06:27.013509+0000","last_unstale":"2024-03-24T12:07:58.552733+0000","last_undegraded":"2024-03-24T12:07:58.552733+0000","last_fullsized":"2024-03-24T12:06:27.000946+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:15.627055+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:15.627055+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045659+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"340'8322","reported_seq":8369,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.353269+0000","last_change":"2024-03-24T12:06:39.747440+0000","last_active":"2024-03-24T12:08:01.353269+0000","last_peered":"2024-03-24T12:08:01.353269+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.017588+0000","last_became_peered":"2024-03-24T12:06:30.017588+0000","last_unstale":"2024-03-24T12:08:01.353269+0000","last_undegraded":"2024-03-24T12:06:39.746623+0000","last_fullsized":"2024-03-24T12:06:30.012557+0000","mapping_epoch":320,"log_start":"330'4000","ondisk_log_start":"330'4000","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4322,"log_dups_size":0,"ondisk_log_size":4322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.337365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":545390592,"num_objects":8322,"num_object_clones":0,"num_object_copies":24966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8322,"num_write_kb":532608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":8322}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":602,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534201+0000","last_change":"2024-03-24T12:06:44.708778+0000","last_active":"2024-03-24T12:07:58.534201+0000","last_peered":"2024-03-24T12:07:58.534201+0000","last_clean":"2024-03-24T12:07:58.534201+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:07:58.534201+0000","last_undegraded":"2024-03-24T12:07:58.534201+0000","last_fullsized":"2024-03-24T12:07:58.534201+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:44.708677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:06:44.708677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.105771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":666,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.552885+0000","last_change":"2024-03-24T12:06:54.925013+0000","last_active":"2024-03-24T12:07:58.552885+0000","last_peered":"2024-03-24T12:07:58.552885+0000","last_clean":"2024-03-24T12:07:58.552885+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:07:58.552885+0000","last_undegraded":"2024-03-24T12:07:58.552885+0000","last_fullsized":"2024-03-24T12:07:58.552885+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:06:54.924751+0000","last_deep_scrub":"195'26","last_deep_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.924751+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:08.422773+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":378,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543820+0000","last_change":"2024-03-24T12:07:40.603920+0000","last_active":"2024-03-24T12:07:58.543820+0000","last_peered":"2024-03-24T12:07:58.543820+0000","last_clean":"2024-03-24T12:07:58.543820+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:07:58.543820+0000","last_undegraded":"2024-03-24T12:07:58.543820+0000","last_fullsized":"2024-03-24T12:07:58.543820+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:07:40.603768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.603768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:56.003708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":620,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534044+0000","last_change":"2024-03-24T12:06:39.791204+0000","last_active":"2024-03-24T12:07:58.534044+0000","last_peered":"2024-03-24T12:07:58.534044+0000","last_clean":"2024-03-24T12:07:58.534044+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:07:58.534044+0000","last_undegraded":"2024-03-24T12:07:58.534044+0000","last_fullsized":"2024-03-24T12:07:58.534044+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:39.791081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.791081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":657,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.533577+0000","last_change":"2024-03-24T12:06:27.008961+0000","last_active":"2024-03-24T12:07:58.533577+0000","last_peered":"2024-03-24T12:07:58.533577+0000","last_clean":"2024-03-24T12:06:22.266015+0000","last_became_active":"2024-03-24T12:06:27.008961+0000","last_became_peered":"2024-03-24T12:06:27.008961+0000","last_unstale":"2024-03-24T12:07:58.533577+0000","last_undegraded":"2024-03-24T12:07:58.533577+0000","last_fullsized":"2024-03-24T12:06:27.001539+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:18.897352+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:18.897352+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"340'8377","reported_seq":8507,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.468860+0000","last_change":"2024-03-24T12:07:46.785479+0000","last_active":"2024-03-24T12:08:01.468860+0000","last_peered":"2024-03-24T12:08:01.468860+0000","last_clean":"2024-03-24T12:08:01.468860+0000","last_became_active":"2024-03-24T12:06:30.017829+0000","last_became_peered":"2024-03-24T12:06:30.017829+0000","last_unstale":"2024-03-24T12:08:01.468860+0000","last_undegraded":"2024-03-24T12:08:01.468860+0000","last_fullsized":"2024-03-24T12:08:01.468860+0000","mapping_epoch":320,"log_start":"329'3800","ondisk_log_start":"329'3800","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7161","last_scrub_stamp":"2024-03-24T12:07:46.785035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:46.785035+0000","objects_scrubbed":7079,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:06.645060+0000","scrub_duration":2151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548995072,"num_objects":8377,"num_object_clones":0,"num_object_copies":25131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8377,"num_write_kb":536128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":612,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.544073+0000","last_change":"2024-03-24T12:06:27.009137+0000","last_active":"2024-03-24T12:07:58.544073+0000","last_peered":"2024-03-24T12:07:58.544073+0000","last_clean":"2024-03-24T12:06:22.261276+0000","last_became_active":"2024-03-24T12:06:27.009137+0000","last_became_peered":"2024-03-24T12:06:27.009137+0000","last_unstale":"2024-03-24T12:07:58.544073+0000","last_undegraded":"2024-03-24T12:07:58.544073+0000","last_fullsized":"2024-03-24T12:06:26.998630+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:24.576146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:05:24.576146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"241'29","reported_seq":655,"reported_epoch":341,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.912518+0000","last_change":"2024-03-24T12:06:27.010256+0000","last_active":"2024-03-24T12:08:01.912518+0000","last_peered":"2024-03-24T12:08:01.912518+0000","last_clean":"2024-03-24T12:06:22.261479+0000","last_became_active":"2024-03-24T12:06:27.010256+0000","last_became_peered":"2024-03-24T12:06:27.010256+0000","last_unstale":"2024-03-24T12:08:01.912518+0000","last_undegraded":"2024-03-24T12:06:26.998858+0000","last_fullsized":"2024-03-24T12:06:26.998745+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-24T12:05:09.620298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.620298+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.312971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":635,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.530901+0000","last_change":"2024-03-24T12:07:39.003839+0000","last_active":"2024-03-24T12:07:58.530901+0000","last_peered":"2024-03-24T12:07:58.530901+0000","last_clean":"2024-03-24T12:07:58.530901+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:07:58.530901+0000","last_undegraded":"2024-03-24T12:07:58.530901+0000","last_fullsized":"2024-03-24T12:07:58.530901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:39.003694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:07:39.003694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:46.052694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":368,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543110+0000","last_change":"2024-03-24T12:06:40.600601+0000","last_active":"2024-03-24T12:07:58.543110+0000","last_peered":"2024-03-24T12:07:58.543110+0000","last_clean":"2024-03-24T12:07:58.543110+0000","last_became_active":"2024-03-24T12:05:44.738926+0000","last_became_peered":"2024-03-24T12:05:44.738926+0000","last_unstale":"2024-03-24T12:07:58.543110+0000","last_undegraded":"2024-03-24T12:07:58.543110+0000","last_fullsized":"2024-03-24T12:07:58.543110+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:40.600197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:40.600197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":667,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.544012+0000","last_change":"2024-03-24T12:06:49.509554+0000","last_active":"2024-03-24T12:07:58.544012+0000","last_peered":"2024-03-24T12:07:58.544012+0000","last_clean":"2024-03-24T12:07:58.544012+0000","last_became_active":"2024-03-24T12:03:15.295100+0000","last_became_peered":"2024-03-24T12:03:15.295100+0000","last_unstale":"2024-03-24T12:07:58.544012+0000","last_undegraded":"2024-03-24T12:07:58.544012+0000","last_fullsized":"2024-03-24T12:07:58.544012+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:49.509435+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.509435+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"341'8530","reported_seq":8582,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:03.175140+0000","last_change":"2024-03-24T12:06:30.018544+0000","last_active":"2024-03-24T12:08:03.175140+0000","last_peered":"2024-03-24T12:08:03.175140+0000","last_clean":"2024-03-24T12:08:03.175140+0000","last_became_active":"2024-03-24T12:06:30.015684+0000","last_became_peered":"2024-03-24T12:06:30.015684+0000","last_unstale":"2024-03-24T12:08:03.175140+0000","last_undegraded":"2024-03-24T12:08:03.175140+0000","last_fullsized":"2024-03-24T12:08:03.175140+0000","mapping_epoch":320,"log_start":"330'4000","ondisk_log_start":"330'4000","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.272797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":559022080,"num_objects":8530,"num_object_clones":0,"num_object_copies":25590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":545920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":54,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.910707+0000","last_change":"2024-03-24T12:06:27.009282+0000","last_active":"2024-03-24T12:08:01.910707+0000","last_peered":"2024-03-24T12:08:01.910707+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.009282+0000","last_became_peered":"2024-03-24T12:06:27.009282+0000","last_unstale":"2024-03-24T12:08:01.910707+0000","last_undegraded":"2024-03-24T12:08:01.910707+0000","last_fullsized":"2024-03-24T12:06:26.997206+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:28.232417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:28.232417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.313343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":671,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.924270+0000","last_change":"2024-03-24T12:06:39.802730+0000","last_active":"2024-03-24T12:08:01.924270+0000","last_peered":"2024-03-24T12:08:01.924270+0000","last_clean":"2024-03-24T12:08:01.924270+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:08:01.924270+0000","last_undegraded":"2024-03-24T12:08:01.924270+0000","last_fullsized":"2024-03-24T12:08:01.924270+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:06:39.802577+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.802577+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.272858+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":457,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.913386+0000","last_change":"2024-03-24T12:07:34.290389+0000","last_active":"2024-03-24T12:08:01.913386+0000","last_peered":"2024-03-24T12:08:01.913386+0000","last_clean":"2024-03-24T12:08:01.913386+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:08:01.913386+0000","last_undegraded":"2024-03-24T12:08:01.913386+0000","last_fullsized":"2024-03-24T12:08:01.913386+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:34.290275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:34.290275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:46.355504+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":628,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.913376+0000","last_change":"2024-03-24T12:07:39.389030+0000","last_active":"2024-03-24T12:08:01.913376+0000","last_peered":"2024-03-24T12:08:01.913376+0000","last_clean":"2024-03-24T12:08:01.913376+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:08:01.913376+0000","last_undegraded":"2024-03-24T12:08:01.913376+0000","last_fullsized":"2024-03-24T12:08:01.913376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:39.387609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:07:39.387609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:03.466139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":610,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.537871+0000","last_change":"2024-03-24T12:06:27.015212+0000","last_active":"2024-03-24T12:07:58.537871+0000","last_peered":"2024-03-24T12:07:58.537871+0000","last_clean":"2024-03-24T12:06:21.982922+0000","last_became_active":"2024-03-24T12:06:27.015212+0000","last_became_peered":"2024-03-24T12:06:27.015212+0000","last_unstale":"2024-03-24T12:07:58.537871+0000","last_undegraded":"2024-03-24T12:07:58.537871+0000","last_fullsized":"2024-03-24T12:06:27.003034+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.016987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.016987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.283510+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":667,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543409+0000","last_change":"2024-03-24T12:06:54.603424+0000","last_active":"2024-03-24T12:07:58.543409+0000","last_peered":"2024-03-24T12:07:58.543409+0000","last_clean":"2024-03-24T12:07:58.543409+0000","last_became_active":"2024-03-24T12:03:15.297422+0000","last_became_peered":"2024-03-24T12:03:15.297422+0000","last_unstale":"2024-03-24T12:07:58.543409+0000","last_undegraded":"2024-03-24T12:07:58.543409+0000","last_fullsized":"2024-03-24T12:07:58.543409+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:54.603313+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.603313+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:09.509099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"340'8419","reported_seq":8468,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.440505+0000","last_change":"2024-03-24T12:06:30.018406+0000","last_active":"2024-03-24T12:08:01.440505+0000","last_peered":"2024-03-24T12:08:01.440505+0000","last_clean":"2024-03-24T12:08:01.440505+0000","last_became_active":"2024-03-24T12:06:30.017471+0000","last_became_peered":"2024-03-24T12:06:30.017471+0000","last_unstale":"2024-03-24T12:08:01.440505+0000","last_undegraded":"2024-03-24T12:08:01.440505+0000","last_fullsized":"2024-03-24T12:08:01.440505+0000","mapping_epoch":320,"log_start":"329'3900","ondisk_log_start":"329'3900","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4519,"log_dups_size":0,"ondisk_log_size":4519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":551747584,"num_objects":8419,"num_object_clones":0,"num_object_copies":25257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":538816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":614,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.531093+0000","last_change":"2024-03-24T12:06:27.009330+0000","last_active":"2024-03-24T12:07:58.531093+0000","last_peered":"2024-03-24T12:07:58.531093+0000","last_clean":"2024-03-24T12:06:22.145617+0000","last_became_active":"2024-03-24T12:06:27.009330+0000","last_became_peered":"2024-03-24T12:06:27.009330+0000","last_unstale":"2024-03-24T12:07:58.531093+0000","last_undegraded":"2024-03-24T12:07:58.531093+0000","last_fullsized":"2024-03-24T12:06:27.001828+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:06.100942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:05:06.100942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.043258+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":859,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543770+0000","last_change":"2024-03-24T12:06:46.537107+0000","last_active":"2024-03-24T12:07:58.543770+0000","last_peered":"2024-03-24T12:07:58.543770+0000","last_clean":"2024-03-24T12:07:58.543770+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:07:58.543770+0000","last_undegraded":"2024-03-24T12:07:58.543770+0000","last_fullsized":"2024-03-24T12:07:58.543770+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:06:46.536897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:46.536897+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":899,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.529815+0000","last_change":"2024-03-24T12:06:57.082478+0000","last_active":"2024-03-24T12:07:58.529815+0000","last_peered":"2024-03-24T12:07:58.529815+0000","last_clean":"2024-03-24T12:07:58.529815+0000","last_became_active":"2024-03-24T12:03:15.298475+0000","last_became_peered":"2024-03-24T12:03:15.298475+0000","last_unstale":"2024-03-24T12:07:58.529815+0000","last_undegraded":"2024-03-24T12:07:58.529815+0000","last_fullsized":"2024-03-24T12:07:58.529815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:57.082342+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:03:16.845832+0000","last_clean_scrub_stamp":"2024-03-24T12:06:57.082342+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.915058+0000","last_change":"2024-03-24T12:06:27.007334+0000","last_active":"2024-03-24T12:08:01.915058+0000","last_peered":"2024-03-24T12:08:01.915058+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.007334+0000","last_became_peered":"2024-03-24T12:06:27.007334+0000","last_unstale":"2024-03-24T12:08:01.915058+0000","last_undegraded":"2024-03-24T12:08:01.915058+0000","last_fullsized":"2024-03-24T12:06:27.001839+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.356759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.356759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":616,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.562079+0000","last_change":"2024-03-24T12:06:27.004802+0000","last_active":"2024-03-24T12:07:58.562079+0000","last_peered":"2024-03-24T12:07:58.562079+0000","last_clean":"2024-03-24T12:06:21.984854+0000","last_became_active":"2024-03-24T12:06:27.004802+0000","last_became_peered":"2024-03-24T12:06:27.004802+0000","last_unstale":"2024-03-24T12:07:58.562079+0000","last_undegraded":"2024-03-24T12:06:26.998482+0000","last_fullsized":"2024-03-24T12:06:26.998367+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:04:08.679904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:08.679904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045674+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.556280+0000","last_change":"2024-03-24T12:06:27.007879+0000","last_active":"2024-03-24T12:07:58.556280+0000","last_peered":"2024-03-24T12:07:58.556280+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.007879+0000","last_became_peered":"2024-03-24T12:06:27.007879+0000","last_unstale":"2024-03-24T12:07:58.556280+0000","last_undegraded":"2024-03-24T12:07:58.556280+0000","last_fullsized":"2024-03-24T12:06:26.999940+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.463242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.463242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"340'8349","reported_seq":8396,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.598731+0000","last_change":"2024-03-24T12:06:39.746288+0000","last_active":"2024-03-24T12:08:01.598731+0000","last_peered":"2024-03-24T12:08:01.598731+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.019333+0000","last_became_peered":"2024-03-24T12:06:30.019333+0000","last_unstale":"2024-03-24T12:08:01.598731+0000","last_undegraded":"2024-03-24T12:06:39.740158+0000","last_fullsized":"2024-03-24T12:06:30.012518+0000","mapping_epoch":320,"log_start":"330'4100","ondisk_log_start":"330'4100","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:06.162364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547160064,"num_objects":8349,"num_object_clones":0,"num_object_copies":25047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":8349}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":76,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.554665+0000","last_change":"2024-03-24T12:06:27.009761+0000","last_active":"2024-03-24T12:07:58.554665+0000","last_peered":"2024-03-24T12:07:58.554665+0000","last_clean":"2024-03-24T12:05:54.824542+0000","last_became_active":"2024-03-24T12:06:27.009761+0000","last_became_peered":"2024-03-24T12:06:27.009761+0000","last_unstale":"2024-03-24T12:07:58.554665+0000","last_undegraded":"2024-03-24T12:07:58.554665+0000","last_fullsized":"2024-03-24T12:06:27.000267+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.646040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":651,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.537139+0000","last_change":"2024-03-24T12:06:27.011400+0000","last_active":"2024-03-24T12:07:58.537139+0000","last_peered":"2024-03-24T12:07:58.537139+0000","last_clean":"2024-03-24T12:06:21.982394+0000","last_became_active":"2024-03-24T12:06:27.011400+0000","last_became_peered":"2024-03-24T12:06:27.011400+0000","last_unstale":"2024-03-24T12:07:58.537139+0000","last_undegraded":"2024-03-24T12:06:27.002875+0000","last_fullsized":"2024-03-24T12:06:27.002745+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:05:08.083956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:08.083956+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.283539+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.555331+0000","last_change":"2024-03-24T12:05:44.739463+0000","last_active":"2024-03-24T12:07:58.555331+0000","last_peered":"2024-03-24T12:07:58.555331+0000","last_clean":"2024-03-24T12:07:58.555331+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:07:58.555331+0000","last_undegraded":"2024-03-24T12:07:58.555331+0000","last_fullsized":"2024-03-24T12:07:58.555331+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.610545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.610545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":281,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:59.952135+0000","last_change":"2024-03-24T12:06:55.648298+0000","last_active":"2024-03-24T12:07:59.952135+0000","last_peered":"2024-03-24T12:07:59.952135+0000","last_clean":"2024-03-24T12:07:59.952135+0000","last_became_active":"2024-03-24T12:05:44.736437+0000","last_became_peered":"2024-03-24T12:05:44.736437+0000","last_unstale":"2024-03-24T12:07:59.952135+0000","last_undegraded":"2024-03-24T12:07:59.952135+0000","last_fullsized":"2024-03-24T12:07:59.952135+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:06:55.648166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:55.648166+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:10.161881+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":143,"num_read_kb":138,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":615,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.530813+0000","last_change":"2024-03-24T12:07:28.069295+0000","last_active":"2024-03-24T12:07:58.530813+0000","last_peered":"2024-03-24T12:07:58.530813+0000","last_clean":"2024-03-24T12:07:58.530813+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:07:58.530813+0000","last_undegraded":"2024-03-24T12:07:58.530813+0000","last_fullsized":"2024-03-24T12:07:58.530813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:28.068587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:28.068587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:49.282076+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":906,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.530044+0000","last_change":"2024-03-24T12:06:56.127437+0000","last_active":"2024-03-24T12:07:58.530044+0000","last_peered":"2024-03-24T12:07:58.530044+0000","last_clean":"2024-03-24T12:07:58.530044+0000","last_became_active":"2024-03-24T12:03:15.298007+0000","last_became_peered":"2024-03-24T12:03:15.298007+0000","last_unstale":"2024-03-24T12:07:58.530044+0000","last_undegraded":"2024-03-24T12:07:58.530044+0000","last_fullsized":"2024-03-24T12:07:58.530044+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:56.127305+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:56.127305+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:13.466011+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"340'8465","reported_seq":8512,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.460932+0000","last_change":"2024-03-24T12:06:39.741514+0000","last_active":"2024-03-24T12:08:01.460932+0000","last_peered":"2024-03-24T12:08:01.460932+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.016584+0000","last_became_peered":"2024-03-24T12:06:30.016584+0000","last_unstale":"2024-03-24T12:08:01.460932+0000","last_undegraded":"2024-03-24T12:06:39.740877+0000","last_fullsized":"2024-03-24T12:06:30.008363+0000","mapping_epoch":320,"log_start":"329'3900","ondisk_log_start":"329'3900","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554762240,"num_objects":8465,"num_object_clones":0,"num_object_copies":25395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":541760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":8465}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":605,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.911498+0000","last_change":"2024-03-24T12:06:47.944634+0000","last_active":"2024-03-24T12:08:01.911498+0000","last_peered":"2024-03-24T12:08:01.911498+0000","last_clean":"2024-03-24T12:08:01.911498+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:08:01.911498+0000","last_undegraded":"2024-03-24T12:08:01.911498+0000","last_fullsized":"2024-03-24T12:08:01.911498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:47.944504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:06:47.944504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:10.660377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"319'43","reported_seq":598,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.557145+0000","last_change":"2024-03-24T12:06:27.007856+0000","last_active":"2024-03-24T12:07:58.557145+0000","last_peered":"2024-03-24T12:07:58.557145+0000","last_clean":"2024-03-24T12:02:22.995365+0000","last_became_active":"2024-03-24T12:06:27.007856+0000","last_became_peered":"2024-03-24T12:06:27.007856+0000","last_unstale":"2024-03-24T12:07:58.557145+0000","last_undegraded":"2024-03-24T12:06:27.001627+0000","last_fullsized":"2024-03-24T12:06:27.001501+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-24T12:04:07.131550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:07.131550+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045714+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.541855+0000","last_change":"2024-03-24T12:06:27.006538+0000","last_active":"2024-03-24T12:07:58.541855+0000","last_peered":"2024-03-24T12:07:58.541855+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.006538+0000","last_became_peered":"2024-03-24T12:06:27.006538+0000","last_unstale":"2024-03-24T12:07:58.541855+0000","last_undegraded":"2024-03-24T12:07:58.541855+0000","last_fullsized":"2024-03-24T12:06:27.003033+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.196749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.196749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.283497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":600,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.567295+0000","last_change":"2024-03-24T12:07:33.984169+0000","last_active":"2024-03-24T12:07:58.567295+0000","last_peered":"2024-03-24T12:07:58.567295+0000","last_clean":"2024-03-24T12:07:58.567295+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:07:58.567295+0000","last_undegraded":"2024-03-24T12:07:58.567295+0000","last_fullsized":"2024-03-24T12:07:58.567295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:33.984042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:33.984042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:44.204090+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":357,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.913103+0000","last_change":"2024-03-24T12:06:27.005399+0000","last_active":"2024-03-24T12:08:01.913103+0000","last_peered":"2024-03-24T12:08:01.913103+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T12:06:27.005399+0000","last_became_peered":"2024-03-24T12:06:27.005399+0000","last_unstale":"2024-03-24T12:08:01.913103+0000","last_undegraded":"2024-03-24T12:08:01.913103+0000","last_fullsized":"2024-03-24T12:06:26.996752+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:47.152371+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:47.152371+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.272743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"340'8463","reported_seq":8513,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.602547+0000","last_change":"2024-03-24T12:06:30.018151+0000","last_active":"2024-03-24T12:08:01.602547+0000","last_peered":"2024-03-24T12:08:01.602547+0000","last_clean":"2024-03-24T12:08:01.602547+0000","last_became_active":"2024-03-24T12:06:30.017875+0000","last_became_peered":"2024-03-24T12:06:30.017875+0000","last_unstale":"2024-03-24T12:08:01.602547+0000","last_undegraded":"2024-03-24T12:08:01.602547+0000","last_fullsized":"2024-03-24T12:08:01.602547+0000","mapping_epoch":320,"log_start":"332'4200","ondisk_log_start":"332'4200","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4263,"log_dups_size":0,"ondisk_log_size":4263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:06.162471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554631168,"num_objects":8463,"num_object_clones":0,"num_object_copies":25389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8463,"num_write_kb":541632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"319'9","reported_seq":807,"reported_epoch":341,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.912952+0000","last_change":"2024-03-24T12:06:27.009093+0000","last_active":"2024-03-24T12:08:01.912952+0000","last_peered":"2024-03-24T12:08:01.912952+0000","last_clean":"2024-03-24T12:06:25.366554+0000","last_became_active":"2024-03-24T12:06:27.009093+0000","last_became_peered":"2024-03-24T12:06:27.009093+0000","last_unstale":"2024-03-24T12:08:01.912952+0000","last_undegraded":"2024-03-24T12:06:26.999413+0000","last_fullsized":"2024-03-24T12:06:26.999301+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:02.500062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.500062+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.313039+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":653,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.542354+0000","last_change":"2024-03-24T12:06:27.010753+0000","last_active":"2024-03-24T12:07:58.542354+0000","last_peered":"2024-03-24T12:07:58.542354+0000","last_clean":"2024-03-24T12:06:21.984527+0000","last_became_active":"2024-03-24T12:06:27.010753+0000","last_became_peered":"2024-03-24T12:06:27.010753+0000","last_unstale":"2024-03-24T12:07:58.542354+0000","last_undegraded":"2024-03-24T12:06:27.004797+0000","last_fullsized":"2024-03-24T12:06:27.004688+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:05:05.061844+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.061844+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.283525+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":364,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.913245+0000","last_change":"2024-03-24T12:06:49.982193+0000","last_active":"2024-03-24T12:08:01.913245+0000","last_peered":"2024-03-24T12:08:01.913245+0000","last_clean":"2024-03-24T12:08:01.913245+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:08:01.913245+0000","last_undegraded":"2024-03-24T12:08:01.913245+0000","last_fullsized":"2024-03-24T12:08:01.913245+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:49.981897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.981897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:05.069700+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":602,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.912399+0000","last_change":"2024-03-24T12:06:27.011434+0000","last_active":"2024-03-24T12:08:01.912399+0000","last_peered":"2024-03-24T12:08:01.912399+0000","last_clean":"2024-03-24T12:06:22.260451+0000","last_became_active":"2024-03-24T12:06:27.011434+0000","last_became_peered":"2024-03-24T12:06:27.011434+0000","last_unstale":"2024-03-24T12:08:01.912399+0000","last_undegraded":"2024-03-24T12:08:01.912399+0000","last_fullsized":"2024-03-24T12:06:26.996364+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:13.648498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.648498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.313294+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":667,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543910+0000","last_change":"2024-03-24T12:07:32.603069+0000","last_active":"2024-03-24T12:07:58.543910+0000","last_peered":"2024-03-24T12:07:58.543910+0000","last_clean":"2024-03-24T12:07:58.543910+0000","last_became_active":"2024-03-24T12:03:15.298983+0000","last_became_peered":"2024-03-24T12:03:15.298983+0000","last_unstale":"2024-03-24T12:07:58.543910+0000","last_undegraded":"2024-03-24T12:07:58.543910+0000","last_fullsized":"2024-03-24T12:07:58.543910+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:32.602779+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:07:32.602779+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:01.532856+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"340'8492","reported_seq":8539,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.463740+0000","last_change":"2024-03-24T12:06:39.755347+0000","last_active":"2024-03-24T12:08:01.463740+0000","last_peered":"2024-03-24T12:08:01.463740+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:06:30.015883+0000","last_became_peered":"2024-03-24T12:06:30.015883+0000","last_unstale":"2024-03-24T12:08:01.463740+0000","last_undegraded":"2024-03-24T12:06:39.745221+0000","last_fullsized":"2024-03-24T12:06:30.008879+0000","mapping_epoch":320,"log_start":"329'4000","ondisk_log_start":"329'4000","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":556531712,"num_objects":8492,"num_object_clones":0,"num_object_copies":25476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8492,"num_write_kb":543488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":8492}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"83'130","reported_seq":659,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.552508+0000","last_change":"2024-03-24T12:07:27.057073+0000","last_active":"2024-03-24T12:07:58.552508+0000","last_peered":"2024-03-24T12:07:58.552508+0000","last_clean":"2024-03-24T12:07:58.552508+0000","last_became_active":"2024-03-24T12:03:19.827283+0000","last_became_peered":"2024-03-24T12:03:19.827283+0000","last_unstale":"2024-03-24T12:07:58.552508+0000","last_undegraded":"2024-03-24T12:07:58.552508+0000","last_fullsized":"2024-03-24T12:07:58.552508+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:27.056921+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:27.056921+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:30.523758+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":622,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.914370+0000","last_change":"2024-03-24T12:06:43.893998+0000","last_active":"2024-03-24T12:08:01.914370+0000","last_peered":"2024-03-24T12:08:01.914370+0000","last_clean":"2024-03-24T12:08:01.914370+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:08:01.914370+0000","last_undegraded":"2024-03-24T12:08:01.914370+0000","last_fullsized":"2024-03-24T12:08:01.914370+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:43.893854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:43.893854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:11.959417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":664,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534156+0000","last_change":"2024-03-24T12:06:13.937378+0000","last_active":"2024-03-24T12:07:58.534156+0000","last_peered":"2024-03-24T12:07:58.534156+0000","last_clean":"2024-03-24T12:07:58.534156+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:07:58.534156+0000","last_undegraded":"2024-03-24T12:07:58.534156+0000","last_fullsized":"2024-03-24T12:07:58.534156+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:06:13.937275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:13.937275+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":629,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.533919+0000","last_change":"2024-03-24T12:07:45.216727+0000","last_active":"2024-03-24T12:07:58.533919+0000","last_peered":"2024-03-24T12:07:58.533919+0000","last_clean":"2024-03-24T12:07:58.533919+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:07:58.533919+0000","last_undegraded":"2024-03-24T12:07:58.533919+0000","last_fullsized":"2024-03-24T12:07:58.533919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:07:45.216581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.216581+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:09.709448+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":599,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.554245+0000","last_change":"2024-03-24T12:07:47.318834+0000","last_active":"2024-03-24T12:07:58.554245+0000","last_peered":"2024-03-24T12:07:58.554245+0000","last_clean":"2024-03-24T12:07:58.554245+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:07:58.554245+0000","last_undegraded":"2024-03-24T12:07:58.554245+0000","last_fullsized":"2024-03-24T12:07:58.554245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:47.318040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:07:47.318040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:09.655964+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"83'130","reported_seq":483,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.553034+0000","last_change":"2024-03-24T12:06:52.949501+0000","last_active":"2024-03-24T12:07:58.553034+0000","last_peered":"2024-03-24T12:07:58.553034+0000","last_clean":"2024-03-24T12:07:58.553034+0000","last_became_active":"2024-03-24T12:03:19.828385+0000","last_became_peered":"2024-03-24T12:03:19.828385+0000","last_unstale":"2024-03-24T12:07:58.553034+0000","last_undegraded":"2024-03-24T12:07:58.553034+0000","last_fullsized":"2024-03-24T12:07:58.553034+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:52.949381+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:52.949381+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.163619+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":645,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.924370+0000","last_change":"2024-03-24T12:07:26.164276+0000","last_active":"2024-03-24T12:08:01.924370+0000","last_peered":"2024-03-24T12:08:01.924370+0000","last_clean":"2024-03-24T12:08:01.924370+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:08:01.924370+0000","last_undegraded":"2024-03-24T12:08:01.924370+0000","last_fullsized":"2024-03-24T12:08:01.924370+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:07:26.164151+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:07:26.164151+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:35.630219+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":101,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.912700+0000","last_change":"2024-03-24T12:06:51.959021+0000","last_active":"2024-03-24T12:08:01.912700+0000","last_peered":"2024-03-24T12:08:01.912700+0000","last_clean":"2024-03-24T12:08:01.912700+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:08:01.912700+0000","last_undegraded":"2024-03-24T12:08:01.912700+0000","last_fullsized":"2024-03-24T12:08:01.912700+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:51.958902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:51.958902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:05.275402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":595,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.556192+0000","last_change":"2024-03-24T12:06:42.950243+0000","last_active":"2024-03-24T12:07:58.556192+0000","last_peered":"2024-03-24T12:07:58.556192+0000","last_clean":"2024-03-24T12:07:58.556192+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:07:58.556192+0000","last_undegraded":"2024-03-24T12:07:58.556192+0000","last_fullsized":"2024-03-24T12:07:58.556192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.950004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.950004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:10.968575+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":98,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534771+0000","last_change":"2024-03-24T12:07:50.278024+0000","last_active":"2024-03-24T12:07:58.534771+0000","last_peered":"2024-03-24T12:07:58.534771+0000","last_clean":"2024-03-24T12:07:58.534771+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:07:58.534771+0000","last_undegraded":"2024-03-24T12:07:58.534771+0000","last_fullsized":"2024-03-24T12:07:58.534771+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:50.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:50.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.542018+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"83'130","reported_seq":349,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.541572+0000","last_change":"2024-03-24T12:06:27.009499+0000","last_active":"2024-03-24T12:07:58.541572+0000","last_peered":"2024-03-24T12:07:58.541572+0000","last_clean":"2024-03-24T11:52:57.225001+0000","last_became_active":"2024-03-24T12:06:27.009499+0000","last_became_peered":"2024-03-24T12:06:27.009499+0000","last_unstale":"2024-03-24T12:07:58.541572+0000","last_undegraded":"2024-03-24T12:06:27.003936+0000","last_fullsized":"2024-03-24T12:06:27.003811+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:48.080209+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.080209+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:59.317828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":625,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.555986+0000","last_change":"2024-03-24T12:06:49.874361+0000","last_active":"2024-03-24T12:07:58.555986+0000","last_peered":"2024-03-24T12:07:58.555986+0000","last_clean":"2024-03-24T12:07:58.555986+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:07:58.555986+0000","last_undegraded":"2024-03-24T12:07:58.555986+0000","last_fullsized":"2024-03-24T12:07:58.555986+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:06:49.873894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.873894+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":624,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543296+0000","last_change":"2024-03-24T12:06:45.566720+0000","last_active":"2024-03-24T12:07:58.543296+0000","last_peered":"2024-03-24T12:07:58.543296+0000","last_clean":"2024-03-24T12:07:58.543296+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:07:58.543296+0000","last_undegraded":"2024-03-24T12:07:58.543296+0000","last_fullsized":"2024-03-24T12:07:58.543296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:45.566608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:06:45.566608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":52,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.566950+0000","last_change":"2024-03-24T12:06:27.013558+0000","last_active":"2024-03-24T12:07:58.566950+0000","last_peered":"2024-03-24T12:07:58.566950+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.013558+0000","last_became_peered":"2024-03-24T12:06:27.013558+0000","last_unstale":"2024-03-24T12:07:58.566950+0000","last_undegraded":"2024-03-24T12:07:58.566950+0000","last_fullsized":"2024-03-24T12:06:27.000884+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:11.242845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:11.242845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":617,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.530401+0000","last_change":"2024-03-24T12:07:30.073083+0000","last_active":"2024-03-24T12:07:58.530401+0000","last_peered":"2024-03-24T12:07:58.530401+0000","last_clean":"2024-03-24T12:07:58.530401+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:07:58.530401+0000","last_undegraded":"2024-03-24T12:07:58.530401+0000","last_fullsized":"2024-03-24T12:07:58.530401+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.072953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.072953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:57.158345+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"83'130","reported_seq":500,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.912949+0000","last_change":"2024-03-24T12:06:27.004114+0000","last_active":"2024-03-24T12:08:01.912949+0000","last_peered":"2024-03-24T12:08:01.912949+0000","last_clean":"2024-03-24T12:06:22.260475+0000","last_became_active":"2024-03-24T12:06:27.004114+0000","last_became_peered":"2024-03-24T12:06:27.004114+0000","last_unstale":"2024-03-24T12:08:01.912949+0000","last_undegraded":"2024-03-24T12:08:01.912949+0000","last_fullsized":"2024-03-24T12:06:26.996349+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:21.650437+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:21.650437+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.313367+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":642,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.552632+0000","last_change":"2024-03-24T12:06:53.944441+0000","last_active":"2024-03-24T12:07:58.552632+0000","last_peered":"2024-03-24T12:07:58.552632+0000","last_clean":"2024-03-24T12:07:58.552632+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:07:58.552632+0000","last_undegraded":"2024-03-24T12:07:58.552632+0000","last_fullsized":"2024-03-24T12:07:58.552632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:06:53.944281+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:06:53.944281+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.886487+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":51,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.562437+0000","last_change":"2024-03-24T12:06:27.010653+0000","last_active":"2024-03-24T12:07:58.562437+0000","last_peered":"2024-03-24T12:07:58.562437+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.010653+0000","last_became_peered":"2024-03-24T12:06:27.010653+0000","last_unstale":"2024-03-24T12:07:58.562437+0000","last_undegraded":"2024-03-24T12:07:58.562437+0000","last_fullsized":"2024-03-24T12:06:27.000433+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.284540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.284540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"319'9","reported_seq":98,"reported_epoch":341,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:08:01.911075+0000","last_change":"2024-03-24T12:06:27.013138+0000","last_active":"2024-03-24T12:08:01.911075+0000","last_peered":"2024-03-24T12:08:01.911075+0000","last_clean":"2024-03-24T11:52:22.536407+0000","last_became_active":"2024-03-24T12:06:27.013138+0000","last_became_peered":"2024-03-24T12:06:27.013138+0000","last_unstale":"2024-03-24T12:08:01.911075+0000","last_undegraded":"2024-03-24T12:06:26.999373+0000","last_fullsized":"2024-03-24T12:06:26.999255+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:03:51.155727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.155727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.312928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":608,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.924166+0000","last_change":"2024-03-24T12:07:58.405060+0000","last_active":"2024-03-24T12:08:01.924166+0000","last_peered":"2024-03-24T12:08:01.924166+0000","last_clean":"2024-03-24T12:08:01.924166+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:08:01.924166+0000","last_undegraded":"2024-03-24T12:08:01.924166+0000","last_fullsized":"2024-03-24T12:08:01.924166+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:58.404932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.404932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:11.711687+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"83'130","reported_seq":760,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.529553+0000","last_change":"2024-03-24T12:06:27.008690+0000","last_active":"2024-03-24T12:07:58.529553+0000","last_peered":"2024-03-24T12:07:58.529553+0000","last_clean":"2024-03-24T12:03:16.298442+0000","last_became_active":"2024-03-24T12:06:27.008690+0000","last_became_peered":"2024-03-24T12:06:27.008690+0000","last_unstale":"2024-03-24T12:07:58.529553+0000","last_undegraded":"2024-03-24T12:07:58.529553+0000","last_fullsized":"2024-03-24T12:06:27.002281+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:10.347460+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:10.347460+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:59.060668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":463,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.530590+0000","last_change":"2024-03-24T12:06:27.013827+0000","last_active":"2024-03-24T12:07:58.530590+0000","last_peered":"2024-03-24T12:07:58.530590+0000","last_clean":"2024-03-24T12:02:22.991378+0000","last_became_active":"2024-03-24T12:06:27.013827+0000","last_became_peered":"2024-03-24T12:06:27.013827+0000","last_unstale":"2024-03-24T12:07:58.530590+0000","last_undegraded":"2024-03-24T12:06:27.004993+0000","last_fullsized":"2024-03-24T12:06:27.004873+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.043233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":612,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.912724+0000","last_change":"2024-03-24T12:07:45.449912+0000","last_active":"2024-03-24T12:08:01.912724+0000","last_peered":"2024-03-24T12:08:01.912724+0000","last_clean":"2024-03-24T12:08:01.912724+0000","last_became_active":"2024-03-24T11:52:15.135410+0000","last_became_peered":"2024-03-24T11:52:15.135410+0000","last_unstale":"2024-03-24T12:08:01.912724+0000","last_undegraded":"2024-03-24T12:08:01.912724+0000","last_fullsized":"2024-03-24T12:08:01.912724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:45.449624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.449624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:11.077383+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":811,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.556980+0000","last_change":"2024-03-24T12:06:10.014367+0000","last_active":"2024-03-24T12:07:58.556980+0000","last_peered":"2024-03-24T12:07:58.556980+0000","last_clean":"2024-03-24T12:07:58.556980+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:07:58.556980+0000","last_undegraded":"2024-03-24T12:07:58.556980+0000","last_fullsized":"2024-03-24T12:07:58.556980+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:06:10.014224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:10.014224+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":600,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.534261+0000","last_change":"2024-03-24T12:06:27.005026+0000","last_active":"2024-03-24T12:07:58.534261+0000","last_peered":"2024-03-24T12:07:58.534261+0000","last_clean":"2024-03-24T12:06:22.261051+0000","last_became_active":"2024-03-24T12:06:27.005026+0000","last_became_peered":"2024-03-24T12:06:27.005026+0000","last_unstale":"2024-03-24T12:07:58.534261+0000","last_undegraded":"2024-03-24T12:07:58.534261+0000","last_fullsized":"2024-03-24T12:06:27.000977+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.041566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.041566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.322525+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":413,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543395+0000","last_change":"2024-03-24T12:07:38.607052+0000","last_active":"2024-03-24T12:07:58.543395+0000","last_peered":"2024-03-24T12:07:58.543395+0000","last_clean":"2024-03-24T12:07:58.543395+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:07:58.543395+0000","last_undegraded":"2024-03-24T12:07:58.543395+0000","last_fullsized":"2024-03-24T12:07:58.543395+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:38.606914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:38.606914+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:03.328962+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":619,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.543492+0000","last_change":"2024-03-24T12:06:27.008866+0000","last_active":"2024-03-24T12:07:58.543492+0000","last_peered":"2024-03-24T12:07:58.543492+0000","last_clean":"2024-03-24T12:06:22.268390+0000","last_became_active":"2024-03-24T12:06:27.008866+0000","last_became_peered":"2024-03-24T12:06:27.008866+0000","last_unstale":"2024-03-24T12:07:58.543492+0000","last_undegraded":"2024-03-24T12:07:58.543492+0000","last_fullsized":"2024-03-24T12:06:27.000626+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:05.334379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:04:05.334379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":609,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.554157+0000","last_change":"2024-03-24T12:07:29.080020+0000","last_active":"2024-03-24T12:07:58.554157+0000","last_peered":"2024-03-24T12:07:58.554157+0000","last_clean":"2024-03-24T12:07:58.554157+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:07:58.554157+0000","last_undegraded":"2024-03-24T12:07:58.554157+0000","last_fullsized":"2024-03-24T12:07:58.554157+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:29.079435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:07:29.079435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:52.217166+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":603,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.912199+0000","last_change":"2024-03-24T12:06:27.013848+0000","last_active":"2024-03-24T12:08:01.912199+0000","last_peered":"2024-03-24T12:08:01.912199+0000","last_clean":"2024-03-24T12:06:23.013980+0000","last_became_active":"2024-03-24T12:06:27.013848+0000","last_became_peered":"2024-03-24T12:06:27.013848+0000","last_unstale":"2024-03-24T12:08:01.912199+0000","last_undegraded":"2024-03-24T12:08:01.912199+0000","last_fullsized":"2024-03-24T12:06:27.006706+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:22.691279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T12:05:22.691279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.313354+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":636,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.533443+0000","last_change":"2024-03-24T12:06:50.625611+0000","last_active":"2024-03-24T12:07:58.533443+0000","last_peered":"2024-03-24T12:07:58.533443+0000","last_clean":"2024-03-24T12:07:58.533443+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:07:58.533443+0000","last_undegraded":"2024-03-24T12:07:58.533443+0000","last_fullsized":"2024-03-24T12:07:58.533443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:06:50.625428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:50.625428+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":629,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.537383+0000","last_change":"2024-03-24T12:06:27.015745+0000","last_active":"2024-03-24T12:07:58.537383+0000","last_peered":"2024-03-24T12:07:58.537383+0000","last_clean":"2024-03-24T12:06:21.984264+0000","last_became_active":"2024-03-24T12:06:27.015745+0000","last_became_peered":"2024-03-24T12:06:27.015745+0000","last_unstale":"2024-03-24T12:07:58.537383+0000","last_undegraded":"2024-03-24T12:07:58.537383+0000","last_fullsized":"2024-03-24T12:06:27.004103+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:13.110170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:13.110170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.283485+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":54,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.562660+0000","last_change":"2024-03-24T12:06:27.010505+0000","last_active":"2024-03-24T12:07:58.562660+0000","last_peered":"2024-03-24T12:07:58.562660+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.010505+0000","last_became_peered":"2024-03-24T12:06:27.010505+0000","last_unstale":"2024-03-24T12:07:58.562660+0000","last_undegraded":"2024-03-24T12:07:58.562660+0000","last_fullsized":"2024-03-24T12:06:27.000808+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:10.080832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:03:10.080832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":594,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.554846+0000","last_change":"2024-03-24T12:06:27.006336+0000","last_active":"2024-03-24T12:07:58.554846+0000","last_peered":"2024-03-24T12:07:58.554846+0000","last_clean":"2024-03-24T12:06:22.633690+0000","last_became_active":"2024-03-24T12:06:27.006336+0000","last_became_peered":"2024-03-24T12:06:27.006336+0000","last_unstale":"2024-03-24T12:07:58.554846+0000","last_undegraded":"2024-03-24T12:07:58.554846+0000","last_fullsized":"2024-03-24T12:06:26.998332+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:25.928874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.928874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":629,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.552710+0000","last_change":"2024-03-24T12:07:56.051237+0000","last_active":"2024-03-24T12:07:58.552710+0000","last_peered":"2024-03-24T12:07:58.552710+0000","last_clean":"2024-03-24T12:07:58.552710+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:07:58.552710+0000","last_undegraded":"2024-03-24T12:07:58.552710+0000","last_fullsized":"2024-03-24T12:07:58.552710+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:07:56.051061+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.051061+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:23.143128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":570,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.543498+0000","last_change":"2024-03-24T12:06:27.011102+0000","last_active":"2024-03-24T12:07:58.543498+0000","last_peered":"2024-03-24T12:07:58.543498+0000","last_clean":"2024-03-24T12:05:37.533531+0000","last_became_active":"2024-03-24T12:06:27.011102+0000","last_became_peered":"2024-03-24T12:06:27.011102+0000","last_unstale":"2024-03-24T12:07:58.543498+0000","last_undegraded":"2024-03-24T12:07:58.543498+0000","last_fullsized":"2024-03-24T12:06:27.001860+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:18.028433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:18.028433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":616,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543272+0000","last_change":"2024-03-24T12:06:42.538679+0000","last_active":"2024-03-24T12:07:58.543272+0000","last_peered":"2024-03-24T12:07:58.543272+0000","last_clean":"2024-03-24T12:07:58.543272+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:07:58.543272+0000","last_undegraded":"2024-03-24T12:07:58.543272+0000","last_fullsized":"2024-03-24T12:07:58.543272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.538568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.538568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:59.350306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":606,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.530136+0000","last_change":"2024-03-24T12:06:39.164062+0000","last_active":"2024-03-24T12:07:58.530136+0000","last_peered":"2024-03-24T12:07:58.530136+0000","last_clean":"2024-03-24T12:07:58.530136+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:07:58.530136+0000","last_undegraded":"2024-03-24T12:07:58.530136+0000","last_fullsized":"2024-03-24T12:07:58.530136+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:39.163951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.163951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"83'130","reported_seq":666,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.914855+0000","last_change":"2024-03-24T12:07:43.871271+0000","last_active":"2024-03-24T12:08:01.914855+0000","last_peered":"2024-03-24T12:08:01.914855+0000","last_clean":"2024-03-24T12:08:01.914855+0000","last_became_active":"2024-03-24T12:03:19.819346+0000","last_became_peered":"2024-03-24T12:03:19.819346+0000","last_unstale":"2024-03-24T12:08:01.914855+0000","last_undegraded":"2024-03-24T12:08:01.914855+0000","last_fullsized":"2024-03-24T12:08:01.914855+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:43.871116+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:43.871116+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:59.932486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":630,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.555472+0000","last_change":"2024-03-24T12:06:13.055590+0000","last_active":"2024-03-24T12:07:58.555472+0000","last_peered":"2024-03-24T12:07:58.555472+0000","last_clean":"2024-03-24T12:07:58.555472+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:07:58.555472+0000","last_undegraded":"2024-03-24T12:07:58.555472+0000","last_fullsized":"2024-03-24T12:07:58.555472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:06:13.055489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:13.055489+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":617,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.544025+0000","last_change":"2024-03-24T12:06:27.005897+0000","last_active":"2024-03-24T12:07:58.544025+0000","last_peered":"2024-03-24T12:07:58.544025+0000","last_clean":"2024-03-24T12:06:22.267726+0000","last_became_active":"2024-03-24T12:06:27.005897+0000","last_became_peered":"2024-03-24T12:06:27.005897+0000","last_unstale":"2024-03-24T12:07:58.544025+0000","last_undegraded":"2024-03-24T12:07:58.544025+0000","last_fullsized":"2024-03-24T12:06:27.001669+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.372798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.372798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":618,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.529972+0000","last_change":"2024-03-24T12:07:55.049337+0000","last_active":"2024-03-24T12:07:58.529972+0000","last_peered":"2024-03-24T12:07:58.529972+0000","last_clean":"2024-03-24T12:07:58.529972+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:07:58.529972+0000","last_undegraded":"2024-03-24T12:07:58.529972+0000","last_fullsized":"2024-03-24T12:07:58.529972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:55.049210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:07:55.049210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:58.773743+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":591,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.554331+0000","last_change":"2024-03-24T12:06:18.013886+0000","last_active":"2024-03-24T12:07:58.554331+0000","last_peered":"2024-03-24T12:07:58.554331+0000","last_clean":"2024-03-24T12:07:58.554331+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:07:58.554331+0000","last_undegraded":"2024-03-24T12:07:58.554331+0000","last_fullsized":"2024-03-24T12:07:58.554331+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:18.013768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:18.013768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":652,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.924567+0000","last_change":"2024-03-24T12:07:51.268007+0000","last_active":"2024-03-24T12:08:01.924567+0000","last_peered":"2024-03-24T12:08:01.924567+0000","last_clean":"2024-03-24T12:08:01.924567+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:08:01.924567+0000","last_undegraded":"2024-03-24T12:08:01.924567+0000","last_fullsized":"2024-03-24T12:08:01.924567+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:51.267854+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.267854+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:03.779559+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":51,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.533200+0000","last_change":"2024-03-24T12:06:27.007241+0000","last_active":"2024-03-24T12:07:58.533200+0000","last_peered":"2024-03-24T12:07:58.533200+0000","last_clean":"2024-03-24T11:52:14.117087+0000","last_became_active":"2024-03-24T12:06:27.007241+0000","last_became_peered":"2024-03-24T12:06:27.007241+0000","last_unstale":"2024-03-24T12:07:58.533200+0000","last_undegraded":"2024-03-24T12:07:58.533200+0000","last_fullsized":"2024-03-24T12:06:27.000285+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.160982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.160982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.322474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":819,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.544276+0000","last_change":"2024-03-24T12:07:51.843434+0000","last_active":"2024-03-24T12:07:58.544276+0000","last_peered":"2024-03-24T12:07:58.544276+0000","last_clean":"2024-03-24T12:07:58.544276+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:07:58.544276+0000","last_undegraded":"2024-03-24T12:07:58.544276+0000","last_fullsized":"2024-03-24T12:07:58.544276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:07:51.843305+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.843305+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:20.495680+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":619,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.531322+0000","last_change":"2024-03-24T12:06:27.009504+0000","last_active":"2024-03-24T12:07:58.531322+0000","last_peered":"2024-03-24T12:07:58.531322+0000","last_clean":"2024-03-24T12:06:22.145780+0000","last_became_active":"2024-03-24T12:06:27.009504+0000","last_became_peered":"2024-03-24T12:06:27.009504+0000","last_unstale":"2024-03-24T12:07:58.531322+0000","last_undegraded":"2024-03-24T12:06:27.002365+0000","last_fullsized":"2024-03-24T12:06:27.002251+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:05:20.119650+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:20.119650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.043218+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":655,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.544266+0000","last_change":"2024-03-24T12:06:27.007431+0000","last_active":"2024-03-24T12:07:58.544266+0000","last_peered":"2024-03-24T12:07:58.544266+0000","last_clean":"2024-03-24T12:06:22.268056+0000","last_became_active":"2024-03-24T12:06:27.007431+0000","last_became_peered":"2024-03-24T12:06:27.007431+0000","last_unstale":"2024-03-24T12:07:58.544266+0000","last_undegraded":"2024-03-24T12:06:27.002523+0000","last_fullsized":"2024-03-24T12:06:27.002390+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:05:25.602806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.602806+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":604,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.555117+0000","last_change":"2024-03-24T12:06:40.021255+0000","last_active":"2024-03-24T12:07:58.555117+0000","last_peered":"2024-03-24T12:07:58.555117+0000","last_clean":"2024-03-24T12:07:58.555117+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:07:58.555117+0000","last_undegraded":"2024-03-24T12:07:58.555117+0000","last_fullsized":"2024-03-24T12:07:58.555117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:06:40.021114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:40.021114+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":619,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.924733+0000","last_change":"2024-03-24T12:07:36.333159+0000","last_active":"2024-03-24T12:08:01.924733+0000","last_peered":"2024-03-24T12:08:01.924733+0000","last_clean":"2024-03-24T12:08:01.924733+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:08:01.924733+0000","last_undegraded":"2024-03-24T12:08:01.924733+0000","last_fullsized":"2024-03-24T12:08:01.924733+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:36.333044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:07:36.333044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:48.766442+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.533563+0000","last_change":"2024-03-24T12:06:27.011917+0000","last_active":"2024-03-24T12:07:58.533563+0000","last_peered":"2024-03-24T12:07:58.533563+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.011917+0000","last_became_peered":"2024-03-24T12:06:27.011917+0000","last_unstale":"2024-03-24T12:07:58.533563+0000","last_undegraded":"2024-03-24T12:07:58.533563+0000","last_fullsized":"2024-03-24T12:06:27.001875+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:04:14.188600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.322489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":561,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.557247+0000","last_change":"2024-03-24T12:07:42.113052+0000","last_active":"2024-03-24T12:07:58.557247+0000","last_peered":"2024-03-24T12:07:58.557247+0000","last_clean":"2024-03-24T12:07:58.557247+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:07:58.557247+0000","last_undegraded":"2024-03-24T12:07:58.557247+0000","last_fullsized":"2024-03-24T12:07:58.557247+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:07:42.112569+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:07:42.112569+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:47.493851+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":624,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.543968+0000","last_change":"2024-03-24T12:06:57.653082+0000","last_active":"2024-03-24T12:07:58.543968+0000","last_peered":"2024-03-24T12:07:58.543968+0000","last_clean":"2024-03-24T12:07:58.543968+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:07:58.543968+0000","last_undegraded":"2024-03-24T12:07:58.543968+0000","last_fullsized":"2024-03-24T12:07:58.543968+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:57.653004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:06:57.653004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:27.542506+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":370,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534434+0000","last_change":"2024-03-24T12:07:53.354298+0000","last_active":"2024-03-24T12:07:58.534434+0000","last_peered":"2024-03-24T12:07:58.534434+0000","last_clean":"2024-03-24T12:07:58.534434+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:07:58.534434+0000","last_undegraded":"2024-03-24T12:07:58.534434+0000","last_fullsized":"2024-03-24T12:07:58.534434+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:07:53.354213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:14.721998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":604,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.534588+0000","last_change":"2024-03-24T12:06:27.008492+0000","last_active":"2024-03-24T12:07:58.534588+0000","last_peered":"2024-03-24T12:07:58.534588+0000","last_clean":"2024-03-24T12:06:22.267344+0000","last_became_active":"2024-03-24T12:06:27.008492+0000","last_became_peered":"2024-03-24T12:06:27.008492+0000","last_unstale":"2024-03-24T12:07:58.534588+0000","last_undegraded":"2024-03-24T12:07:58.534588+0000","last_fullsized":"2024-03-24T12:06:27.002393+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:53.809021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:53.809021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.322514+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":662,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.556446+0000","last_change":"2024-03-24T12:07:41.142569+0000","last_active":"2024-03-24T12:07:58.556446+0000","last_peered":"2024-03-24T12:07:58.556446+0000","last_clean":"2024-03-24T12:07:58.556446+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:07:58.556446+0000","last_undegraded":"2024-03-24T12:07:58.556446+0000","last_fullsized":"2024-03-24T12:07:58.556446+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:07:41.142421+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.142421+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:00.469591+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":617,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.557644+0000","last_change":"2024-03-24T12:07:32.002134+0000","last_active":"2024-03-24T12:07:58.557644+0000","last_peered":"2024-03-24T12:07:58.557644+0000","last_clean":"2024-03-24T12:07:58.557644+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:07:58.557644+0000","last_undegraded":"2024-03-24T12:07:58.557644+0000","last_fullsized":"2024-03-24T12:07:58.557644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:07:32.001718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:32.001718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:33.487127+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":607,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.534809+0000","last_change":"2024-03-24T12:06:42.722854+0000","last_active":"2024-03-24T12:07:58.534809+0000","last_peered":"2024-03-24T12:07:58.534809+0000","last_clean":"2024-03-24T12:07:58.534809+0000","last_became_active":"2024-03-24T12:05:44.736383+0000","last_became_peered":"2024-03-24T12:05:44.736383+0000","last_unstale":"2024-03-24T12:07:58.534809+0000","last_undegraded":"2024-03-24T12:07:58.534809+0000","last_fullsized":"2024-03-24T12:07:58.534809+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.722733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.722733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:10.779416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":595,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.553085+0000","last_change":"2024-03-24T12:06:27.005514+0000","last_active":"2024-03-24T12:07:58.553085+0000","last_peered":"2024-03-24T12:07:58.553085+0000","last_clean":"2024-03-24T12:06:21.983027+0000","last_became_active":"2024-03-24T12:06:27.005514+0000","last_became_peered":"2024-03-24T12:06:27.005514+0000","last_unstale":"2024-03-24T12:07:58.553085+0000","last_undegraded":"2024-03-24T12:07:58.553085+0000","last_fullsized":"2024-03-24T12:06:26.999436+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:19.844413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:19.844413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045627+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":638,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.533685+0000","last_change":"2024-03-24T12:07:40.191735+0000","last_active":"2024-03-24T12:07:58.533685+0000","last_peered":"2024-03-24T12:07:58.533685+0000","last_clean":"2024-03-24T12:07:58.533685+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:07:58.533685+0000","last_undegraded":"2024-03-24T12:07:58.533685+0000","last_fullsized":"2024-03-24T12:07:58.533685+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:40.191513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.191513+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:01.275179+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":624,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.530793+0000","last_change":"2024-03-24T12:06:27.009141+0000","last_active":"2024-03-24T12:07:58.530793+0000","last_peered":"2024-03-24T12:07:58.530793+0000","last_clean":"2024-03-24T12:06:22.145387+0000","last_became_active":"2024-03-24T12:06:27.009141+0000","last_became_peered":"2024-03-24T12:06:27.009141+0000","last_unstale":"2024-03-24T12:07:58.530793+0000","last_undegraded":"2024-03-24T12:07:58.530793+0000","last_fullsized":"2024-03-24T12:06:27.001371+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.177509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.043201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":54,"reported_epoch":341,"state":"active+undersized","last_fresh":"2024-03-24T12:08:01.912085+0000","last_change":"2024-03-24T12:06:27.009488+0000","last_active":"2024-03-24T12:08:01.912085+0000","last_peered":"2024-03-24T12:08:01.912085+0000","last_clean":"2024-03-24T11:52:18.347513+0000","last_became_active":"2024-03-24T12:06:27.009488+0000","last_became_peered":"2024-03-24T12:06:27.009488+0000","last_unstale":"2024-03-24T12:08:01.912085+0000","last_undegraded":"2024-03-24T12:08:01.912085+0000","last_fullsized":"2024-03-24T12:06:26.998029+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.099551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.099551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.312912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":600,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.913482+0000","last_change":"2024-03-24T12:07:46.478165+0000","last_active":"2024-03-24T12:08:01.913482+0000","last_peered":"2024-03-24T12:08:01.913482+0000","last_clean":"2024-03-24T12:08:01.913482+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:08:01.913482+0000","last_undegraded":"2024-03-24T12:08:01.913482+0000","last_fullsized":"2024-03-24T12:08:01.913482+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:46.478049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:07:46.478049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:51.084877+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":628,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.533608+0000","last_change":"2024-03-24T12:06:56.609759+0000","last_active":"2024-03-24T12:07:58.533608+0000","last_peered":"2024-03-24T12:07:58.533608+0000","last_clean":"2024-03-24T12:07:58.533608+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:07:58.533608+0000","last_undegraded":"2024-03-24T12:07:58.533608+0000","last_fullsized":"2024-03-24T12:07:58.533608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:06:56.609621+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:06:56.609621+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:22.054130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":634,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.530451+0000","last_change":"2024-03-24T12:07:58.109759+0000","last_active":"2024-03-24T12:07:58.530451+0000","last_peered":"2024-03-24T12:07:58.530451+0000","last_clean":"2024-03-24T12:07:58.530451+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:07:58.530451+0000","last_undegraded":"2024-03-24T12:07:58.530451+0000","last_fullsized":"2024-03-24T12:07:58.530451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:58.109623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.109623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:15.562400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":626,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.529902+0000","last_change":"2024-03-24T12:07:40.020936+0000","last_active":"2024-03-24T12:07:58.529902+0000","last_peered":"2024-03-24T12:07:58.529902+0000","last_clean":"2024-03-24T12:07:58.529902+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:07:58.529902+0000","last_undegraded":"2024-03-24T12:07:58.529902+0000","last_fullsized":"2024-03-24T12:07:58.529902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:40.020836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.020836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:47.183576+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.537606+0000","last_change":"2024-03-24T12:06:27.013806+0000","last_active":"2024-03-24T12:07:58.537606+0000","last_peered":"2024-03-24T12:07:58.537606+0000","last_clean":"2024-03-24T11:52:20.361608+0000","last_became_active":"2024-03-24T12:06:27.013806+0000","last_became_peered":"2024-03-24T12:06:27.013806+0000","last_unstale":"2024-03-24T12:07:58.537606+0000","last_undegraded":"2024-03-24T12:07:58.537606+0000","last_fullsized":"2024-03-24T12:06:27.001624+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.117875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.117875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.283469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":642,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.533120+0000","last_change":"2024-03-24T12:07:51.280566+0000","last_active":"2024-03-24T12:07:58.533120+0000","last_peered":"2024-03-24T12:07:58.533120+0000","last_clean":"2024-03-24T12:07:58.533120+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:07:58.533120+0000","last_undegraded":"2024-03-24T12:07:58.533120+0000","last_fullsized":"2024-03-24T12:07:58.533120+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:07:51.280424+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.280424+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:53.648699+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":607,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.567133+0000","last_change":"2024-03-24T12:06:27.011373+0000","last_active":"2024-03-24T12:07:58.567133+0000","last_peered":"2024-03-24T12:07:58.567133+0000","last_clean":"2024-03-24T12:06:21.982614+0000","last_became_active":"2024-03-24T12:06:27.011373+0000","last_became_peered":"2024-03-24T12:06:27.011373+0000","last_unstale":"2024-03-24T12:07:58.567133+0000","last_undegraded":"2024-03-24T12:07:58.567133+0000","last_fullsized":"2024-03-24T12:06:26.998012+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:55.731711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:03:55.731711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045700+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":84,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.543089+0000","last_change":"2024-03-24T12:06:27.007261+0000","last_active":"2024-03-24T12:07:58.543089+0000","last_peered":"2024-03-24T12:07:58.543089+0000","last_clean":"2024-03-24T12:06:22.261369+0000","last_became_active":"2024-03-24T12:06:27.007261+0000","last_became_peered":"2024-03-24T12:06:27.007261+0000","last_unstale":"2024-03-24T12:07:58.543089+0000","last_undegraded":"2024-03-24T12:07:58.543089+0000","last_fullsized":"2024-03-24T12:06:26.999466+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.683700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.683700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":600,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-24T12:07:58.533498+0000","last_change":"2024-03-24T12:06:27.012256+0000","last_active":"2024-03-24T12:07:58.533498+0000","last_peered":"2024-03-24T12:07:58.533498+0000","last_clean":"2024-03-24T12:06:22.261185+0000","last_became_active":"2024-03-24T12:06:27.012256+0000","last_became_peered":"2024-03-24T12:06:27.012256+0000","last_unstale":"2024-03-24T12:07:58.533498+0000","last_undegraded":"2024-03-24T12:07:58.533498+0000","last_fullsized":"2024-03-24T12:06:26.999262+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.241824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.241824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.322501+0000","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":674,"reported_epoch":340,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:07:58.555573+0000","last_change":"2024-03-24T12:06:27.007085+0000","last_active":"2024-03-24T12:07:58.555573+0000","last_peered":"2024-03-24T12:07:58.555573+0000","last_clean":"2024-03-24T12:06:21.988852+0000","last_became_active":"2024-03-24T12:06:27.007085+0000","last_became_peered":"2024-03-24T12:06:27.007085+0000","last_unstale":"2024-03-24T12:07:58.555573+0000","last_undegraded":"2024-03-24T12:06:27.000133+0000","last_fullsized":"2024-03-24T12:06:26.999091+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:04:50.623172+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:04:50.623172+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:01.045729+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":613,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-24T12:08:01.910540+0000","last_change":"2024-03-24T12:07:30.248462+0000","last_active":"2024-03-24T12:08:01.910540+0000","last_peered":"2024-03-24T12:08:01.910540+0000","last_clean":"2024-03-24T12:08:01.910540+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:08:01.910540+0000","last_undegraded":"2024-03-24T12:08:01.910540+0000","last_fullsized":"2024-03-24T12:08:01.910540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.248337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.248337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:52.482002+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":73,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.552540+0000","last_change":"2024-03-24T12:05:56.863247+0000","last_active":"2024-03-24T12:07:58.552540+0000","last_peered":"2024-03-24T12:07:58.552540+0000","last_clean":"2024-03-24T12:07:58.552540+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:07:58.552540+0000","last_undegraded":"2024-03-24T12:07:58.552540+0000","last_fullsized":"2024-03-24T12:07:58.552540+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:05:12.732801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":599,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-24T12:07:58.553136+0000","last_change":"2024-03-24T12:07:48.265595+0000","last_active":"2024-03-24T12:07:58.553136+0000","last_peered":"2024-03-24T12:07:58.553136+0000","last_clean":"2024-03-24T12:07:58.553136+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:07:58.553136+0000","last_undegraded":"2024-03-24T12:07:58.553136+0000","last_fullsized":"2024-03-24T12:07:58.553136+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:48.265295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:48.265295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:03.912096+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":8875278336,"num_objects":135426,"num_object_clones":0,"num_object_copies":406278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135426,"num_write_kb":8667264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23337238528,"data_stored":23337238528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75419,"ondisk_log_size":75419,"up":42,"acting":42,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":488689,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":80,"num_write":49,"num_write_kb":658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":483028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":52,"acting":52,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":138,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":58,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":783,"ondisk_log_size":783,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005249,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2770420,"kb_used_data":2157548,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":90974732,"statfs":{"total":95995035648,"available":93158125568,"internally_reserved":0,"allocated":2209329152,"data_stored":2208846947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":627571146},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332296,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4338884,"kb_used_data":3757948,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":89406268,"statfs":{"total":95995035648,"available":91552018432,"internally_reserved":0,"allocated":3848138752,"data_stored":3847640449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2261244,"kb_used_data":1616436,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91483908,"statfs":{"total":95995035648,"available":93679521792,"internally_reserved":0,"allocated":1655230464,"data_stored":1654739464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738598,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6109452,"kb_used_data":5442432,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":87635700,"statfs":{"total":95995035648,"available":89738956800,"internally_reserved":0,"allocated":5573050368,"data_stored":5572554083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411548,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2131608,"kb_used_data":1635344,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91613544,"statfs":{"total":95995035648,"available":93812269056,"internally_reserved":0,"allocated":1674592256,"data_stored":1674137994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117206,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3415760,"kb_used_data":2712648,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":90329392,"statfs":{"total":95995035648,"available":92497297408,"internally_reserved":0,"allocated":2777751552,"data_stored":2777241466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":719976903},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920977,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5951232,"kb_used_data":5378616,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":87793920,"statfs":{"total":95995035648,"available":89900974080,"internally_reserved":0,"allocated":5507702784,"data_stored":5507207723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659320,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":573604,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93171548,"statfs":{"total":95995035648,"available":95407665152,"internally_reserved":0,"allocated":1007616,"data_stored":637309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5621219328,"data_stored":5621219328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1661927424,"data_stored":1661927424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1679163392,"data_stored":1679163392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2776432640,"data_stored":2776432640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5511512064,"data_stored":5511512064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3870031872,"data_stored":3870031872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2216951808,"data_stored":2216951808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:08:04.607 INFO:tasks.thrashosds.thrasher:Setting 2.13 to [6, 3, 0] 2024-03-24T12:08:04.607 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.13', '6', '3', '0'] 2024-03-24T12:08:04.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 2.13 6 3 0 2024-03-24T12:08:04.624 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:04.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:04.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:04.759 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 85 16 138751 138735 101.981 71.5625 0.0136581 0.00980052 2024-03-24T12:08:04.760 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:04.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:04.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:04 smithi064 ceph-mon[30223]: pgmap v717: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 50455/406872 objects degraded (12.401%) 2024-03-24T12:08:04.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:04 smithi064 ceph-mon[30223]: from='client.15792 v1:172.21.15.64:0/2385921180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:08:04.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:04 smithi064 ceph-mon[42243]: pgmap v717: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 50455/406872 objects degraded (12.401%) 2024-03-24T12:08:04.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:04 smithi064 ceph-mon[42243]: from='client.15792 v1:172.21.15.64:0/2385921180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:08:04.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:04 smithi193 ceph-mon[33593]: pgmap v717: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 50455/406872 objects degraded (12.401%) 2024-03-24T12:08:04.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:04 smithi193 ceph-mon[33593]: from='client.15792 v1:172.21.15.64:0/2385921180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:08:04.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:04.891 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:04.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:04.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:05.036 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:05.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:05.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:05.200 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:05.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:05.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:05.342 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:05.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:05.443 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:05.477 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:05.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:05.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:05.614 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:05.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:05.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:05.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:05.759 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 86 16 139921 139905 101.646 73.125 0.0103124 0.00983317 2024-03-24T12:08:05.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:05.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:06.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:06.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:06.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:06.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:06.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:06.287 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:08:06.316 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:06.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:06.328 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:08:06.369 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:08:06.410 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:08:06.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:06.446 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:06.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:06.453 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:08:06.495 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:08:06.535 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:08:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:08:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:08:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:08:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:08:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:08:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:08:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:08:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:08:06.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:06.585 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:06.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:06.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:06.723 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:06.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:06.759 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 87 16 141063 141047 101.298 71.375 0.0078885 0.00986745 2024-03-24T12:08:06.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:06.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:06.887 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:06 smithi064 ceph-mon[30223]: pgmap v718: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s; 50455/406872 objects degraded (12.401%) 2024-03-24T12:08:06.887 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:06 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2831093038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:06.888 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:06 smithi064 ceph-mon[42243]: pgmap v718: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s; 50455/406872 objects degraded (12.401%) 2024-03-24T12:08:06.888 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:06 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2831093038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:06.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:07.015 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:07.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:07.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:06 smithi193 ceph-mon[33593]: pgmap v718: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s; 50455/406872 objects degraded (12.401%) 2024-03-24T12:08:07.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:06 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2831093038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:07.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:07.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:07.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:07.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:07.402 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:07.434 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:07.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:07.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:07.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:07.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:07.669 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:07.700 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:07.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:07.760 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 88 16 142254 142238 100.992 74.4375 0.0111022 0.00989729 2024-03-24T12:08:07.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:07 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 50455/406872 objects degraded (12.401%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:07.760 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:08:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:08:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:07.761 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:07 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 50455/406872 objects degraded (12.401%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:07.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:08:07.808 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:08:07.842 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:07.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:07.844 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:07.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:07.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:08.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:07 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 50455/406872 objects degraded (12.401%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:08.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:08:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:08:08.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:08.124 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:08.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:08.225 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:08.250 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:08.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:08.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:08.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:08.388 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:08.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:08.490 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:08.523 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:08.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:08.625 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:08.653 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:08.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:08.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:08.760 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 89 16 143347 143331 100.625 68.3125 0.0121617 0.00993251 2024-03-24T12:08:08.784 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:08.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:08.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:08.932 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:08.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:09.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:08 smithi064 ceph-mon[30223]: pgmap v719: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 52170/420930 objects degraded (12.394%) 2024-03-24T12:08:09.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:08 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:09.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:08 smithi064 ceph-mon[42243]: pgmap v719: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 52170/420930 objects degraded (12.394%) 2024-03-24T12:08:09.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:08 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:09.033 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:09.060 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:09.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:09.095 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:08:09.095 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:08:09.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:72990 8.a 8:5a5d6a33:::benchmark_data_smithi064_120708_object72989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.012505+0000", 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 507.079538329, 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14825962200000001, 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 72990 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.096 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012505+0000", 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012505+0000", 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012507+0000", 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1900000000000002e-06 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012529+0000", 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1361000000000001e-05 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012545+0000", 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6024000000000001e-05 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012553+0000", 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4349999999999997e-06 2024-03-24T12:08:09.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012614+0000", 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0362999999999998e-05 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.012877+0000", 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263007 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.013019+0000", 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014215199999999999 2024-03-24T12:08:09.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.018482+0000", 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054628760000000002 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.018522+0000", 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0413000000000001e-05 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.160670+0000", 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14214798300000001 2024-03-24T12:08:09.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.160731+0000", 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1137999999999996e-05 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.160765+0000", 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3680000000000003e-05 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:72992 8.a 8:558bd69d:::benchmark_data_smithi064_120708_object72991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.015224+0000", 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 507.07681996700001, 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14865694300000001, 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 72992 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015224+0000", 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015224+0000", 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015226+0000", 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9089999999999998e-06 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015249+0000", 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3125999999999998e-05 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015259+0000", 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0322999999999999e-05 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015267+0000", 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5270000000000001e-06 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015702+0000", 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043556900000000002 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015829+0000", 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127119 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.015993+0000", 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016342900000000001 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.019252+0000", 2024-03-24T12:08:09.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032590649999999998 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.027970+0000", 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087185219999999994 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163751+0000", 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13578027500000001 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163809+0000", 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8662999999999997e-05 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163881+0000", 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1415999999999995e-05 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:73001 8.a 8:54eaffae:::benchmark_data_smithi064_120708_object73000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.020718+0000", 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 507.07132560600002, 2024-03-24T12:08:09.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14328966200000001, 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73001 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020718+0000", 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020718+0000", 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020720+0000", 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3530000000000002e-06 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020744+0000", 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3294000000000001e-05 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020757+0000", 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.306e-05 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020764+0000", 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4089999999999997e-06 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020808+0000", 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3523999999999999e-05 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.020930+0000", 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012192 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.021097+0000", 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016748500000000001 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.027982+0000", 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068853250000000003 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.028019+0000", 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6984999999999999e-05 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163926+0000", 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.135906215 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.163972+0000", 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6162999999999998e-05 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.164008+0000", 2024-03-24T12:08:09.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5929000000000001e-05 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:73008 8.a 8:545958b9:::benchmark_data_smithi064_120708_object73007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.024320+0000", 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 507.067723925, 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.142947031, 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73008 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.113 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024320+0000", 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024320+0000", 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024322+0000", 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.012e-06 2024-03-24T12:08:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024345+0000", 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3402000000000001e-05 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024359+0000", 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4269e-05 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024367+0000", 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0870000000000007e-06 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024419+0000", 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.198e-05 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024544+0000", 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012440800000000001 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.024675+0000", 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013087600000000001 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.028147+0000", 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003472691 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.029846+0000", 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016989259999999999 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.167145+0000", 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137298485 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.167218+0000", 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3440000000000002e-05 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.167267+0000", 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8455000000000003e-05 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:73013 8.a 8:5ddefa19:::benchmark_data_smithi064_120708_object73012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T11:59:42.030757+0000", 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 507.06128682600001, 2024-03-24T12:08:09.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13935418199999999, 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73013 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030757+0000", 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030757+0000", 2024-03-24T12:08:09.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030759+0000", 2024-03-24T12:08:09.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2299999999999998e-06 2024-03-24T12:08:09.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030782+0000", 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3312999999999999e-05 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030794+0000", 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2130000000000001e-05 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030802+0000", 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.841e-06 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030857+0000", 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4812000000000003e-05 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.030982+0000", 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012492499999999999 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,5", 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.031123+0000", 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140987 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.033678+0000", 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025544970000000002 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.035597+0000", 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001919104 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.169997+0000", 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13440000399999999 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.170056+0000", 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9839000000000001e-05 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T11:59:42.170111+0000", 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-05 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200634 9.2 e296/246 9:4deab3a2:::benchmark_data_smithi064_148274_object74821:head v 296'12490, mlcod=296'12487)", 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.84452440199999, 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.1099523, 2024-03-24T12:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247521+0000", 2024-03-24T12:08:09.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8390000000000001e-06 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247528+0000", 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.374e-06 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247540+0000", 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1324e-05 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247548+0000", 2024-03-24T12:08:09.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3110000000000006e-06 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.248032+0000", 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000483993 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.248052+0000", 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0441999999999998e-05 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357431+0000", 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10937805 2024-03-24T12:08:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357472+0000", 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0967000000000003e-05 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200636 9.0 e296/246 9:059695cb:::benchmark_data_smithi064_148274_object74823:head v 296'12488, mlcod=296'12486)", 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.842005124, 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12092386400000001, 2024-03-24T12:08:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250040+0000", 2024-03-24T12:08:09.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250050+0000", 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7540000000000001e-06 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346892+0000", 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096842284000000001 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346904+0000", 2024-03-24T12:08:09.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1429e-05 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.349515+0000", 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026114250000000001 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.349570+0000", 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5211999999999999e-05 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.370781+0000", 2024-03-24T12:08:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021210360000000001 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.370962+0000", 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001818 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200640 9.6 e296/246 9:66c6e6e1:::benchmark_data_smithi064_148274_object74827:head v 296'12828, mlcod=296'12826)", 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.839622077, 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10521612800000001, 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252423+0000", 2024-03-24T12:08:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.575e-06 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252432+0000", 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9539999999999993e-06 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346205+0000", 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093772446999999995 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346228+0000", 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3819999999999999e-05 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.348705+0000", 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024768310000000001 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.348731+0000", 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6122000000000002e-05 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357526+0000", 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087951639999999994 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357638+0000", 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111215 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.24379.0:3048 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94338803870976 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:06:26.485518+0000", 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 102.60652557500001, 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52480097400000003, 2024-03-24T12:08:09.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.24379", 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3048 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485518+0000", 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485518+0000", 2024-03-24T12:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485520+0000", 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485527+0000", 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0299999999999999e-06 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485538+0000", 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1627000000000001e-05 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485549+0000", 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0404999999999999e-05 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485629+0000", 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0885999999999998e-05 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:09.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485640+0000", 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0437e-05 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:27.008076+0000", 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52243590100000004 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:27.008259+0000", 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018303300000000001 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:27.010319+0000", 2024-03-24T12:08:09.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020602189999999999 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:74458 10.5 10:a4cf6209:::benchmark_data_smithi064_174908_object74457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:16.720882+0000", 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.371161409999999, 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10285928799999999, 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 74458 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.141 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.720882+0000", 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.720882+0000", 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.720884+0000", 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.041e-06 2024-03-24T12:08:09.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.720907+0000", 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2753000000000001e-05 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.720926+0000", 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8817999999999998e-05 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.720941+0000", 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5011000000000001e-05 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.721379+0000", 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043789600000000001 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.721522+0000", 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014377300000000001 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.721666+0000", 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143435 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.725327+0000", 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00366155 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.727028+0000", 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017003999999999999 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.823636+0000", 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.09660763 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.823696+0000", 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0038999999999999e-05 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:16.823741+0000", 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5942000000000002e-05 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:75276 10.5 10:aafebecd:::benchmark_data_smithi064_174908_object75275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:17.342920+0000", 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.749123511000001, 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.083392514000000001, 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75276 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.342920+0000", 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.342920+0000", 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.342922+0000", 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2800000000000002e-06 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.342944+0000", 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2092000000000001e-05 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.342956+0000", 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1525999999999999e-05 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.342965+0000", 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5089999999999993e-06 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.343383+0000", 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041811799999999998 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.343556+0000", 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017313300000000001 2024-03-24T12:08:09.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.343700+0000", 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014426599999999999 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.346443+0000", 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027429020000000002 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347039+0000", 2024-03-24T12:08:09.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059636400000000005 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.426201+0000", 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079161881000000003 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.426267+0000", 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5918000000000001e-05 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.426313+0000", 2024-03-24T12:08:09.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5525000000000002e-05 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:75283 10.5 10:a4917b35:::benchmark_data_smithi064_174908_object75282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:17.347138+0000", 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.744905703000001, 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087060567000000005, 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75283 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347138+0000", 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347138+0000", 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347140+0000", 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.84e-06 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347157+0000", 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7589999999999999e-05 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347165+0000", 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9880000000000004e-06 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347171+0000", 2024-03-24T12:08:09.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3120000000000001e-06 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347207+0000", 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.625e-05 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347301+0000", 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4114e-05 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.347390+0000", 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9400999999999993e-05 2024-03-24T12:08:09.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.349321+0000", 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019302430000000001 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.349390+0000", 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9787000000000001e-05 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.433779+0000", 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084388424000000004 2024-03-24T12:08:09.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.433968+0000", 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018900899999999999 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:17.434198+0000", 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000230609 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:77347 10.5 10:acf9b25b:::benchmark_data_smithi064_174908_object77346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:18.841487+0000", 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.250556238999998, 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.083008905999999993, 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77347 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841487+0000", 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841487+0000", 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841490+0000", 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.65e-06 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841514+0000", 2024-03-24T12:08:09.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4256e-05 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841530+0000", 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6013999999999999e-05 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841540+0000", 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7100000000000002e-06 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841614+0000", 2024-03-24T12:08:09.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4388000000000007e-05 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841773+0000", 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015844500000000001 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.841931+0000", 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015820100000000001 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.847566+0000", 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056350849999999997 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.849438+0000", 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001871438 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.924345+0000", 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074907715 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.924437+0000", 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1274999999999996e-05 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.924496+0000", 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9728999999999997e-05 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:77354 10.5 10:ad01777c:::benchmark_data_smithi064_174908_object77353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:18.844427+0000", 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.247617022, 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.083970810000000007, 2024-03-24T12:08:09.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77354 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844427+0000", 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844427+0000", 2024-03-24T12:08:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844428+0000", 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479999999999999e-06 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844454+0000", 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5602999999999999e-05 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844466+0000", 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2101e-05 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844475+0000", 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1269999999999992e-06 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844535+0000", 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0053000000000003e-05 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844741+0000", 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020550399999999999 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.844972+0000", 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000230831 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.847754+0000", 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027821120000000002 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.849518+0000", 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017639610000000001 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.928178+0000", 2024-03-24T12:08:09.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078660404000000003 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.928344+0000", 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165365 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:18.928397+0000", 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3900999999999998e-05 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:83698 10.5 10:ab0e02bf:::benchmark_data_smithi064_174908_object83697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-24T12:08:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:23.367321+0000", 2024-03-24T12:08:09.170 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.724722305, 2024-03-24T12:08:09.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082625353999999998, 2024-03-24T12:08:09.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.170 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:09.171 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83698 2024-03-24T12:08:09.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.171 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367321+0000", 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367321+0000", 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367323+0000", 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1629999999999999e-06 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367348+0000", 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4867000000000001e-05 2024-03-24T12:08:09.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367362+0000", 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3412e-05 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367372+0000", 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8479999999999996e-06 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367435+0000", 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3870999999999999e-05 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367577+0000", 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014185900000000001 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.367712+0000", 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013489700000000001 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.370921+0000", 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032088820000000001 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.372809+0000", 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001888228 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.449825+0000", 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077015479999999997 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.449893+0000", 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8212000000000004e-05 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:23.449947+0000", 2024-03-24T12:08:09.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3634999999999997e-05 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:117006 10.5 10:ab67b4fc:::benchmark_data_smithi064_174908_object117005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.643580+0000", 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.448463660000002, 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681031999999998, 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117006 2024-03-24T12:08:09.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.643580+0000", 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.643580+0000", 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.643583+0000", 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.146e-06 2024-03-24T12:08:09.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.643611+0000", 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8095000000000001e-05 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.643636+0000", 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4533e-05 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.643645+0000", 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.85e-06 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.643711+0000", 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6795e-05 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.643910+0000", 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019825999999999999 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.644060+0000", 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014995000000000001 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.658691+0000", 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014631382 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.658719+0000", 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8160000000000001e-05 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.698958+0000", 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040238666999999999 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.699123+0000", 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164809 2024-03-24T12:08:09.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.699261+0000", 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013838500000000001 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:117049 10.5 10:af8b9378:::benchmark_data_smithi064_174908_object117048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.694075+0000", 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.397968356, 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059732881000000002, 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.181 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117049 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.694075+0000", 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.694075+0000", 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.694078+0000", 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4490000000000002e-06 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.694117+0000", 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890999999999997e-05 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.694141+0000", 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4538000000000001e-05 2024-03-24T12:08:09.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.694152+0000", 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0397999999999999e-05 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.704462+0000", 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010310208 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.704617+0000", 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015538300000000001 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.705066+0000", 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044849899999999999 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.715620+0000", 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010554365 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.726217+0000", 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010597166999999999 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.753345+0000", 2024-03-24T12:08:09.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027127779000000001 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.753636+0000", 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029143000000000002 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.753808+0000", 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171774 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:122398 10.5 10:a350ea81:::benchmark_data_smithi064_174908_object122397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:51.609428+0000", 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.482616, 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058319071, 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 122398 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.609428+0000", 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.609428+0000", 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.609430+0000", 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.774e-06 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.609456+0000", 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6046000000000001e-05 2024-03-24T12:08:09.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.609472+0000", 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.609482+0000", 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0438000000000001e-05 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.621214+0000", 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011731629 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.621355+0000", 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014084199999999999 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.624898+0000", 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003542791 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.643987+0000", 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019089251000000002 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.657329+0000", 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013342053 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.667252+0000", 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099229639999999994 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.667487+0000", 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023462799999999999 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:51.667747+0000", 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026003899999999999 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:141455 10.5 10:a60e7f91:::benchmark_data_smithi064_174908_object141454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:07.117726+0000", 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.974317962, 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052340469000000001, 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141455 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.117726+0000", 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.117726+0000", 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.117728+0000", 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0320000000000002e-06 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.117760+0000", 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2656e-05 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.117771+0000", 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0541e-05 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.117781+0000", 2024-03-24T12:08:09.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0322e-05 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.134176+0000", 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016395090000000001 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.134314+0000", 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013776399999999999 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.134467+0000", 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153127 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.153827+0000", 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019359914999999998 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.153914+0000", 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7016999999999997e-05 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.169682+0000", 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015768332999999999 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.169904+0000", 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022117999999999999 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.170066+0000", 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162492 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:141474 10.5 10:aec9b1f1:::benchmark_data_smithi064_174908_object141473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:08:09.197 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:07.133014+0000", 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.9590299529999999, 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049403781000000001, 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141474 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.133014+0000", 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.133014+0000", 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.133017+0000", 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0769999999999999e-06 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.133047+0000", 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0286000000000001e-05 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:09.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.133069+0000", 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1908000000000001e-05 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.133080+0000", 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1445e-05 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.144920+0000", 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011839644 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.145055+0000", 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013450800000000001 2024-03-24T12:08:09.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.145201+0000", 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000146945 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.170231+0000", 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0250295 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.182072+0000", 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011840702999999999 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.182161+0000", 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9549000000000006e-05 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.182243+0000", 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1819999999999999e-05 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.182417+0000", 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174396 2024-03-24T12:08:09.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.203 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:09.203 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:09.203 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:08:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:09.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:09.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:09.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:08 smithi193 ceph-mon[33593]: pgmap v719: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 52170/420930 objects degraded (12.394%) 2024-03-24T12:08:09.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:08 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:09.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:09.375 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:09.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:09.476 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:09.505 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:09.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:09.607 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:09.637 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:09.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:09.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total time run: 90.0169 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total writes made: 144414 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Write size: 65536 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object size: 65536 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Bandwidth (MB/sec): 100.269 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Bandwidth: 33.7466 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max bandwidth (MB/sec): 232.562 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min bandwidth (MB/sec): 68.3125 2024-03-24T12:08:09.762 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average IOPS: 1604 2024-03-24T12:08:09.763 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev IOPS: 539.946 2024-03-24T12:08:09.763 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max IOPS: 3721 2024-03-24T12:08:09.763 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min IOPS: 1093 2024-03-24T12:08:09.763 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average Latency(s): 0.00996854 2024-03-24T12:08:09.763 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Latency(s): 0.00836804 2024-03-24T12:08:09.763 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max latency(s): 0.159647 2024-03-24T12:08:09.763 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min latency(s): 0.00129496 2024-03-24T12:08:09.763 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:08:09.764 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:09.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:09.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:08:09.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:09.895 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:09.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:09.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:09.997 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:10.024 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:10.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:10.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:10.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:08:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:10.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:10.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:10.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:10.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:10.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:10.483 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:10.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:10.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:10.617 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:10.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:10.719 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:10.746 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:10.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:10.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:10.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:10.898 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:10.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:11.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:11.036 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:11.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:11.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:11.180 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:11.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:11.281 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:11.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:10 smithi064 ceph-mon[30223]: pgmap v720: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 52552/424146 objects degraded (12.390%) 2024-03-24T12:08:11.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:10 smithi064 ceph-mon[42243]: pgmap v720: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 52552/424146 objects degraded (12.390%) 2024-03-24T12:08:11.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:11.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:11.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:10 smithi193 ceph-mon[33593]: pgmap v720: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 52552/424146 objects degraded (12.390%) 2024-03-24T12:08:11.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:11.452 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:11.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:11.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:11.586 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:11.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:11.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:11.721 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:11.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:11.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:11.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:11.852 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:11.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:11.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:11.990 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:11.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:12.092 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:12.122 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:12.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:12.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:12.255 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:12.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:12.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:11 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 52552/424146 objects degraded (12.390%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:11 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 52552/424146 objects degraded (12.390%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:12.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:11 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 52552/424146 objects degraded (12.390%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:12.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:12.388 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:12.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:12.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:12.533 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:12.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:12.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:12.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:12.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:12.796 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:12.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:12.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:12.929 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:12.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:13.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:13.089 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:13.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:13.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:13.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:08:13.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[30223]: pgmap v721: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.17k op/s; 52876/425442 objects degraded (12.428%) 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[30223]: osdmap e342: 8 total, 7 up, 8 in 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[42243]: pgmap v721: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.17k op/s; 52876/425442 objects degraded (12.428%) 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[42243]: osdmap e342: 8 total, 7 up, 8 in 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:13.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:13.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:13.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:12 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:08:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:12 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:08:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:12 smithi193 ceph-mon[33593]: pgmap v721: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.17k op/s; 52876/425442 objects degraded (12.428%) 2024-03-24T12:08:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:12 smithi193 ceph-mon[33593]: osdmap e342: 8 total, 7 up, 8 in 2024-03-24T12:08:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:13.349 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:13.378 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:13.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:13.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:13.514 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:13.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:13.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:13.656 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:13.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:13.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:13.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:13.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:13.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:13.932 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:13.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:14.034 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:14.063 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:14.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:14.133 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:14.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:14.165 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:14.217 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:14.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:14.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:14.352 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:14.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:14.454 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:14.494 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:14.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:14.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:14.633 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:14.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:14.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:14.779 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:14.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:14.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:14.909 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:14.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:14.975 INFO:teuthology.orchestra.run.smithi064.stderr:set 2.13 pg_upmap mapping to [6,3,0] 2024-03-24T12:08:15.012 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:15.038 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:15.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:15.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:15.169 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:15.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:15.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:15.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:14 smithi064 ceph-mon[30223]: pgmap v723: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 1.15k op/s; 53001/426453 objects degraded (12.428%) 2024-03-24T12:08:15.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4193580369' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]: dispatch 2024-03-24T12:08:15.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:14 smithi064 ceph-mon[30223]: from='client.35651 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]: dispatch 2024-03-24T12:08:15.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:14 smithi064 ceph-mon[42243]: pgmap v723: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 1.15k op/s; 53001/426453 objects degraded (12.428%) 2024-03-24T12:08:15.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4193580369' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]: dispatch 2024-03-24T12:08:15.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:14 smithi064 ceph-mon[42243]: from='client.35651 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]: dispatch 2024-03-24T12:08:15.312 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:15.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:14 smithi193 ceph-mon[33593]: pgmap v723: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 1.15k op/s; 53001/426453 objects degraded (12.428%) 2024-03-24T12:08:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4193580369' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]: dispatch 2024-03-24T12:08:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:14 smithi193 ceph-mon[33593]: from='client.35651 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]: dispatch 2024-03-24T12:08:15.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:15.452 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:15.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:15.552 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:15.585 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:15.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:15.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:15.717 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:15.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:15.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:15.855 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:15.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:15.956 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:15.979 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:08:15.985 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:15.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:16.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:16.122 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:16.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:16.224 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:16.255 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:16.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:16.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:16 smithi064 ceph-mon[30223]: from='client.35651 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]': finished 2024-03-24T12:08:16.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:16 smithi064 ceph-mon[30223]: osdmap e343: 8 total, 7 up, 8 in 2024-03-24T12:08:16.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:16 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/688512907' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:08:16.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:16 smithi064 ceph-mon[42243]: from='client.35651 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]': finished 2024-03-24T12:08:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:16 smithi064 ceph-mon[42243]: osdmap e343: 8 total, 7 up, 8 in 2024-03-24T12:08:16.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:16 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/688512907' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:08:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:16 smithi193 ceph-mon[33593]: from='client.35651 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [6, 3, 0]}]': finished 2024-03-24T12:08:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:16 smithi193 ceph-mon[33593]: osdmap e343: 8 total, 7 up, 8 in 2024-03-24T12:08:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:16 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/688512907' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:08:16.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:16.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:16.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:16.553 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:16.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:16.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:16.693 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:16.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:16.795 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:16.824 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:16.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:16.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:16.966 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:16.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:17.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[30223]: pgmap v725: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 34 MiB/s wr, 845 op/s; 53001/426453 objects degraded (12.428%) 2024-03-24T12:08:17.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:17.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/688512907' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:08:17.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[30223]: osdmap e344: 8 total, 7 up, 8 in 2024-03-24T12:08:17.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 53001/426453 objects degraded (12.428%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:17.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[30223]: osdmap e345: 8 total, 7 up, 8 in 2024-03-24T12:08:17.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[42243]: pgmap v725: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 34 MiB/s wr, 845 op/s; 53001/426453 objects degraded (12.428%) 2024-03-24T12:08:17.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:17.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/688512907' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:08:17.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[42243]: osdmap e344: 8 total, 7 up, 8 in 2024-03-24T12:08:17.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 53001/426453 objects degraded (12.428%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:17.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-mon[42243]: osdmap e345: 8 total, 7 up, 8 in 2024-03-24T12:08:17.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:17.100 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:17.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:17.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:17.247 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:17.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:17 smithi193 ceph-mon[33593]: pgmap v725: 168 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 45 active+undersized, 98 active+clean; 8.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 34 MiB/s wr, 845 op/s; 53001/426453 objects degraded (12.428%) 2024-03-24T12:08:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:17 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/688512907' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:08:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:17 smithi193 ceph-mon[33593]: osdmap e344: 8 total, 7 up, 8 in 2024-03-24T12:08:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:17 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 53001/426453 objects degraded (12.428%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:17 smithi193 ceph-mon[33593]: osdmap e345: 8 total, 7 up, 8 in 2024-03-24T12:08:17.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:17.401 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:17.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:17.502 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:17.539 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:17.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:17.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:17.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:17.682 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:17.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:17.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:17.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:08:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:08:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:17.816 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:17.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:17.918 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:17.947 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:17.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:18.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:08:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:08:18.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:18.090 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:18.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:18.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:18.230 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:18.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:18.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:18 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:08:18.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:18 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:08:18.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:18 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:08:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:18 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:08:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:18 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:08:18.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:18 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:08:18.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:18 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:08:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:18 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:08:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:18 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:08:18.366 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:18.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:18.468 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:18.503 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:18.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:18.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:18.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:18.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:18.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:18.773 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:18.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:18.876 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:18.911 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:18.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:19.012 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:19.043 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:19.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:19.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:19.168 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:19.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:19.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:19 smithi064 ceph-mon[30223]: pgmap v728: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 926 B/s rd, 6.2 MiB/s wr, 1.16k op/s; 51186/411933 objects degraded (12.426%) 2024-03-24T12:08:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:19 smithi064 ceph-mon[30223]: osdmap e346: 8 total, 7 up, 8 in 2024-03-24T12:08:19.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:19 smithi064 ceph-mon[42243]: pgmap v728: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 926 B/s rd, 6.2 MiB/s wr, 1.16k op/s; 51186/411933 objects degraded (12.426%) 2024-03-24T12:08:19.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:19 smithi064 ceph-mon[42243]: osdmap e346: 8 total, 7 up, 8 in 2024-03-24T12:08:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:19.303 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:19.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:19 smithi193 ceph-mon[33593]: pgmap v728: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 926 B/s rd, 6.2 MiB/s wr, 1.16k op/s; 51186/411933 objects degraded (12.426%) 2024-03-24T12:08:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:19 smithi193 ceph-mon[33593]: osdmap e346: 8 total, 7 up, 8 in 2024-03-24T12:08:19.404 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:19.436 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:19.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:19.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:19.581 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:19.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:19.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:19.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:19.721 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:19.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:19.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:19.866 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:19.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:19.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:20.023 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:20.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:20.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:20.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:20.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:20.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:20.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:20.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:08:20.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:08:20.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[30223]: osdmap e347: 8 total, 7 up, 8 in 2024-03-24T12:08:20.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:20.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:20.289 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:08:20.289 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:08:20.290 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[42243]: osdmap e347: 8 total, 7 up, 8 in 2024-03-24T12:08:20.290 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:20.290 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:20 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:20.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:20 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:08:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:20 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:08:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:20 smithi193 ceph-mon[33593]: osdmap e347: 8 total, 7 up, 8 in 2024-03-24T12:08:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:20 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:20 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:20.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:20.463 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:20.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:20.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:20.607 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:20.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:20.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:20.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:20.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:20.845 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:20.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:20.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:21.011 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:21.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:21.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:21.151 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:21.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:21.253 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:21.287 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:21.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:21 smithi064 ceph-mon[30223]: pgmap v731: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 50847/408762 objects degraded (12.439%) 2024-03-24T12:08:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:21 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:08:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:21 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:08:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:21 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:21 smithi064 ceph-mon[42243]: pgmap v731: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 50847/408762 objects degraded (12.439%) 2024-03-24T12:08:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:21 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:08:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:21 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:08:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:21 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:21 smithi193 ceph-mon[33593]: pgmap v731: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 50847/408762 objects degraded (12.439%) 2024-03-24T12:08:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:21 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:08:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:21 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:08:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:21 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:21.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:21.417 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:21.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:21.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:21.560 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:21.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:21.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:21.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:21.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:21.832 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:21.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:21.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:21.969 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:21.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:22.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:22.111 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:22.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:22.209 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:22 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 50847/408762 objects degraded (12.439%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:22.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:22 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 50847/408762 objects degraded (12.439%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:22.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:22.221 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:08:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:22.251 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:22.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:22.253 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:08:22.296 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:159991 10.f e347/320 10:f85e5d37:::benchmark_data_smithi064_174908_object15574:head v 347'9982, mlcod=347'9980)", 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:22.322525+0000", 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.001857952, 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001917097, 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:22.331 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.322525+0000", 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.322525+0000", 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.322526+0000", 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.322535+0000", 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9679999999999995e-06 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.322545+0000", 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0787e-05 2024-03-24T12:08:22.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.322552+0000", 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0210000000000002e-06 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.322800+0000", 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024710500000000001 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.322819+0000", 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9218000000000001e-05 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:159993 10.7 e347/320 10:e32f5029:::benchmark_data_smithi064_174908_object15576:head v 347'10066, mlcod=347'10064)", 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:22.323870+0000", 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00051230799999999995, 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065011500000000005, 2024-03-24T12:08:22.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.323870+0000", 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.323870+0000", 2024-03-24T12:08:22.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.323871+0000", 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1000000000000001e-06 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.323880+0000", 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6419999999999996e-06 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.323892+0000", 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1955e-05 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.323899+0000", 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9940000000000003e-06 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.323985+0000", 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6230000000000006e-05 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.324006+0000", 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0387e-05 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:159986 10.5 e347/320 10:a7526279:::benchmark_data_smithi064_174908_object15569:head v 347'10304, mlcod=347'10300)", 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:22.317958+0000", 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0064247499999999999, 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066383759999999997, 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.317958+0000", 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:22.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.317958+0000", 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.317959+0000", 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.514e-06 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.317968+0000", 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2009999999999999e-06 2024-03-24T12:08:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.317983+0000", 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5421999999999999e-05 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.317992+0000", 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8850000000000005e-06 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:22.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.318079+0000", 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6569000000000004e-05 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:22.318106+0000", 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7804999999999999e-05 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 3 2024-03-24T12:08:22.343 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:08:22.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:08:22.344 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:22 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 50847/408762 objects degraded (12.439%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:22.347 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:08:22.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:22.387 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:08:22.426 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:08:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:22.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:22.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:22.470 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:08:22.470 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:08:22.470 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:08:22.470 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:08:22.470 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:08:22.470 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:08:22.470 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:08:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:08:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:08:22.546 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:22.583 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:22.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:22.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:22.731 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:22.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:22.833 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:22.860 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:22.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:22.962 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:22.999 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:23.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:23.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:23.133 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:23.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:23.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:23.275 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:23.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:23 smithi064 ceph-mon[30223]: pgmap v732: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 935 B/s rd, 0 B/s wr, 1.30k op/s; 50029/405138 objects degraded (12.349%) 2024-03-24T12:08:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:23 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3646618881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:23.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:23 smithi064 ceph-mon[42243]: pgmap v732: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 935 B/s rd, 0 B/s wr, 1.30k op/s; 50029/405138 objects degraded (12.349%) 2024-03-24T12:08:23.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:23 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3646618881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:23 smithi193 ceph-mon[33593]: pgmap v732: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 935 B/s rd, 0 B/s wr, 1.30k op/s; 50029/405138 objects degraded (12.349%) 2024-03-24T12:08:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:23 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3646618881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:23.377 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:23.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:08:23.411 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:23.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:23.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:23.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:23.556 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:23.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:23.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:23.658 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:23.693 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:23.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:23.795 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:23.828 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:23.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:23.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:23.964 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:23.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:24.065 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:24.091 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:24.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:08:24.095 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:08:24.142 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:24.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:24.205 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:24.250 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:24.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:24.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:24 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:08:24.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:24 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:08:24.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:24 smithi064 ceph-mon[30223]: osdmap e348: 8 total, 7 up, 8 in 2024-03-24T12:08:24.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:24 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:08:24.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:24 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:08:24.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:24 smithi064 ceph-mon[42243]: osdmap e348: 8 total, 7 up, 8 in 2024-03-24T12:08:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:24 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:08:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:24 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:08:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:24 smithi193 ceph-mon[33593]: osdmap e348: 8 total, 7 up, 8 in 2024-03-24T12:08:24.352 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:24.375 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:24.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:24.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:24.515 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:24.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:24.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:24.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:24.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:24.796 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:24.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:24.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:24.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:24.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:25.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:25.078 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:25.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:25.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:25.184 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:25 smithi064 ceph-mon[42243]: pgmap v734: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 48375/388917 objects degraded (12.438%) 2024-03-24T12:08:25.184 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:25 smithi064 ceph-mon[30223]: pgmap v734: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 48375/388917 objects degraded (12.438%) 2024-03-24T12:08:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:25.218 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:25.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:25.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:25 smithi193 ceph-mon[33593]: pgmap v734: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 48375/388917 objects degraded (12.438%) 2024-03-24T12:08:25.352 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:25.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:25.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:25.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:25.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:25.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:25.621 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:25.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:25.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:25.650 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:08:25.650 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:08:25.650 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-24T12:08:25.650 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-03-24T12:08:25.650 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T12:08:25.665 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl start ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.5 2024-03-24T12:08:25.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:25.762 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:25.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:25.863 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:25.902 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:25.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:26.003 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:26.033 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:26.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:26.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:25 smithi193 systemd[1]: Starting Ceph osd.5 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:08:26.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:08:26.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:26.169 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:26.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:26.271 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:26.304 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:26.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:26.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:26.451 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:26.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:26.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:26.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:26.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:26.731 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:26.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:26.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:26.871 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:26.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:26.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:27.010 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:27.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:27.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:27.146 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:27.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:27.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:27.287 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:27.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:27.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:27 smithi193 ceph-mon[33593]: pgmap v735: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 1.03k op/s; 48375/388917 objects degraded (12.438%) 2024-03-24T12:08:27.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:27 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 48375/388917 objects degraded (12.438%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:27.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:27 smithi193 ceph-mon[33593]: osdmap e349: 8 total, 7 up, 8 in 2024-03-24T12:08:27.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:27.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:26 smithi193 podman[134874]: 2024-03-24 12:08:26.865367248 +0000 UTC m=+0.104526456 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:08:27.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:27 smithi193 podman[134874]: 2024-03-24 12:08:27.042680128 +0000 UTC m=+0.281839323 container create 700bea2ed7d44e4a04bb619a09176ab05eced7ff6f2abacb0376d602bf353dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T12:08:27.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-mon[30223]: pgmap v735: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 1.03k op/s; 48375/388917 objects degraded (12.438%) 2024-03-24T12:08:27.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 48375/388917 objects degraded (12.438%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:27.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-mon[30223]: osdmap e349: 8 total, 7 up, 8 in 2024-03-24T12:08:27.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:27.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-mon[42243]: pgmap v735: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 1.03k op/s; 48375/388917 objects degraded (12.438%) 2024-03-24T12:08:27.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 48375/388917 objects degraded (12.438%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:27.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-mon[42243]: osdmap e349: 8 total, 7 up, 8 in 2024-03-24T12:08:27.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:27.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:27.426 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:27.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:27.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:27.564 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:27.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:27.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:27.700 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:27.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:27.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:27.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:08:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:08:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:27.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:27.831 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:08:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:08:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:27.846 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:27.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:27.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:27.989 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:27.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:28.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:28.094 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:08:28.125 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:28.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:28.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:28.264 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:28.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:28.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:27 smithi193 podman[134874]: 2024-03-24 12:08:27.954291764 +0000 UTC m=+1.193450962 container init 700bea2ed7d44e4a04bb619a09176ab05eced7ff6f2abacb0376d602bf353dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T12:08:28.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:27 smithi193 podman[134874]: 2024-03-24 12:08:27.967070642 +0000 UTC m=+1.206229835 container start 700bea2ed7d44e4a04bb619a09176ab05eced7ff6f2abacb0376d602bf353dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:08:28.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 podman[134874]: 2024-03-24 12:08:28.07933977 +0000 UTC m=+1.318498965 container attach 700bea2ed7d44e4a04bb619a09176ab05eced7ff6f2abacb0376d602bf353dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=) 2024-03-24T12:08:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:08:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:08:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1310396527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:08:28.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:28.401 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:28.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:28.503 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:28.537 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:28.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:28.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:28 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:08:28.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:28 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:08:28.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1310396527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:08:28.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:28 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:08:28.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:28 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:08:28.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:28 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1310396527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:08:28.639 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:28.680 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:28.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:28.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:28.820 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:28.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate[134905]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate[134905]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 bash[134874]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 bash[134874]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 bash[134874]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 bash[134874]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 bash[134874]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 bash[134874]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 bash[134874]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T12:08:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate[134905]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T12:08:28.837 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate[134905]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T12:08:28.837 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate[134905]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T12:08:28.837 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate[134905]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T12:08:28.837 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate[134905]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T12:08:28.837 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:28 smithi193 podman[134874]: 2024-03-24 12:08:28.74684527 +0000 UTC m=+1.986004463 container died 700bea2ed7d44e4a04bb619a09176ab05eced7ff6f2abacb0376d602bf353dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T12:08:28.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:28.953 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:28.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:29.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:29.097 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:29.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:29.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:29.232 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:29.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:29.277 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:08:29.278 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:08:29.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:29.366 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:29.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:29.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:29.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:29.491 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:29.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:29.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:29 smithi193 ceph-mon[33593]: pgmap v737: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 46172/371985 objects degraded (12.412%) 2024-03-24T12:08:29.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:29 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:29.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:29 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:29.543 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:29 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1310396527' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:08:29.543 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:29 smithi193 ceph-mon[33593]: osdmap e350: 8 total, 7 up, 8 in 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[30223]: pgmap v737: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 46172/371985 objects degraded (12.412%) 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1310396527' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[30223]: osdmap e350: 8 total, 7 up, 8 in 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[42243]: pgmap v737: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 46172/371985 objects degraded (12.412%) 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1310396527' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:08:29.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:29 smithi064 ceph-mon[42243]: osdmap e350: 8 total, 7 up, 8 in 2024-03-24T12:08:29.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:29.630 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:29.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:29.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:29.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:29.768 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:29.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:29.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:29 smithi193 podman[135138]: 2024-03-24 12:08:29.543569334 +0000 UTC m=+0.784387230 container remove 700bea2ed7d44e4a04bb619a09176ab05eced7ff6f2abacb0376d602bf353dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-activate, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T12:08:29.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:29.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:29.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:30.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:30.044 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:30.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:30.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:30.169 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:30.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:30.176 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:30 smithi193 ceph-mon[33593]: 10.2 scrub starts 2024-03-24T12:08:30.176 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:29 smithi193 podman[135246]: 2024-03-24 12:08:29.762728418 +0000 UTC m=+0.037194313 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:08:30.176 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:29 smithi193 podman[135246]: 2024-03-24 12:08:29.894846655 +0000 UTC m=+0.169312533 container create f0dfecc84f55207dcc31a0efda53523fa109a691d44617bedd4025ea0457f7ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, ceph=True) 2024-03-24T12:08:30.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:30.307 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:30.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:30.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:08:30.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:30.442 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:30.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:30.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:30.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:30 smithi064 ceph-mon[30223]: 10.2 scrub starts 2024-03-24T12:08:30.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:30 smithi064 ceph-mon[42243]: 10.2 scrub starts 2024-03-24T12:08:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:30.583 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:30.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:30.585 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:30 smithi193 podman[135246]: 2024-03-24 12:08:30.280985739 +0000 UTC m=+0.555451630 container init f0dfecc84f55207dcc31a0efda53523fa109a691d44617bedd4025ea0457f7ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:08:30.585 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:30 smithi193 podman[135246]: 2024-03-24 12:08:30.292246336 +0000 UTC m=+0.566712219 container start f0dfecc84f55207dcc31a0efda53523fa109a691d44617bedd4025ea0457f7ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:08:30.585 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:30 smithi193 bash[135246]: f0dfecc84f55207dcc31a0efda53523fa109a691d44617bedd4025ea0457f7ff 2024-03-24T12:08:30.585 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:30 smithi193 systemd[1]: Started Ceph osd.5 for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T12:08:30.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:30.721 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:30.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:30.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:30.866 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:30.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:30.876 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:08:30.968 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:31.004 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:31.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:31.103 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:30 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:08:30.837+0000 7f2d0b69b7c0 -1 Falling back to public interface 2024-03-24T12:08:31.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:31.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:31.164 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:31.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:31.291 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:31.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:31.359 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:31 smithi193 ceph-mon[33593]: pgmap v739: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 798 B/s rd, 0 B/s wr, 1.28k op/s; 45776/368118 objects degraded (12.435%) 2024-03-24T12:08:31.359 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:31 smithi193 ceph-mon[33593]: 10.b scrub starts 2024-03-24T12:08:31.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:31.433 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:31.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:31.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:31 smithi064 ceph-mon[30223]: pgmap v739: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 798 B/s rd, 0 B/s wr, 1.28k op/s; 45776/368118 objects degraded (12.435%) 2024-03-24T12:08:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:31 smithi064 ceph-mon[30223]: 10.b scrub starts 2024-03-24T12:08:31.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:31 smithi064 ceph-mon[42243]: pgmap v739: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 798 B/s rd, 0 B/s wr, 1.28k op/s; 45776/368118 objects degraded (12.435%) 2024-03-24T12:08:31.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:31 smithi064 ceph-mon[42243]: 10.b scrub starts 2024-03-24T12:08:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:31.569 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:31.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:31.670 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:31.687 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:08:31.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:31.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:31.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:31.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:31.839 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:31.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:31.940 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:31.968 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:31.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:32.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:32.111 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:32.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:32.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:32.253 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:32.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:32.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:32 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T12:08:32.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:32 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T12:08:32.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:32 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 45776/368118 objects degraded (12.435%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:32.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:32 smithi064 ceph-mon[30223]: osdmap e351: 8 total, 7 up, 8 in 2024-03-24T12:08:32.262 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:32 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T12:08:32.262 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:32 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T12:08:32.262 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:32 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 45776/368118 objects degraded (12.435%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:32.262 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:32 smithi064 ceph-mon[42243]: osdmap e351: 8 total, 7 up, 8 in 2024-03-24T12:08:32.325 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:32 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T12:08:32.326 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:32 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T12:08:32.326 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:32 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 45776/368118 objects degraded (12.435%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:32.326 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:32 smithi193 ceph-mon[33593]: osdmap e351: 8 total, 7 up, 8 in 2024-03-24T12:08:32.355 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:32.392 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:32.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:32.493 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:32.525 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:32.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:32.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:32.661 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:32.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:32.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:32.801 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:32.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:32.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:32.944 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:32.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:33.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:33.067 INFO:teuthology.orchestra.run.smithi193.stderr:no valid command found; 10 closest matches: 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:0 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:1 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:2 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:abort 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:assert 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:config diff 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:config diff get 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:config get 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:config help [] 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:config set ... 2024-03-24T12:08:33.068 INFO:teuthology.orchestra.run.smithi193.stderr:admin_socket: invalid command 2024-03-24T12:08:33.084 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:33.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:33.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:33.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:08:33.251 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:33.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:33 smithi193 ceph-mon[33593]: pgmap v740: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 44955/364413 objects degraded (12.336%) 2024-03-24T12:08:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:33 smithi193 ceph-mon[33593]: 10.2 scrub ok 2024-03-24T12:08:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:33 smithi193 ceph-mon[33593]: 10.b scrub ok 2024-03-24T12:08:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:33 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T12:08:33.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:33 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T12:08:33.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:33.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[30223]: pgmap v740: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 44955/364413 objects degraded (12.336%) 2024-03-24T12:08:33.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[30223]: 10.2 scrub ok 2024-03-24T12:08:33.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[30223]: 10.b scrub ok 2024-03-24T12:08:33.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T12:08:33.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T12:08:33.385 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[42243]: pgmap v740: 168 pgs: 24 active+undersized+degraded, 45 active+undersized, 99 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 44955/364413 objects degraded (12.336%) 2024-03-24T12:08:33.385 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[42243]: 10.2 scrub ok 2024-03-24T12:08:33.385 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[42243]: 10.b scrub ok 2024-03-24T12:08:33.385 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T12:08:33.385 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:33 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T12:08:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:33.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:33.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:33.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:33.542 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:33.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:33.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:33.668 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:33.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:33.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:33.806 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:33.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:33.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:33.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:33.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:34.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:34.074 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:34.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:34.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:33 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:08:33.647+0000 7f2d0b69b7c0 -1 osd.5 317 log_to_monitors true 2024-03-24T12:08:34.175 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:34.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:34.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:08:34.312 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-24T12:08:34.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:34.355 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:34.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:34.427 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:08:34.186+0000 7f2cfcaf3700 -1 osd.5 317 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:08:34.427 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 10.8 scrub starts 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: from='osd.5 v1:172.21.15.193:6804/1531860860' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:08:34.428 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:34 smithi193 ceph-mon[33593]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:08:34.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:34.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:34.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 10.8 scrub starts 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: from='osd.5 v1:172.21.15.193:6804/1531860860' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:08:34.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[30223]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 10.8 scrub starts 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: from='osd.5 v1:172.21.15.193:6804/1531860860' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:08:34.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:34 smithi064 ceph-mon[42243]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:08:34.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:34.619 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:34.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:34.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:34.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:34.870 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:34.902 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:34.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:35.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:35.036 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:35.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:35.106 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:08:35.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:35.149 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:08:35.181 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:35.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:35.185 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:08:35.214 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:08:35.235 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:08:35.256 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:08:35.275 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:08:35.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:35.295 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:08:35.296 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:08:35.296 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:08:35.296 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:08:35.296 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:08:35.297 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:08:35.297 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:08:35.297 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:08:35.297 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:08:35.307 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:35.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:35.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:35.447 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:35.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:35.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: pgmap v742: 168 pgs: 2 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 97 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.31k op/s; 43338/348747 objects degraded (12.427%) 2024-03-24T12:08:35.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:08:35.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: from='osd.5 v1:172.21.15.193:6804/1531860860' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T12:08:35.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: osdmap e352: 8 total, 7 up, 8 in 2024-03-24T12:08:35.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T12:08:35.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T12:08:35.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T12:08:35.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T12:08:35.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T12:08:35.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1407276267' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:35.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: pgmap v742: 168 pgs: 2 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 97 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.31k op/s; 43338/348747 objects degraded (12.427%) 2024-03-24T12:08:35.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:08:35.488 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: from='osd.5 v1:172.21.15.193:6804/1531860860' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T12:08:35.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: osdmap e352: 8 total, 7 up, 8 in 2024-03-24T12:08:35.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T12:08:35.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T12:08:35.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T12:08:35.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T12:08:35.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T12:08:35.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1407276267' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:35.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:35.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: pgmap v742: 168 pgs: 2 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 97 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.31k op/s; 43338/348747 objects degraded (12.427%) 2024-03-24T12:08:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:08:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: from='osd.5 v1:172.21.15.193:6804/1531860860' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T12:08:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: osdmap e352: 8 total, 7 up, 8 in 2024-03-24T12:08:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T12:08:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T12:08:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T12:08:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T12:08:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T12:08:35.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1407276267' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:35.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:35.702 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:35.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:35.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:35.828 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:35.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:35.930 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:35.954 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:35.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:35.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:36.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:36.079 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:36.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:36.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:36.205 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:36.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:36.307 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:36.330 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:36.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:36.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:36.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:36.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:36.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:08:36.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:08:36.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[30223]: 10.8 scrub ok 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[30223]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[30223]: osd.5 v1:172.21.15.193:6804/1531860860 boot 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[30223]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[30223]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[42243]: 10.8 scrub ok 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[42243]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[42243]: osd.5 v1:172.21.15.193:6804/1531860860 boot 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[42243]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:08:36.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:36 smithi064 ceph-mon[42243]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T12:08:36.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:36.581 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:36.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:36.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:36 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:08:36.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:36 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:08:36.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:36 smithi193 ceph-mon[33593]: 10.8 scrub ok 2024-03-24T12:08:36.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:36 smithi193 ceph-mon[33593]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:08:36.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:36 smithi193 ceph-mon[33593]: osd.5 v1:172.21.15.193:6804/1531860860 boot 2024-03-24T12:08:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:36 smithi193 ceph-mon[33593]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T12:08:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:36 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:08:36.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:36 smithi193 ceph-mon[33593]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T12:08:36.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:36.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:36.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:36.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:36.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:36.872 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:36.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:36.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:36.997 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:36.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:37.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:37.124 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:37.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:37.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:37.261 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:37.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:37.361 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:37.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:37.428 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:37.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:37.475 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:08:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:37.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:08:37.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:08:37.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: pgmap v745: 168 pgs: 2 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 97 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 43338/348747 objects degraded (12.427%) 2024-03-24T12:08:37.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: 1.6 scrub starts 2024-03-24T12:08:37.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: 1.6 scrub ok 2024-03-24T12:08:37.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: 1.4 scrub starts 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: 1.4 scrub ok 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: 1.10 scrub starts 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: 1.10 scrub ok 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 43338/348747 objects degraded (12.427%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 352'11078 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[30223]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: pgmap v745: 168 pgs: 2 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 97 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 43338/348747 objects degraded (12.427%) 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: 1.6 scrub starts 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: 1.6 scrub ok 2024-03-24T12:08:37.476 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: 1.4 scrub starts 2024-03-24T12:08:37.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: 1.4 scrub ok 2024-03-24T12:08:37.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:08:37.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:08:37.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: 1.10 scrub starts 2024-03-24T12:08:37.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: 1.10 scrub ok 2024-03-24T12:08:37.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 43338/348747 objects degraded (12.427%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:37.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 352'11078 2024-03-24T12:08:37.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:37 smithi064 ceph-mon[42243]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T12:08:37.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:37.564 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:37.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:08:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:08:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: pgmap v745: 168 pgs: 2 active+clean+scrubbing, 24 active+undersized+degraded, 45 active+undersized, 97 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 43338/348747 objects degraded (12.427%) 2024-03-24T12:08:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: 1.6 scrub starts 2024-03-24T12:08:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: 1.6 scrub ok 2024-03-24T12:08:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: 1.4 scrub starts 2024-03-24T12:08:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: 1.4 scrub ok 2024-03-24T12:08:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:08:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:08:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: 1.10 scrub starts 2024-03-24T12:08:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: 1.10 scrub ok 2024-03-24T12:08:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 43338/348747 objects degraded (12.427%), 24 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-24T12:08:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 352'11078 2024-03-24T12:08:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-mon[33593]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T12:08:37.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:37.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:37.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:37.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:37.846 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:37.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:37.950 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:37.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:38.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:08:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:08:38.093 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:38.127 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:38.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:38.229 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:38.266 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:38.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:38.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:38.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:38.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:38.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:38.540 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:38.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:38.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T12:08:38.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T12:08:38.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T12:08:38.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T12:08:38.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 352'11166 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 352'10828 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 352'10911 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 352'11061 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[30223]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 352'10906 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 352'11166 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 352'10828 2024-03-24T12:08:38.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 352'10911 2024-03-24T12:08:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 352'11061 2024-03-24T12:08:38.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:38 smithi064 ceph-mon[42243]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 352'10906 2024-03-24T12:08:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T12:08:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T12:08:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T12:08:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T12:08:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 352'11166 2024-03-24T12:08:38.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 352'10828 2024-03-24T12:08:38.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 352'10911 2024-03-24T12:08:38.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 352'11061 2024-03-24T12:08:38.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:38 smithi193 ceph-mon[33593]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 352'10906 2024-03-24T12:08:38.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:38.674 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:38.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:38.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:08:38.777 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:38.812 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:38.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:38.906 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104600 8.18 8:195e728d:::benchmark_data_smithi064_120708_object104599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 506.14495628700001, 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0728234670000001, 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:08:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104600 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759455+0000", 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759458+0000", 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6309999999999999e-06 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759492+0000", 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4329000000000003e-05 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759509+0000", 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6705e-05 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759520+0000", 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.111e-05 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759583+0000", 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3483000000000002e-05 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759592+0000", 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804373+0000", 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0447811650000001 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804557+0000", 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018477399999999999 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.804989+0000", 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043201600000000002 2024-03-24T12:08:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809404+0000", 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044143280000000003 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813653+0000", 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042489600000000004 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.815161+0000", 2024-03-24T12:08:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001507911 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831615+0000", 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016453925000000001 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831679+0000", 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4657000000000001e-05 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832279+0000", 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059939900000000003 2024-03-24T12:08:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104606 8.18 8:1f771a39:::benchmark_data_smithi064_120708_object104605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 506.14487083300003, 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.073005889, 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104606 2024-03-24T12:08:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759541+0000", 2024-03-24T12:08:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759564+0000", 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2459999999999998e-05 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759572+0000", 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8399999999999995e-06 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759578+0000", 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5830000000000003e-06 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759626+0000", 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7576e-05 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759632+0000", 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4500000000000001e-06 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.806996+0000", 2024-03-24T12:08:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0473636369999999 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.807166+0000", 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170104 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.807568+0000", 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000401455 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813591+0000", 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006023919 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.815217+0000", 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016253210000000001 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831571+0000", 2024-03-24T12:08:38.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016353873000000001 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832375+0000", 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080415300000000005 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832441+0000", 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5860999999999998e-05 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832547+0000", 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010589799999999999 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:104642 8.18 8:1b5ea0e5:::benchmark_data_smithi064_120708_object104641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e199)", 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 506.14482350499998, 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.073407242, 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104642 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759588+0000", 2024-03-24T12:08:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759589+0000", 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759610+0000", 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0948e-05 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759616+0000", 2024-03-24T12:08:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8589999999999999e-06 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759622+0000", 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9519999999999999e-06 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759649+0000", 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6659999999999999e-05 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:12.759653+0000", 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9419999999999997e-06 2024-03-24T12:08:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809440+0000", 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0497873879999999 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.809603+0000", 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016285099999999999 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5,7", 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.810439+0000", 2024-03-24T12:08:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083558799999999996 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.813599+0000", 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031601060000000002 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.831369+0000", 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017770563 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832331+0000", 2024-03-24T12:08:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096128100000000003 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832805+0000", 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000473864 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832873+0000", 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8516999999999995e-05 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:13.832995+0000", 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122139 2024-03-24T12:08:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216862 9.2 9:4d09947b:::benchmark_data_smithi064_148274_object91049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.169187327, 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0907798720000002, 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216862 2024-03-24T12:08:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735226+0000", 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.272e-06 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735233+0000", 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.494e-06 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735246+0000", 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.277e-05 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735253+0000", 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7889999999999995e-06 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.744476+0000", 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092227190000000007 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.744486+0000", 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5070000000000006e-06 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722078+0000", 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97759208500000006 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722094+0000", 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6337999999999998e-05 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818237+0000", 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096142892 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818340+0000", 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010314099999999999 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818710+0000", 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036944499999999998 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821730+0000", 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00302075 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825432+0000", 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037018089999999999 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825705+0000", 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027328399999999999 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825805+0000", 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9975000000000004e-05 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825875+0000", 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9665999999999993e-05 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826004+0000", 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012893599999999999 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216877 9.2 9:41202982:::benchmark_data_smithi064_148274_object91064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.14737316899999, 2024-03-24T12:08:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0695391550000002, 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216877 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757041+0000", 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4339999999999999e-06 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757048+0000", 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6369999999999999e-06 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757062+0000", 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.359e-05 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757072+0000", 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0125999999999999e-05 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757145+0000", 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2819999999999997e-05 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757158+0000", 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2721999999999999e-05 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722171+0000", 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96501314900000001 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722181+0000", 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0164e-05 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821770+0000", 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0995892380000001 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821855+0000", 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4513999999999998e-05 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821947+0000", 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2659999999999997e-05 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825725+0000", 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037774430000000001 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825754+0000", 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9048e-05 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826079+0000", 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000325015 2024-03-24T12:08:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826391+0000", 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031238199999999999 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826454+0000", 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2303000000000003e-05 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826577+0000", 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012391 2024-03-24T12:08:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216879 9.2 9:4db289b8:::benchmark_data_smithi064_148274_object91066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.14584535500001, 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0685427669999998, 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216879 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758568+0000", 2024-03-24T12:08:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9309999999999998e-06 2024-03-24T12:08:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758574+0000", 2024-03-24T12:08:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000002e-06 2024-03-24T12:08:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758586+0000", 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2075999999999999e-05 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758595+0000", 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5930000000000006e-06 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758682+0000", 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7328000000000001e-05 2024-03-24T12:08:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758694+0000", 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2271e-05 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722222+0000", 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.963528159 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722232+0000", 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2650000000000002e-06 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824044+0000", 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1018119829999999 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824123+0000", 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9806000000000001e-05 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824206+0000", 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2086e-05 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826021+0000", 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001815126 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826703+0000", 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068264900000000002 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826730+0000", 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6710999999999999e-05 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826787+0000", 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7014e-05 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826892+0000", 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105351 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827109+0000", 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021650799999999999 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216888 9.2 9:4c303ed8:::benchmark_data_smithi064_148274_object91075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.143605581, 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.079817389, 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216888 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760808+0000", 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9139999999999998e-06 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760813+0000", 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6699999999999999e-06 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760826+0000", 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2533e-05 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760835+0000", 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3319999999999998e-06 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760923+0000", 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7826999999999998e-05 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760940+0000", 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6396999999999999e-05 2024-03-24T12:08:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722255+0000", 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96131583300000001 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722263+0000", 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0669999999999992e-06 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827147+0000", 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.104883244 2024-03-24T12:08:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827325+0000", 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001784 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827519+0000", 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019378400000000001 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835233+0000", 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077136690000000003 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835278+0000", 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5664000000000002e-05 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835321+0000", 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2817e-05 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840424+0000", 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051024970000000001 2024-03-24T12:08:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840504+0000", 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0844e-05 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840623+0000", 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118897 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216890 9.2 9:4575dfdc:::benchmark_data_smithi064_148274_object91077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.140311271, 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0768427539999998, 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216890 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764103+0000", 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4870000000000001e-06 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764110+0000", 2024-03-24T12:08:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2609999999999996e-06 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764123+0000", 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3369e-05 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764133+0000", 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8179999999999999e-06 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764206+0000", 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2471999999999993e-05 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764244+0000", 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8655000000000002e-05 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722287+0000", 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95804261999999996 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722295+0000", 2024-03-24T12:08:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6129999999999998e-06 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830293+0000", 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079985539999999 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830416+0000", 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012251699999999999 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830550+0000", 2024-03-24T12:08:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013488000000000001 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835368+0000", 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048179049999999999 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840637+0000", 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005268928 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840707+0000", 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9367000000000002e-05 2024-03-24T12:08:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840755+0000", 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8789000000000003e-05 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840813+0000", 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7757999999999998e-05 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840943+0000", 2024-03-24T12:08:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012976099999999999 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216938 9.2 9:41fc8627:::benchmark_data_smithi064_148274_object91125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.09644078400001, 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.033607655, 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216938 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807972+0000", 2024-03-24T12:08:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807977+0000", 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8450000000000002e-06 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807990+0000", 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3091999999999999e-05 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807998+0000", 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6769999999999995e-06 2024-03-24T12:08:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.808072+0000", 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3719000000000003e-05 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.808082+0000", 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0648999999999999e-05 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722352+0000", 2024-03-24T12:08:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.9142692 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722361+0000", 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7030000000000001e-06 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832521+0000", 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.110159882 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832692+0000", 2024-03-24T12:08:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171242 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832843+0000", 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015096200000000001 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840408+0000", 2024-03-24T12:08:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075646669999999997 2024-03-24T12:08:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840644+0000", 2024-03-24T12:08:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023627500000000001 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840991+0000", 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034632899999999998 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841121+0000", 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130445 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841180+0000", 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8777999999999999e-05 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841578+0000", 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039849699999999999 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216940 9.2 9:46d14649:::benchmark_data_smithi064_148274_object91127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.09487717499999, 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.032468546, 2024-03-24T12:08:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216940 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:08:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809536+0000", 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.57e-06 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809541+0000", 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0470000000000001e-06 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809555+0000", 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.417e-05 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809563+0000", 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.622e-06 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809619+0000", 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.643e-05 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809631+0000", 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1477e-05 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722387+0000", 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.91275614100000002 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722395+0000", 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4279999999999996e-06 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835404+0000", 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1130083449999999 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835526+0000", 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122069 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835639+0000", 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113329 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840651+0000", 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050119439999999999 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841055+0000", 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000404127 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841634+0000", 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057911699999999998 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841735+0000", 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100603 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841807+0000", 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1997999999999997e-05 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842003+0000", 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019612899999999999 2024-03-24T12:08:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216945 9.2 9:46734ef5:::benchmark_data_smithi064_148274_object91132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.08932478099999, 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.027242598, 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216945 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815089+0000", 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5110000000000002e-06 2024-03-24T12:08:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815098+0000", 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815115+0000", 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833999999999998e-05 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815125+0000", 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7019999999999996e-06 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815205+0000", 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0593999999999994e-05 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815214+0000", 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7630000000000006e-06 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722420+0000", 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.90720568000000001 2024-03-24T12:08:38.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722428+0000", 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3610000000000001e-06 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837728+0000", 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.115299724 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837849+0000", 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121298 2024-03-24T12:08:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837993+0000", 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014408300000000001 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841007+0000", 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030141429999999999 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841682+0000", 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067442900000000004 2024-03-24T12:08:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842071+0000", 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038890700000000002 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842145+0000", 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4214999999999993e-05 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842218+0000", 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3461999999999996e-05 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842329+0000", 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110983 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216966 9.2 9:44cdb63b:::benchmark_data_smithi064_148274_object91153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:08:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.051787464, 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0008592350000001, 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216966 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852626+0000", 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852629+0000", 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3390000000000001e-06 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852638+0000", 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5720000000000003e-06 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852643+0000", 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5369999999999997e-06 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852719+0000", 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6086000000000003e-05 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852728+0000", 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3000000000000002e-06 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722453+0000", 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86972569 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722461+0000", 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2830000000000001e-06 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842378+0000", 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1199161879999999 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842492+0000", 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114769 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842616+0000", 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012353200000000001 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851293+0000", 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086773370000000002 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851656+0000", 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036241099999999998 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851701+0000", 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5457999999999997e-05 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853267+0000", 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001566095 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853330+0000", 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2955999999999998e-05 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853483+0000", 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153022 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216969 9.2 9:4ad9ca10:::benchmark_data_smithi064_148274_object91156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.04301246200001, 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9923648649999999, 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216969 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:08:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861400+0000", 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.438e-06 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861404+0000", 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9469999999999996e-06 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861415+0000", 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0396e-05 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861421+0000", 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2600000000000002e-06 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861479+0000", 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7921000000000003e-05 2024-03-24T12:08:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861486+0000", 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6359999999999997e-06 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722487+0000", 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86100120400000002 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722495+0000", 2024-03-24T12:08:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2300000000000008e-06 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.844900+0000", 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1224051230000001 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.845026+0000", 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126062 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.845364+0000", 2024-03-24T12:08:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000337809 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851301+0000", 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059373220000000001 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851745+0000", 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044327599999999999 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851788+0000", 2024-03-24T12:08:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3272999999999997e-05 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853579+0000", 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001791555 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853644+0000", 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4944999999999996e-05 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853764+0000", 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119468 2024-03-24T12:08:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:116402 10.3 10:c1f1e178:::benchmark_data_smithi064_174908_object116401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.133056+0000", 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.771355219999997, 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14694664800000001, 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116402 2024-03-24T12:08:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133056+0000", 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133056+0000", 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133059+0000", 2024-03-24T12:08:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3460000000000001e-06 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133198+0000", 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013925499999999999 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133321+0000", 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012289000000000001 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133330+0000", 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0310000000000004e-06 2024-03-24T12:08:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133387+0000", 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7463000000000002e-05 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133541+0000", 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015386999999999999 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133695+0000", 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015424500000000001 2024-03-24T12:08:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.143037+0000", 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093416969999999995 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.279825+0000", 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.136788142 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.279901+0000", 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5309999999999996e-05 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.280003+0000", 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010239900000000001 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:116409 10.3 10:c7c388dc:::benchmark_data_smithi064_174908_object116408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.135330+0000", 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.769080998, 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14482766799999999, 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116409 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135330+0000", 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135330+0000", 2024-03-24T12:08:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135332+0000", 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135383+0000", 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0902999999999997e-05 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135394+0000", 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0686000000000001e-05 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135401+0000", 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4479999999999997e-06 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.143198+0000", 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077970599999999998 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.143540+0000", 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034143800000000001 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.144004+0000", 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046368700000000001 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.150341+0000", 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063375949999999997 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.280014+0000", 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.129673288 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.280072+0000", 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7744999999999997e-05 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.280158+0000", 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5958000000000006e-05 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:151540 10.5 e343/320 10:a323d81c:::benchmark_data_smithi064_174908_object7123:head v 343'9751, mlcod=343'9749)", 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:15.930468+0000", 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.973943131999999, 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036543720000000002, 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.930468+0000", 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.930468+0000", 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.930470+0000", 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4720000000000001e-06 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.930477+0000", 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9789999999999996e-06 2024-03-24T12:08:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.930490+0000", 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3591000000000001e-05 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.930498+0000", 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.588e-06 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.939585+0000", 2024-03-24T12:08:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090872309999999994 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.939615+0000", 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9649999999999999e-05 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.966102+0000", 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026487295000000001 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:15.967012+0000", 2024-03-24T12:08:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090991399999999995 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:153837 10.5 e345/320 10:a0a89e11:::benchmark_data_smithi064_174908_object9420:head v 345'9920, mlcod=345'9913)", 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:17.595237+0000", 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.309174795000001, 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040956147999999998, 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.595237+0000", 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.595237+0000", 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.595238+0000", 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.486e-06 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.595246+0000", 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3629999999999996e-06 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.595275+0000", 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9907e-05 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.595283+0000", 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8140000000000001e-06 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.600662+0000", 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053791469999999999 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.600680+0000", 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8019000000000001e-05 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.635685+0000", 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035004204999999997 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:39.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.636193+0000", 2024-03-24T12:08:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050820699999999999 2024-03-24T12:08:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:39.026 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:39.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:153839 10.5 e345/320 10:a9808ced:::benchmark_data_smithi064_174908_object9422:head v 345'9921, mlcod=345'9913)", 2024-03-24T12:08:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:17.599777+0000", 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.304634445000001, 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036724482000000003, 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.599777+0000", 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.599777+0000", 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.599778+0000", 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.221e-06 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.599785+0000", 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.599794+0000", 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8279999999999992e-06 2024-03-24T12:08:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.599801+0000", 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.007e-06 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.606553+0000", 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067523770000000004 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.606609+0000", 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6236000000000002e-05 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.636199+0000", 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029589969000000001 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:17.636502+0000", 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030208500000000001 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:176786 10.3 10:c36c4db1:::benchmark_data_smithi064_174908_object32369:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:36.141359+0000", 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.7630529159999999, 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.021666908, 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176786 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141359+0000", 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141359+0000", 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141361+0000", 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3269999999999999e-06 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141368+0000", 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6109999999999999e-06 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141383+0000", 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5525000000000001e-05 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141395+0000", 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2238e-05 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141640+0000", 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244532 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141651+0000", 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1308000000000001e-05 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.148627+0000", 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0069758929999999 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.148720+0000", 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3406999999999995e-05 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5", 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.148817+0000", 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6154000000000001e-05 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.152066+0000", 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032490890000000001 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.154076+0000", 2024-03-24T12:08:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020105990000000001 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.162885+0000", 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088089559999999997 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.162927+0000", 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2169999999999998e-05 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.163025+0000", 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8098999999999997e-05 2024-03-24T12:08:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:176800 10.3 10:c90e6f21:::benchmark_data_smithi064_174908_object32383:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:36.141870+0000", 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.7625411390000001, 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.021365817, 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176800 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141870+0000", 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141870+0000", 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141872+0000", 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.485e-06 2024-03-24T12:08:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141877+0000", 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0320000000000003e-06 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141888+0000", 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1093e-05 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141896+0000", 2024-03-24T12:08:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2709999999999992e-06 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141949+0000", 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3224999999999999e-05 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141960+0000", 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543999999999999e-05 2024-03-24T12:08:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.150568+0000", 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.008608478 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.150705+0000", 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013608899999999999 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5", 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.150849+0000", 2024-03-24T12:08:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014446000000000001 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.152226+0000", 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001376927 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.154185+0000", 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019586059999999999 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.163081+0000", 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088962069999999997 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.163140+0000", 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8943999999999997e-05 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.163236+0000", 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6455999999999999e-05 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:08:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:08:39.044 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:08:39.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:39.166 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:39.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:39.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:39.300 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:39.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:39 smithi064 ceph-mon[42243]: pgmap v748: 168 pgs: 5 remapped+peering, 1 active+clean+scrubbing, 8 active+undersized+degraded, 26 active+undersized, 128 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 7092/337743 objects degraded (2.100%) 2024-03-24T12:08:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:39 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:39 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T12:08:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:39 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T12:08:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:39 smithi064 ceph-mon[30223]: pgmap v748: 168 pgs: 5 remapped+peering, 1 active+clean+scrubbing, 8 active+undersized+degraded, 26 active+undersized, 128 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 7092/337743 objects degraded (2.100%) 2024-03-24T12:08:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:39 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:39.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:39 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T12:08:39.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:39 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T12:08:39.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:08:39.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:39.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:39.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:39.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:39 smithi193 ceph-mon[33593]: pgmap v748: 168 pgs: 5 remapped+peering, 1 active+clean+scrubbing, 8 active+undersized+degraded, 26 active+undersized, 128 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 7092/337743 objects degraded (2.100%) 2024-03-24T12:08:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:39 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:39 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T12:08:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:39 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T12:08:39.593 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:39.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:39.695 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:39.742 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:39.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:39.797 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:08:39.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:39.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:39.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:40.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:40.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:40.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:40.149 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:40.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:40.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:40.277 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:40.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:40.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:40.415 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:40.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:40.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:40.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:40 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T12:08:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:40 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:08:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:40 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:08:40.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:40 smithi064 ceph-mon[30223]: 6.1f scrub ok 2024-03-24T12:08:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:40 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T12:08:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:40 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:08:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:40 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:08:40.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:40 smithi064 ceph-mon[42243]: 6.1f scrub ok 2024-03-24T12:08:40.555 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:40.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:40 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T12:08:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:40 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:08:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:40 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:08:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:40 smithi193 ceph-mon[33593]: 6.1f scrub ok 2024-03-24T12:08:40.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:40.689 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:40.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:40.790 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:40.824 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:40.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:40.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:08:40.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:41.032 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:41.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:41.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:41.181 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:41.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:41.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:41.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:41 smithi193 ceph-mon[33593]: pgmap v749: 168 pgs: 5 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 956 B/s rd, 0 B/s wr, 806 op/s; 6686/335811 objects degraded (1.991%); 1.8 MiB/s, 28 objects/s recovering 2024-03-24T12:08:41.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:41 smithi193 ceph-mon[33593]: 2.1e scrub starts 2024-03-24T12:08:41.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:41 smithi193 ceph-mon[33593]: 2.1e scrub ok 2024-03-24T12:08:41.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:41 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:08:41.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:41 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:08:41.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:41 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T12:08:41.334 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:41 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T12:08:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:41.344 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:41.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:41.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:41.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[30223]: pgmap v749: 168 pgs: 5 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 956 B/s rd, 0 B/s wr, 806 op/s; 6686/335811 objects degraded (1.991%); 1.8 MiB/s, 28 objects/s recovering 2024-03-24T12:08:41.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[30223]: 2.1e scrub starts 2024-03-24T12:08:41.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[30223]: 2.1e scrub ok 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[42243]: pgmap v749: 168 pgs: 5 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 956 B/s rd, 0 B/s wr, 806 op/s; 6686/335811 objects degraded (1.991%); 1.8 MiB/s, 28 objects/s recovering 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[42243]: 2.1e scrub starts 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[42243]: 2.1e scrub ok 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:08:41.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T12:08:41.453 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:41 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T12:08:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:41.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:41.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:41.631 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:41.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:41.665 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:41.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:41.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:41.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:41.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:41.908 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:41.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:42.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:42.047 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:42.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:42.149 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:42.185 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:42.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:42.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:42.330 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:42.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:42.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:42.481 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:42 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:08:42.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:42 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:08:42.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:42 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 6686/335811 objects degraded (1.991%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:08:42.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:42.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:42.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4292442984' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:08:42.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:42.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:42.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:08:42.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:08:42.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 6686/335811 objects degraded (1.991%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:08:42.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:42.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:42.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4292442984' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:08:42.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:08:42.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:08:42.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 6686/335811 objects degraded (1.991%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:08:42.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:42.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:42.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4292442984' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:08:42.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:42.626 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:42.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:42.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:42.766 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:42.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:42.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:42.899 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:42.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:42.993 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:08:43.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:43.030 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:43.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.15714.0:183650 10.9 e355/354 10:9c22a781:::benchmark_data_smithi064_174908_object39233:head v 355'11371, mlcod=355'11369)", 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:43.044539+0000", 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.001459775, 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014920350000000001, 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.046 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.044539+0000", 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.044539+0000", 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.044541+0000", 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2709999999999999e-06 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.047 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.044552+0000", 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0801999999999999e-05 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.044573+0000", 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1413e-05 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.044583+0000", 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8740000000000007e-06 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.048 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.044667+0000", 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3592999999999997e-05 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.044691+0000", 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4221000000000001e-05 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.15714.0:183654 10.6 e355/354 10:66854ff6:::benchmark_data_smithi064_174908_object39237:head v 355'11238, mlcod=355'11236)", 2024-03-24T12:08:43.049 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:43.045109+0000", 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.00088931100000000005, 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00098937000000000005, 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.045109+0000", 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.045109+0000", 2024-03-24T12:08:43.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.045111+0000", 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5549999999999999e-06 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.045119+0000", 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5730000000000008e-06 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.045131+0000", 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1647e-05 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.045139+0000", 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6869999999999994e-06 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.045218+0000", 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.941e-05 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:43.045239+0000", 2024-03-24T12:08:43.053 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0404e-05 2024-03-24T12:08:43.053 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:43.053 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:43.053 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:43.053 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:43.053 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T12:08:43.053 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 2 2024-03-24T12:08:43.053 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:08:43.132 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:43.174 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:43.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:43.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: pgmap v750: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 872 op/s; 25937/332157 objects degraded (7.809%); 7.6 MiB/s, 121 objects/s recovering 2024-03-24T12:08:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T12:08:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:08:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:08:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:08:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4292442984' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[30223]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: pgmap v750: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 872 op/s; 25937/332157 objects degraded (7.809%); 7.6 MiB/s, 121 objects/s recovering 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:08:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4292442984' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:08:43.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:43 smithi064 ceph-mon[42243]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T12:08:43.319 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:43.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:43.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:43.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:43.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:43.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: pgmap v750: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 872 op/s; 25937/332157 objects degraded (7.809%); 7.6 MiB/s, 121 objects/s recovering 2024-03-24T12:08:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T12:08:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:08:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:08:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:08:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T12:08:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:08:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4292442984' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:08:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:43 smithi193 ceph-mon[33593]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T12:08:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:43.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:43.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:43.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:43.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:43.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:43.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:43.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:43.874 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:43.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:43.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:44.009 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:44.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:44.028 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:08:44.054 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:08:44.087 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:08:44.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:44.127 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:08:44.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:44.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:44.170 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:08:44.209 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:08:44.242 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:08:44.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:44.277 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:08:44.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:08:44.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:08:44.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:08:44.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:08:44.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:08:44.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:08:44.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:08:44.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:08:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:44.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:44.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:44.414 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:44.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:44.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:44.538 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:44.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T12:08:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T12:08:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: 3.2 scrub starts 2024-03-24T12:08:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: 3.2 scrub ok 2024-03-24T12:08:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T12:08:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1188400773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: 3.2 scrub starts 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: 3.2 scrub ok 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:08:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1188400773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T12:08:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T12:08:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: 3.2 scrub starts 2024-03-24T12:08:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: 3.2 scrub ok 2024-03-24T12:08:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T12:08:44.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T12:08:44.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:08:44.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:08:44.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1188400773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:44.640 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:44.667 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:44.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:44.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:44.802 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:44.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:44.903 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:44.926 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:44.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:45.027 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:45.053 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:45.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:45.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:45.191 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:45.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:45.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:45.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:45.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:45.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:45.455 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:45.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:45.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:45 smithi064 ceph-mon[30223]: pgmap v752: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.29k op/s; 37627/320241 objects degraded (11.750%); 17 MiB/s, 278 objects/s recovering 2024-03-24T12:08:45.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:45 smithi064 ceph-mon[30223]: 1.9 scrub starts 2024-03-24T12:08:45.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:45 smithi064 ceph-mon[30223]: 1.9 scrub ok 2024-03-24T12:08:45.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:45 smithi064 ceph-mon[42243]: pgmap v752: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.29k op/s; 37627/320241 objects degraded (11.750%); 17 MiB/s, 278 objects/s recovering 2024-03-24T12:08:45.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:45 smithi064 ceph-mon[42243]: 1.9 scrub starts 2024-03-24T12:08:45.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:45 smithi064 ceph-mon[42243]: 1.9 scrub ok 2024-03-24T12:08:45.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:45 smithi193 ceph-mon[33593]: pgmap v752: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.29k op/s; 37627/320241 objects degraded (11.750%); 17 MiB/s, 278 objects/s recovering 2024-03-24T12:08:45.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:45 smithi193 ceph-mon[33593]: 1.9 scrub starts 2024-03-24T12:08:45.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:45 smithi193 ceph-mon[33593]: 1.9 scrub ok 2024-03-24T12:08:45.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:45.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:45.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:45.774 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:45.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:45.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:45.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:45.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:46.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:46.042 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:46.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:46.181 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:46.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:46.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:46.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:46.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:46.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:46.592 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:46.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:46.743 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:46.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:46.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:46.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:47.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:08:47.018 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:47.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:47.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:47.148 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:47.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:47.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:47.299 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:47.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[42243]: pgmap v753: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 597 B/s rd, 0 B/s wr, 738 op/s; 37627/320241 objects degraded (11.750%); 15 MiB/s, 239 objects/s recovering 2024-03-24T12:08:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[42243]: 5.9 deep-scrub starts 2024-03-24T12:08:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[42243]: 5.9 deep-scrub ok 2024-03-24T12:08:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:08:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:08:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 37627/320241 objects degraded (11.750%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[42243]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T12:08:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[30223]: pgmap v753: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 597 B/s rd, 0 B/s wr, 738 op/s; 37627/320241 objects degraded (11.750%); 15 MiB/s, 239 objects/s recovering 2024-03-24T12:08:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[30223]: 5.9 deep-scrub starts 2024-03-24T12:08:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[30223]: 5.9 deep-scrub ok 2024-03-24T12:08:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:08:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:08:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 37627/320241 objects degraded (11.750%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:47.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-mon[30223]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T12:08:47.400 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:47.435 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:47.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:47.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:47.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:47 smithi193 ceph-mon[33593]: pgmap v753: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 597 B/s rd, 0 B/s wr, 738 op/s; 37627/320241 objects degraded (11.750%); 15 MiB/s, 239 objects/s recovering 2024-03-24T12:08:47.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:47 smithi193 ceph-mon[33593]: 5.9 deep-scrub starts 2024-03-24T12:08:47.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:47 smithi193 ceph-mon[33593]: 5.9 deep-scrub ok 2024-03-24T12:08:47.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:47 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:08:47.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:47 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:08:47.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:47 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 37627/320241 objects degraded (11.750%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:47.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:47 smithi193 ceph-mon[33593]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T12:08:47.572 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:47.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:47.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:47.711 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:47.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:47.772 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:08:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:08:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:47.813 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:47.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:08:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:08:47.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:08:47.848 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:47.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:47.949 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:47.987 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:47.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:48.089 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:48.144 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:48.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:48.246 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:48.267 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:48 smithi193 ceph-mon[33593]: 4.19 scrub starts 2024-03-24T12:08:48.267 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:48 smithi193 ceph-mon[33593]: 4.19 scrub ok 2024-03-24T12:08:48.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:48.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:48.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:48.438 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:48.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:48.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:48 smithi064 ceph-mon[30223]: 4.19 scrub starts 2024-03-24T12:08:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:48 smithi064 ceph-mon[30223]: 4.19 scrub ok 2024-03-24T12:08:48.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:48 smithi064 ceph-mon[42243]: 4.19 scrub starts 2024-03-24T12:08:48.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:48 smithi064 ceph-mon[42243]: 4.19 scrub ok 2024-03-24T12:08:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:48.577 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:48.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:48.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:48.717 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:48.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:48.780 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:08:48.780 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:08:48.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:48.857 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:48.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:48.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:48.987 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:48.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:49.089 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:49.119 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:49.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:49.220 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:49.259 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:49.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:49.336 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:49.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:49.410 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:49.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:49.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:49 smithi064 ceph-mon[30223]: pgmap v755: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 34522/305904 objects degraded (11.285%); 26 MiB/s, 411 objects/s recovering 2024-03-24T12:08:49.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:49.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:49 smithi064 ceph-mon[30223]: 10.c scrub starts 2024-03-24T12:08:49.485 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:49 smithi064 ceph-mon[42243]: pgmap v755: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 34522/305904 objects degraded (11.285%); 26 MiB/s, 411 objects/s recovering 2024-03-24T12:08:49.485 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:49.485 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:49 smithi064 ceph-mon[42243]: 10.c scrub starts 2024-03-24T12:08:49.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:49.549 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:49.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:49 smithi193 ceph-mon[33593]: pgmap v755: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 34522/305904 objects degraded (11.285%); 26 MiB/s, 411 objects/s recovering 2024-03-24T12:08:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:49 smithi193 ceph-mon[33593]: 10.c scrub starts 2024-03-24T12:08:49.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:49.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:08:49.688 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:49.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:49.789 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:49.827 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:49.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:49.928 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:49.973 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:49.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:50.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:50.119 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:50.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:50.220 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:50.269 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:50.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:50.290 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:08:50.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:50.406 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:50.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:50.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:50.546 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:50.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:50.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:50 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T12:08:50.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:50 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T12:08:50.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:50 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T12:08:50.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:50 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T12:08:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:50 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T12:08:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:50 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T12:08:50.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:50.695 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:50.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:50.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:50.832 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:50.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:50.930 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:08:50.930 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:08:50.930 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-24T12:08:50.930 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.902539 2024-03-24T12:08:50.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 2 0.9025388472584885 2024-03-24T12:08:50.933 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:50.974 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:50.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:51.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:51.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:51.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:51.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:51.269 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:08:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:51.272 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:51.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:51.373 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:51.409 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:51.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:51.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:51.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[30223]: pgmap v756: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 33492/302721 objects degraded (11.064%); 27 MiB/s, 428 objects/s recovering 2024-03-24T12:08:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[30223]: 10.5 scrub starts 2024-03-24T12:08:51.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[30223]: 10.c scrub ok 2024-03-24T12:08:51.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3505606048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:08:51.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[30223]: from='client.25690 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:08:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[42243]: pgmap v756: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 33492/302721 objects degraded (11.064%); 27 MiB/s, 428 objects/s recovering 2024-03-24T12:08:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[42243]: 10.5 scrub starts 2024-03-24T12:08:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[42243]: 10.c scrub ok 2024-03-24T12:08:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3505606048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:08:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:51 smithi064 ceph-mon[42243]: from='client.25690 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:08:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:51 smithi193 ceph-mon[33593]: pgmap v756: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 33492/302721 objects degraded (11.064%); 27 MiB/s, 428 objects/s recovering 2024-03-24T12:08:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:51 smithi193 ceph-mon[33593]: 10.5 scrub starts 2024-03-24T12:08:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:51 smithi193 ceph-mon[33593]: 10.c scrub ok 2024-03-24T12:08:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:51 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3505606048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:08:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:51 smithi193 ceph-mon[33593]: from='client.25690 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:08:51.649 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:51.686 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:51.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:51.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:51.821 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:51.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:51.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:51.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:52.087 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:52.124 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:52.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:52.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:52.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:52.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:52.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:52.399 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:52.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:52.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:52.532 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:52.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[30223]: from='client.25690 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:08:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[30223]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T12:08:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[30223]: pgmap v758: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 912 op/s; 31754/298347 objects degraded (10.643%); 23 MiB/s, 363 objects/s recovering 2024-03-24T12:08:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:08:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 33492/302721 objects degraded (11.064%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[30223]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T12:08:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[42243]: from='client.25690 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:08:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[42243]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T12:08:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[42243]: pgmap v758: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 912 op/s; 31754/298347 objects degraded (10.643%); 23 MiB/s, 363 objects/s recovering 2024-03-24T12:08:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:08:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 33492/302721 objects degraded (11.064%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:52.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:52 smithi064 ceph-mon[42243]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T12:08:52.576 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:52 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:08:52.576 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:52 smithi193 ceph-mon[33593]: from='client.25690 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:08:52.576 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:52 smithi193 ceph-mon[33593]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T12:08:52.576 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:52 smithi193 ceph-mon[33593]: pgmap v758: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 912 op/s; 31754/298347 objects degraded (10.643%); 23 MiB/s, 363 objects/s recovering 2024-03-24T12:08:52.576 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:52 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:08:52.576 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:52 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 33492/302721 objects degraded (11.064%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:52.576 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:52 smithi193 ceph-mon[33593]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T12:08:52.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:52.669 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:52.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:52.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:52.821 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:52.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:52.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:52.954 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:52.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:53.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:53.100 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:53.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:53.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:53.235 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:53.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:53.331 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:08:53.331 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:08:53.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:53.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:53.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:53.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:53.510 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:53.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:53.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:53.647 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:53.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:53.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:53.786 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:53.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:53.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:53.888 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:53.923 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:53.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:54.024 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:54.079 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:08:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:54.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:54.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:54.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:54.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:54.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:54.381 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:54.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:54.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:54.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:54.515 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:54.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:54.518 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:08:54.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:54.654 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:54.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:54.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:54.789 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:54.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:54.796 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:54 smithi193 ceph-mon[33593]: pgmap v760: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 1.13k op/s; 29504/285888 objects degraded (10.320%); 27 MiB/s, 436 objects/s recovering 2024-03-24T12:08:54.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:54 smithi064 ceph-mon[30223]: pgmap v760: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 1.13k op/s; 29504/285888 objects degraded (10.320%); 27 MiB/s, 436 objects/s recovering 2024-03-24T12:08:54.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:54 smithi064 ceph-mon[42243]: pgmap v760: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 1.13k op/s; 29504/285888 objects degraded (10.320%); 27 MiB/s, 436 objects/s recovering 2024-03-24T12:08:54.891 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:54.931 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:54.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:55.033 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:55.077 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:55.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:55.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:55.216 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:55.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:55.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:55.352 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:55.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:55.454 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:55.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:55.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:55.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:08:55.623 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:55.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:08:55.725 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:55.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:55.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:55.869 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:55.905 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:55.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:56.007 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:56.047 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:56.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:56.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:56.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:56.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:56.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:56.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:56.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:56.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:56.461 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:56.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:56.562 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:56.599 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:56.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:56.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:56.737 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:56.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:56.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:56 smithi064 ceph-mon[30223]: pgmap v761: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 834 op/s; 29504/285888 objects degraded (10.320%); 22 MiB/s, 358 objects/s recovering 2024-03-24T12:08:56.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:56 smithi064 ceph-mon[42243]: pgmap v761: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 834 op/s; 29504/285888 objects degraded (10.320%); 22 MiB/s, 358 objects/s recovering 2024-03-24T12:08:56.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:56 smithi193 ceph-mon[33593]: pgmap v761: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 834 op/s; 29504/285888 objects degraded (10.320%); 22 MiB/s, 358 objects/s recovering 2024-03-24T12:08:56.838 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:56.879 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:56.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:56.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:57.015 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:57.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:57.117 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:57.154 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:57.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:57.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:57.295 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:57.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:57.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:57.444 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:57.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:57.546 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:57.583 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:57.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:57.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:57.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:08:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:57.714 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:57.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:57.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:57 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 29504/285888 objects degraded (10.320%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:57.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:57.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:57.731 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:08:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:08:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:57.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:57 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 29504/285888 objects degraded (10.320%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:57.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:57.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:57.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:57.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:08:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:08:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:57 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 29504/285888 objects degraded (10.320%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:08:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:08:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:57.846 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:57.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:57.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:57.969 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:57.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:58.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:58.104 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:58.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:58.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:58.246 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:58.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:58.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:08:58.386 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:58.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:08:58.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:08:58.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:08:58.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:08:58.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:08:58.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:08:58.413 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:08:58.440 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:08:58.477 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:08:58.488 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:08:58.503 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:08:58.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:08:58.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:08:58.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:08:58.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:08:58.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:08:58.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:08:58.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:08:58.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:08:58.526 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:58.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:58.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:08:58.627 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:08:58.658 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:58.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:08:58.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:08:58.804 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:58.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:58.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:08:58.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:58.944 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:58.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:59.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:59.078 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:59.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:59.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:59.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:59.218 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:59.219 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:59 smithi193 ceph-mon[33593]: pgmap v762: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 26550/271413 objects degraded (9.782%); 27 MiB/s, 431 objects/s recovering 2024-03-24T12:08:59.219 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:59.219 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:08:59 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1873825165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:59.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:59.321 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:08:59.357 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:59.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:08:59.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:59.499 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:59.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:59.513 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:59 smithi064 ceph-mon[30223]: pgmap v762: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 26550/271413 objects degraded (9.782%); 27 MiB/s, 431 objects/s recovering 2024-03-24T12:08:59.513 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:59.513 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:08:59 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1873825165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:59.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:59 smithi064 ceph-mon[42243]: pgmap v762: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 26550/271413 objects degraded (9.782%); 27 MiB/s, 431 objects/s recovering 2024-03-24T12:08:59.514 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:59.514 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:08:59 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1873825165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:08:59.553 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:08:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T12:08:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:35816 8.2 8:48b7c5af:::benchmark_data_smithi064_120708_object35815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:15.851911+0000", 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 583.303569176, 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.12931609299999999, 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 35816 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851911+0000", 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851911+0000", 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851913+0000", 2024-03-24T12:08:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8300000000000001e-06 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851965+0000", 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2689000000000003e-05 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851976+0000", 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1001e-05 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.851982+0000", 2024-03-24T12:08:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8289999999999996e-06 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852025+0000", 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2972000000000001e-05 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852110+0000", 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4437000000000005e-05 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.852190+0000", 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.038e-05 2024-03-24T12:08:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.853775+0000", 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0015849149999999999 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.853958+0000", 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018299500000000001 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.981109+0000", 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.127151032 2024-03-24T12:08:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.981170+0000", 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0961000000000001e-05 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:15.981227+0000", 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7052000000000002e-05 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:61320 8.2 8:43c827b4:::benchmark_data_smithi064_120708_object61319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:32.665860+0000", 2024-03-24T12:08:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 566.48962005299995, 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.119858833, 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 61320 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.665860+0000", 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.665860+0000", 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.665863+0000", 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8260000000000001e-06 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.665903+0000", 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0735999999999998e-05 2024-03-24T12:08:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.665925+0000", 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1877000000000001e-05 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.665936+0000", 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0891e-05 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.666018+0000", 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2118000000000002e-05 2024-03-24T12:08:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.666211+0000", 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019257899999999999 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.666365+0000", 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000153821 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.669023+0000", 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026581159999999999 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.669128+0000", 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000105438 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.785651+0000", 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.11652228100000001 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.785688+0000", 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7628e-05 2024-03-24T12:08:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:32.785719+0000", 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0521999999999997e-05 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:72995 8.2 8:402b8c35:::benchmark_data_smithi064_120708_object72994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.015532+0000", 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 557.13994821699998, 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14821208299999999, 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 72995 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.015532+0000", 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.015532+0000", 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.015534+0000", 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.148e-06 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.015683+0000", 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014912899999999999 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.015697+0000", 2024-03-24T12:08:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3602e-05 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.015705+0000", 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7860000000000002e-06 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.015767+0000", 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1985999999999999e-05 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.015896+0000", 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000129004 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.016024+0000", 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012765899999999999 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023406+0000", 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0073816949999999997 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023432+0000", 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6478e-05 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163643+0000", 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14021113099999999 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163701+0000", 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8112000000000002e-05 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163744+0000", 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2352999999999999e-05 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:72999 8.2 8:46ada980:::benchmark_data_smithi064_120708_object72998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T11:59:42.019726+0000", 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 557.13575383700004, 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.144168888, 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 72999 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019726+0000", 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019726+0000", 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019731+0000", 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7539999999999997e-06 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019785+0000", 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3897000000000003e-05 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019806+0000", 2024-03-24T12:08:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1552e-05 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.019816+0000", 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0132999999999999e-05 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.020087+0000", 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027032899999999998 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.020197+0000", 2024-03-24T12:08:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000110339 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.020322+0000", 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000125239 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.023538+0000", 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003215529 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.026626+0000", 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030884580000000001 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163801+0000", 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13717497200000001 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163855+0000", 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3483000000000003e-05 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T11:59:42.163895+0000", 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0203000000000001e-05 2024-03-24T12:08:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:137370 8.2 8:42eec382:::benchmark_data_smithi064_120708_object15344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:44.772433+0000", 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 494.38304735200001, 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.112467682, 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 137370 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.772433+0000", 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.772433+0000", 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.772434+0000", 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3200000000000001e-06 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.772441+0000", 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8750000000000002e-06 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.772452+0000", 2024-03-24T12:08:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1334e-05 2024-03-24T12:08:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.772460+0000", 2024-03-24T12:08:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0069999999999997e-06 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.773906+0000", 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001445911 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.774018+0000", 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000112441 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.774286+0000", 2024-03-24T12:08:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0002675 2024-03-24T12:08:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.779973+0000", 2024-03-24T12:08:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0056866379999999999 2024-03-24T12:08:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.784237+0000", 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042638770000000001 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.884768+0000", 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.10053173899999999 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.884852+0000", 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3489e-05 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.884900+0000", 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8551000000000003e-05 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:137373 8.2 8:4279fd42:::benchmark_data_smithi064_120708_object15347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:44.774165+0000", 2024-03-24T12:08:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 494.381314621, 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.111045148, 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 137373 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.774165+0000", 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.774165+0000", 2024-03-24T12:08:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.774167+0000", 2024-03-24T12:08:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2049999999999999e-06 2024-03-24T12:08:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.774170+0000", 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3050000000000001e-06 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.774178+0000", 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8210000000000002e-06 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.774184+0000", 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9730000000000002e-06 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.776119+0000", 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001934912 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.776229+0000", 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000110895 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.776489+0000", 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025976699999999999 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.779978+0000", 2024-03-24T12:08:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003489172 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.784259+0000", 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042801580000000001 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.885081+0000", 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.100822643 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.885164+0000", 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3307999999999996e-05 2024-03-24T12:08:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.885210+0000", 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5989000000000003e-05 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35033.0:137379 8.2 8:459037ef:::benchmark_data_smithi064_120708_object15353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:00:44.777189+0000", 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 494.37829112999998, 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.108286804, 2024-03-24T12:08:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35033", 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 137379 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.777189+0000", 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.777189+0000", 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.777191+0000", 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9039999999999999e-06 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.777195+0000", 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7550000000000002e-06 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.777205+0000", 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8959999999999998e-06 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.777214+0000", 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1429999999999998e-06 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.780257+0000", 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030436819999999998 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.780403+0000", 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014552900000000001 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.780769+0000", 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00036587799999999998 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.784265+0000", 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034965780000000002 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.784588+0000", 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00032233099999999997 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.885378+0000", 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.100790457 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.885436+0000", 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7677999999999999e-05 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:00:44.885476+0000", 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9972999999999998e-05 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35306.0:60099 9.4 9:2a79c4dd:::benchmark_data_smithi064_148274_object60098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:03:26.530769+0000", 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 332.62471107900001, 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13911789999999999, 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35306", 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 60099 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.530769+0000", 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.530769+0000", 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.530771+0000", 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8729999999999999e-06 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.530816+0000", 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5606999999999999e-05 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.530828+0000", 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.137e-05 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.530835+0000", 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2300000000000002e-06 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.530893+0000", 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7982000000000002e-05 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.531027+0000", 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000134177 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.531226+0000", 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019901599999999999 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.533154+0000", 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0019276899999999999 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.533251+0000", 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7468999999999998e-05 2024-03-24T12:08:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.669749+0000", 2024-03-24T12:08:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.136498069 2024-03-24T12:08:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.669827+0000", 2024-03-24T12:08:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7540000000000006e-05 2024-03-24T12:08:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:03:26.669887+0000", 2024-03-24T12:08:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9877000000000003e-05 2024-03-24T12:08:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35306.0:130304 9.4 9:2cfbc72c:::benchmark_data_smithi064_148274_object4491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:04:31.560488+0000", 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 267.59499189399997, 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.108788596, 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35306", 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 130304 2024-03-24T12:08:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560488+0000", 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560488+0000", 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560490+0000", 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1720000000000001e-06 2024-03-24T12:08:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560497+0000", 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9439999999999999e-06 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560512+0000", 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5165e-05 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560522+0000", 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6539999999999994e-06 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560582+0000", 2024-03-24T12:08:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9786e-05 2024-03-24T12:08:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560734+0000", 2024-03-24T12:08:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015266300000000001 2024-03-24T12:08:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.560863+0000", 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012833199999999999 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.563608+0000", 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0027451200000000002 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.566824+0000", 2024-03-24T12:08:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032165879999999998 2024-03-24T12:08:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.669166+0000", 2024-03-24T12:08:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.10234103999999999 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.669230+0000", 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4760999999999999e-05 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:04:31.669277+0000", 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6371e-05 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35306.0:175954 9.4 9:2ddfe4c1:::benchmark_data_smithi064_148274_object50141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:05:06.084115+0000", 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 233.071364585, 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17091791200000001, 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35306", 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 175954 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084115+0000", 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084115+0000", 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084117+0000", 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0490000000000002e-06 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084123+0000", 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5690000000000004e-06 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084138+0000", 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4790000000000001e-05 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084147+0000", 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1160000000000005e-06 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084610+0000", 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00046276800000000001 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084718+0000", 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010838800000000001 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.084834+0000", 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000115603 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.614 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.087334+0000", 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024999190000000002 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.095660+0000", 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083259939999999998 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.254956+0000", 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15929689899999999 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.254992+0000", 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5101000000000002e-05 2024-03-24T12:08:59.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.255033+0000", 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1715999999999998e-05 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35306.0:175977 9.4 9:28b33ece:::benchmark_data_smithi064_148274_object50164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:05:06.106966+0000", 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 233.04851408499999, 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.150183125, 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.616 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35306", 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 175977 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.106966+0000", 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.106966+0000", 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.106968+0000", 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.367e-06 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.106973+0000", 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9919999999999998e-06 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.106984+0000", 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0342e-05 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.618 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.106990+0000", 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2559999999999999e-06 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.107049+0000", 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9360000000000001e-05 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.107186+0000", 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013634899999999999 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.107301+0000", 2024-03-24T12:08:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000115776 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.109705+0000", 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024032390000000002 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.109820+0000", 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000115777 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.256999+0000", 2024-03-24T12:08:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.14717871099999999 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.257102+0000", 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000103175 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:05:06.257149+0000", 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6780999999999998e-05 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.621 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24379.0:3049 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94338803869824 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:06:26.485490+0000", 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 152.66998970700001, 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.528722833, 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24379", 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3049 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.622 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:26.485490+0000", 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:26.485490+0000", 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:26.485494+0000", 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9349999999999996e-06 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:26.485505+0000", 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1303e-05 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:26.485519+0000", 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3742e-05 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.624 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:26.485532+0000", 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2609000000000001e-05 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:26.485600+0000", 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8548999999999997e-05 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:26.485610+0000", 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5880000000000004e-06 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.625 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:27.011504+0000", 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.52589372499999998 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:27.011640+0000", 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013594400000000001 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:06:27.014213+0000", 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025734379999999999 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:154581 10.1 10:846ee239:::benchmark_data_smithi064_174908_object10164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:18.225232+0000", 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 40.930247663999999, 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.107035724, 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 154581 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.627 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.225232+0000", 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.225232+0000", 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.225233+0000", 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.009e-06 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.225237+0000", 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1799999999999998e-06 2024-03-24T12:08:59.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.225248+0000", 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0285999999999999e-05 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.225255+0000", 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.934e-06 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.228810+0000", 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0035552980000000001 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.228927+0000", 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000116606 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.228999+0000", 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2785000000000002e-05 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.231030+0000", 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002030245 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.630 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.231681+0000", 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00065116700000000002 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.332170+0000", 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.100489669 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.332226+0000", 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5639e-05 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.631 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.332268+0000", 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1906000000000002e-05 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24379.0:3314 5.e 5:7759931f:::notify.3:head [watch ping cookie 94338803869824 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.391067+0000", 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 23.764413246, 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.78275026999999997, 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24379", 2024-03-24T12:08:59.632 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3314 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391067+0000", 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391067+0000", 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391068+0000", 2024-03-24T12:08:59.633 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2399999999999997e-07 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391071+0000", 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0359999999999998e-06 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391077+0000", 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3829999999999997e-06 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391085+0000", 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7319999999999998e-06 2024-03-24T12:08:59.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391225+0000", 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014055799999999999 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391240+0000", 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4735e-05 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.173541+0000", 2024-03-24T12:08:59.635 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.78230145100000004 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.173726+0000", 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018453299999999999 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.173817+0000", 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1018000000000002e-05 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24379.0:3322 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94338803869824 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.391466+0000", 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 23.764013834, 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.78383282399999998, 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24379", 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3322 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391466+0000", 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391466+0000", 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391467+0000", 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.051e-06 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.638 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391471+0000", 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5219999999999999e-06 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391479+0000", 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6789999999999994e-06 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391485+0000", 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4179999999999996e-06 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391535+0000", 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9874999999999999e-05 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.391544+0000", 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0899999999999994e-06 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.173984+0000", 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.78244009199999998 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.174076+0000", 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2126999999999994e-05 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.175299+0000", 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012229700000000001 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176802 10.9 10:9a51c5d9:::benchmark_data_smithi064_174908_object32385:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:36.143045+0000", 2024-03-24T12:08:59.641 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 23.012435135, 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.034723396, 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176802 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.642 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143045+0000", 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143045+0000", 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143048+0000", 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.903e-06 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.643 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143055+0000", 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4540000000000001e-06 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143070+0000", 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5276000000000001e-05 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143085+0000", 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4878999999999999e-05 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.644 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143217+0000", 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013128400000000001 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143226+0000", 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2699999999999993e-06 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.147245+0000", 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040196069999999997 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:08:59.645 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.147256+0000", 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0429e-05 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.165120+0000", 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0178639540000001 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.166984+0000", 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018638140000000001 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.167147+0000", 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000163392 2024-03-24T12:08:59.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.169562+0000", 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024147639999999998 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177615+0000", 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0080527099999999994 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.647 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177655+0000", 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0516999999999998e-05 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177694+0000", 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9369999999999997e-05 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177768+0000", 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3773e-05 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176812 10.9 10:9dafeb35:::benchmark_data_smithi064_174908_object32395:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:36.143114+0000", 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 23.012365725999999, 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.034834289, 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176812 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.649 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143114+0000", 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143114+0000", 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143115+0000", 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3900000000000003e-07 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.650 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143119+0000", 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2139999999999998e-06 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143127+0000", 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7789999999999996e-06 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143134+0000", 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5080000000000002e-06 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143281+0000", 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000147739 2024-03-24T12:08:59.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting_for_map not empty", 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143287+0000", 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4790000000000004e-06 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.147537+0000", 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004250485 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.652 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.147546+0000", 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3760000000000008e-06 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.169595+0000", 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.022049505 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.169698+0000", 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010289300000000001 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.653 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.169826+0000", 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000128341 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177558+0000", 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0077313019999999998 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177646+0000", 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8708000000000005e-05 2024-03-24T12:08:59.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177777+0000", 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000130383 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177841+0000", 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4134999999999995e-05 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177949+0000", 2024-03-24T12:08:59.655 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107503 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176826 10.9 10:924ed710:::benchmark_data_smithi064_174908_object32409:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:36.143143+0000", 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 23.012337206000002, 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.03774744, 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176826 2024-03-24T12:08:59.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143143+0000", 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143143+0000", 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.657 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143144+0000", 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6700000000000003e-07 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143146+0000", 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.836e-06 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143152+0000", 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7679999999999997e-06 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.658 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143158+0000", 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4179999999999996e-06 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143315+0000", 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000157181 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting_for_map not empty", 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143319+0000", 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5689999999999998e-06 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.147766+0000", 2024-03-24T12:08:59.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0044463460000000003 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.147776+0000", 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0728e-05 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.174954+0000", 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0271777769999999 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.175078+0000", 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000123819 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.175212+0000", 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000133922 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.177995+0000", 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002783298 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.661 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.178010+0000", 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4745e-05 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.178295+0000", 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028545300000000002 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.180742+0000", 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002446199 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.662 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:37.180890+0000", 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014861400000000001 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:181366 10.e 10:78356f5f:::benchmark_data_smithi064_174908_object36949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:41.031745+0000", 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 18.123735226000001, 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.11972917600000001, 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:08:59.663 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 181366 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031745+0000", 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031745+0000", 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.664 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031747+0000", 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0949999999999998e-06 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031752+0000", 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6149999999999996e-06 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031764+0000", 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1984000000000001e-05 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031773+0000", 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1349999999999992e-06 2024-03-24T12:08:59.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.032461+0000", 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00068837499999999997 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.106451+0000", 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.073990231000000004 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.106611+0000", 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015964700000000001 2024-03-24T12:08:59.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.111206+0000", 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045954259999999997 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.148990+0000", 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037784259000000001 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.667 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.151261+0000", 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002270762 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.151350+0000", 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9099999999999997e-05 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.151474+0000", 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000123547 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.668 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:181367 10.9 10:9ac76393:::benchmark_data_smithi064_174908_object36950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:41.031792+0000", 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 18.123687521000001, 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.11891956300000001, 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 181367 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031792+0000", 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031792+0000", 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031793+0000", 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3799999999999996e-07 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031797+0000", 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3330000000000001e-06 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.670 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031803+0000", 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0340000000000002e-06 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.031808+0000", 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4e-06 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.033254+0000", 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014458019999999999 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.033371+0000", 2024-03-24T12:08:59.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011699199999999999 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.033494+0000", 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000123113 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.112040+0000", 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.078546427000000002 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.112079+0000", 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8976000000000001e-05 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.150559+0000", 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.038479588000000002 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.150621+0000", 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2512000000000001e-05 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:08:59.673 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:41.150712+0000", 2024-03-24T12:08:59.674 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0648000000000004e-05 2024-03-24T12:08:59.674 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.674 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.674 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.674 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:08:59.674 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:08:59.674 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:08:59.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:08:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:08:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:59.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:08:59.853 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:08:59.893 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:08:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:59.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:08:59.995 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:00.034 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:00.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:00.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:00.172 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:00.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:00.274 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:00.308 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:00.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:00.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:00.442 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:00.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:00.544 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:00.579 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:00.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:00.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:00.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:00.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:00.816 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:00.874 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:00.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:00.876 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:09:00.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:01.105 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.162 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:01.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:01.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:01.313 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:01.318 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:01.359 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:01.419 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:01.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:01 smithi064 ceph-mon[30223]: pgmap v763: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 25713/268317 objects degraded (9.583%); 31 MiB/s, 495 objects/s recovering 2024-03-24T12:09:01.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:01 smithi064 ceph-mon[42243]: pgmap v763: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 25713/268317 objects degraded (9.583%); 31 MiB/s, 495 objects/s recovering 2024-03-24T12:09:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.465 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:01.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:01.501 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:01 smithi193 ceph-mon[33593]: pgmap v763: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 25713/268317 objects degraded (9.583%); 31 MiB/s, 495 objects/s recovering 2024-03-24T12:09:01.566 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:01.605 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:01.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:09:01.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:01.744 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:01.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.891 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:01.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:01.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:02.030 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:02.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:02.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:02.171 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:02.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:02.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:02 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 25713/268317 objects degraded (9.583%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:02.263 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:02 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 25713/268317 objects degraded (9.583%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:02.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:02.331 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:02.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:02.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:02.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:02.515 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:02 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 25713/268317 objects degraded (9.583%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:02.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:02.604 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:02.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:02.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:02.741 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:02.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:02.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:02.885 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:02.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:02.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:03.011 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:03.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:03.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.2 primary-affinity to 0.902539 (8591482) 2024-03-24T12:09:03.139 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:03.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:03.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:03.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:03.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:03.404 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:03.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:03.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:03.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:03.543 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:03.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:03.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:03 smithi064 ceph-mon[30223]: pgmap v764: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 25354/267240 objects degraded (9.487%); 21 MiB/s, 331 objects/s recovering 2024-03-24T12:09:03.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:03 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2276042756' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.9025388472584885}]: dispatch 2024-03-24T12:09:03.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:03 smithi064 ceph-mon[42243]: pgmap v764: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 25354/267240 objects degraded (9.487%); 21 MiB/s, 331 objects/s recovering 2024-03-24T12:09:03.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:03 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2276042756' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.9025388472584885}]: dispatch 2024-03-24T12:09:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:03 smithi193 ceph-mon[33593]: pgmap v764: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 25354/267240 objects degraded (9.487%); 21 MiB/s, 331 objects/s recovering 2024-03-24T12:09:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:03 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2276042756' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.9025388472584885}]: dispatch 2024-03-24T12:09:03.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:03.683 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:03.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:03.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:03.822 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:03.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:03.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:03.954 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:03.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:04.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:04.096 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:04.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:04.197 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:04.224 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:04.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:04.326 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:04.354 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:04 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2276042756' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.9025388472584885}]': finished 2024-03-24T12:09:04.354 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:04 smithi064 ceph-mon[42243]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T12:09:04.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2276042756' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.9025388472584885}]': finished 2024-03-24T12:09:04.355 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:04 smithi064 ceph-mon[30223]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T12:09:04.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:04.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:04.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:04.502 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:04.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:04 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2276042756' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.9025388472584885}]': finished 2024-03-24T12:09:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:04 smithi193 ceph-mon[33593]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T12:09:04.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:04.627 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:04.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:04.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:04.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:04.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:04.907 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:04.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:05.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:05.046 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:05.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:05.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:05.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:05.242 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:05.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:05.318 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:05.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:05.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:05.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:05.456 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:05 smithi064 ceph-mon[30223]: pgmap v766: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 22229/251796 objects degraded (8.828%); 24 MiB/s, 387 objects/s recovering 2024-03-24T12:09:05.456 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:05 smithi064 ceph-mon[30223]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T12:09:05.457 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:05 smithi064 ceph-mon[42243]: pgmap v766: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 22229/251796 objects degraded (8.828%); 24 MiB/s, 387 objects/s recovering 2024-03-24T12:09:05.457 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:05 smithi064 ceph-mon[42243]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T12:09:05.557 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:05.580 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:05.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:05 smithi193 ceph-mon[33593]: pgmap v766: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 22229/251796 objects degraded (8.828%); 24 MiB/s, 387 objects/s recovering 2024-03-24T12:09:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:05 smithi193 ceph-mon[33593]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T12:09:05.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:05.720 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:05.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:05.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:05.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:05.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:06.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:06.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:06.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:06.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:06.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:06.266 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:06.300 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:06.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:06.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:06.438 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:06.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:06.540 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:06.577 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:06.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:06.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:06.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:06.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:06.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:06.849 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:06.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:06.950 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:06.988 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:06.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:07.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:07.127 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:07.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:07.153 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:09:07.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:07.275 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:07.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:07.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:07.419 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:07.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:07.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:07.479 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:07 smithi193 ceph-mon[33593]: pgmap v768: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 817 op/s; 22229/251796 objects degraded (8.828%); 20 MiB/s, 317 objects/s recovering 2024-03-24T12:09:07.480 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:07 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3036444601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:07.480 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:07 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 22229/251796 objects degraded (8.828%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:07.504 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:07 smithi064 ceph-mon[30223]: pgmap v768: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 817 op/s; 22229/251796 objects degraded (8.828%); 20 MiB/s, 317 objects/s recovering 2024-03-24T12:09:07.504 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:07 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3036444601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:07.504 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:07 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 22229/251796 objects degraded (8.828%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:07.504 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:09:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:09:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:07.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:07 smithi064 ceph-mon[42243]: pgmap v768: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 817 op/s; 22229/251796 objects degraded (8.828%); 20 MiB/s, 317 objects/s recovering 2024-03-24T12:09:07.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:07 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3036444601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:07.505 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:07 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 22229/251796 objects degraded (8.828%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:07.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:07.552 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:07.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:07.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:07.687 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:07.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:07.789 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:07.825 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:07.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:07.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:09:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:09:07.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:07.965 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:07.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:211444 10.a 10:58e64c59:::benchmark_data_smithi064_174908_object67027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:07.688511+0000", 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.00187217, 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001905681, 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 211444 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688511+0000", 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688511+0000", 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:08.057 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688513+0000", 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.792e-06 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688517+0000", 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.532e-06 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688527+0000", 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1300000000000007e-06 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:08.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688534+0000", 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7680000000000008e-06 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688639+0000", 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000104671 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688777+0000", 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013840900000000001 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:07.688918+0000", 2024-03-24T12:09:08.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014083800000000001 2024-03-24T12:09:08.060 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:08.060 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:08.060 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:08.060 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:08.060 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T12:09:08.060 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 1 2024-03-24T12:09:08.060 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:09:08.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:08.098 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:08.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:08.200 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:08.239 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:08.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:08 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:08 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3036444601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:08 smithi193 ceph-mon[33593]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T12:09:08.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:08.378 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:08.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:08.404 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:08 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:08.404 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:08 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3036444601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:08.404 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:08 smithi064 ceph-mon[30223]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T12:09:08.404 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:08 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:08.404 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:08 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3036444601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:08.404 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:08 smithi064 ceph-mon[42243]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T12:09:08.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:08.516 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:09:08.517 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:08.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:08.554 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:09:08.594 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:09:08.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:08.632 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:09:08.654 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:08.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:08.673 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:09:08.715 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:09:08.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:08.760 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:09:08.761 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:09:08.792 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:08.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:08.798 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:09:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:09:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:09:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:09:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:09:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:09:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:09:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:09:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:09:08.894 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:08.932 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:08.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:09.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:09.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:09.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:09.184 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:09.216 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:09.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:09.317 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:09.363 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:09.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:09.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:09.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:09.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:09 smithi193 ceph-mon[33593]: pgmap v770: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.60k op/s; 19570/238395 objects degraded (8.209%); 34 MiB/s, 538 objects/s recovering 2024-03-24T12:09:09.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:09.478 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:09 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2917211228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:09.498 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:09.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:09 smithi064 ceph-mon[30223]: pgmap v770: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.60k op/s; 19570/238395 objects degraded (8.209%); 34 MiB/s, 538 objects/s recovering 2024-03-24T12:09:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:09 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2917211228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:09.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:09 smithi064 ceph-mon[42243]: pgmap v770: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.60k op/s; 19570/238395 objects degraded (8.209%); 34 MiB/s, 538 objects/s recovering 2024-03-24T12:09:09.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:09.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:09 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2917211228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:09.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:09.640 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:09.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:09.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:09.784 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:09.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:09.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:09.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:09:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:09.927 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:09.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:10.028 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:10.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:10.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:10.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:10.242 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:10.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:10.253 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:09:10.253 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:09:10.253 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-24T12:09:10.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:09:10.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:10.402 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:10.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:10.411 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:10.503 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:10.541 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:10.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:10.643 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:10.681 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:10.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:10.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:10.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:10.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:10.957 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:10.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:11.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:11.095 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:11.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:11.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:11.244 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:11.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:11.344 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:11.384 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:11.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:11.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:11 smithi193 ceph-mon[33593]: pgmap v771: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.03k op/s; 18774/235302 objects degraded (7.979%); 26 MiB/s, 416 objects/s recovering 2024-03-24T12:09:11.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:11.523 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:11.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:11 smithi064 ceph-mon[30223]: pgmap v771: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.03k op/s; 18774/235302 objects degraded (7.979%); 26 MiB/s, 416 objects/s recovering 2024-03-24T12:09:11.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:11 smithi064 ceph-mon[42243]: pgmap v771: 168 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.03k op/s; 18774/235302 objects degraded (7.979%); 26 MiB/s, 416 objects/s recovering 2024-03-24T12:09:11.608 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:09:11.608 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:09:11.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:11.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:11.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:09:11.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:11.809 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:11.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:11.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:11.949 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:11.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:12.051 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:12.087 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:12.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:12.189 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:12.225 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:12.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:12 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 18774/235302 objects degraded (7.979%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:12 smithi064 ceph-mon[30223]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T12:09:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:12 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 18774/235302 objects degraded (7.979%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:12 smithi064 ceph-mon[42243]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T12:09:12.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:12.327 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:12 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 18774/235302 objects degraded (7.979%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:12 smithi193 ceph-mon[33593]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T12:09:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:12.368 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:12.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:12.387 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:09:12.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:12.511 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:12.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:12.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:12.647 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:12.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:12.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:12.803 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:12.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:12.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:12.921 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:12.937 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:12.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:13.039 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:13.081 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:13.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:13.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:13.224 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:13.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:13.248 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:13 smithi064 ceph-mon[30223]: pgmap v772: 168 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 732 op/s; 18418/234234 objects degraded (7.863%); 16 MiB/s, 248 objects/s recovering 2024-03-24T12:09:13.249 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:13 smithi064 ceph-mon[42243]: pgmap v772: 168 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 732 op/s; 18418/234234 objects degraded (7.863%); 16 MiB/s, 248 objects/s recovering 2024-03-24T12:09:13.325 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:13 smithi193 ceph-mon[33593]: pgmap v772: 168 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 732 op/s; 18418/234234 objects degraded (7.863%); 16 MiB/s, 248 objects/s recovering 2024-03-24T12:09:13.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:13.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:13.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:13.499 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:13.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:13.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:13.600 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:13.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:13.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:13.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:13.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:13.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:13.915 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:13.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:14.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:14.049 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:14.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:14.151 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:14.202 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:14.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:14.203 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:09:14.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:14 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:09:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:14 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:09:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1213641515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:09:14.339 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:14.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:14.440 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:14.499 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:14.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:14 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:09:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:14 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:09:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1213641515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:09:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:14 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:09:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:14 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:09:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1213641515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:09:14.600 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:14.637 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:14.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:14.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:14.776 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:14.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:14.877 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:14.904 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:14.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:15.006 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:15.042 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:15.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:15.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:15.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:15.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:15.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:15.279 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:15.317 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:15.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:15 smithi193 ceph-mon[33593]: pgmap v774: 168 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 15539/217206 objects degraded (7.154%); 30 MiB/s, 486 objects/s recovering 2024-03-24T12:09:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:15 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:15 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1213641515' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:09:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:15 smithi193 ceph-mon[33593]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T12:09:15.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:15.446 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:15.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:15.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:15 smithi064 ceph-mon[30223]: pgmap v774: 168 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 15539/217206 objects degraded (7.154%); 30 MiB/s, 486 objects/s recovering 2024-03-24T12:09:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:15 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:15 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1213641515' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:09:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:15 smithi064 ceph-mon[30223]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T12:09:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:15 smithi064 ceph-mon[42243]: pgmap v774: 168 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 15539/217206 objects degraded (7.154%); 30 MiB/s, 486 objects/s recovering 2024-03-24T12:09:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:15 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:15 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1213641515' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:09:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:15 smithi064 ceph-mon[42243]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T12:09:15.580 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:15.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:15.583 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:09:15.583 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:09:15.682 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:15.721 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:15.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:15.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:15.861 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:15.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:15.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:16.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:16.005 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:16.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:16.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:16.149 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:16.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:16.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:16.287 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:16.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:16.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:16.428 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:16.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:16.529 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:16.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:16.562 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:16.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:16.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:16.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:16.814 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:16.851 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:16.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:16.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:16.984 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:16.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:17.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:17.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:09:17.136 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:17.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:17.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:17.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:17.323 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:09:17.323 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:09:17.323 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:09:17.323 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:09:17.323 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:09:17.339 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:09:17.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:17.385 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:09:17.402 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[30223]: pgmap v776: 168 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 883 op/s; 15539/217206 objects degraded (7.154%); 20 MiB/s, 322 objects/s recovering 2024-03-24T12:09:17.402 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:09:17.402 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 15539/217206 objects degraded (7.154%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[30223]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3358595160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[42243]: pgmap v776: 168 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 883 op/s; 15539/217206 objects degraded (7.154%); 20 MiB/s, 322 objects/s recovering 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 15539/217206 objects degraded (7.154%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[42243]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T12:09:17.403 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3358595160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:17.424 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:09:17.428 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:17.428 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:09:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:09:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:09:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:09:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:09:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:09:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:09:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:09:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:17.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:17.474 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:09:17.474 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:09:17.474 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 365, 2024-03-24T12:09:17.474 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T12:09:17.474 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-03-24T11:46:40.657065+0000", 2024-03-24T12:09:17.474 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-03-24T12:09:16.926452+0000", 2024-03-24T12:09:17.474 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-03-24T12:08:35.127870+0000", 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-03-24T11:51:43.667453+0000", 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5804032, 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 19, 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-03-24T12:09:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 10, 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:49:56.369674+0000", 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:09:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 21, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 21, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 21, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 21, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 21, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:09:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "260", 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "258", 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:09:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:09:17.479 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:09:17.480 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.8999999761581421, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.8999999761581421, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:09:17.481 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.8999999761581421, 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.8999999761581421, 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".rgw.root", 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:13.871974+0000", 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:09:17.482 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:09:17.483 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "68", 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:09:17.484 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:09:17.485 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:09:17.486 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.75, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.75, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-03-24T12:09:17.487 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "datapool", 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:14.117018+0000", 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 3, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 3, 2024-03-24T12:09:17.488 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 3, 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 3, 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 3, 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "71", 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:09:17.489 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 2, 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 71, 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:09:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:09:17.491 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:09:17.492 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.6700000762939453, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.6700000762939453, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 4, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:15.453744+0000", 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:09:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:09:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "70", 2024-03-24T12:09:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:09:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:09:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:09:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.25, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.25, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.25, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.25, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 5, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:17.611004+0000", 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:09:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:09:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "72", 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:09:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:09:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:09:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.75, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.75, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:09:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 6, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:19.701416+0000", 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:09:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:09:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "74", 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:09:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:09:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:09:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:09:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:09:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:09:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:09:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_bias": 4 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:09:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 10, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "unique_pool_3", 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T12:06:28.568197+0000", 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:09:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 16, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 16, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 16, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 16, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 16, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:09:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "323", 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:09:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:09:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:09:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rados": {} 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 3, 2024-03-24T12:09:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 3, 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d", 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:09:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 8, 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 354, 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-03-24T12:09:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/2675992571", 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6802/2675992571", 2024-03-24T12:09:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6804/2675992571", 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6803/2675992571", 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9", 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:09:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.2537689208984375, 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 353, 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:09:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:09:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-03-24T12:09:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6805/841667271", 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6806/841667271", 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6808/841667271", 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6807/841667271", 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b", 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.3165130615234375, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.90252685546875, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 18, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 353, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:09:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-03-24T12:09:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:09:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/435147390", 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6810/435147390", 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6812/435147390", 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6811/435147390", 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:09:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479", 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 27, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 354, 2024-03-24T12:09:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6813/3355318038", 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6814/3355318038", 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6816/3355318038", 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6815/3355318038", 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:09:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b", 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 33, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 364, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:09:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6800", 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6801", 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:09:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6803", 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6802", 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6800/3851339176", 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6801/3851339176", 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3851339176", 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3851339176", 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6", 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 41, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 317, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 353, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 353, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 318, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6804", 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6805", 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6807", 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6806", 2024-03-24T12:09:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6804/1531860860", 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6805/1531860860", 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6807/1531860860", 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6806/1531860860", 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996", 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 51, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 354, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:09:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6808", 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6809", 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:09:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6811", 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6810", 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6808/2408237278", 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6809/2408237278", 2024-03-24T12:09:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6811/2408237278", 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6810/2408237278", 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015", 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:09:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 61, 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 353, 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6812", 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6813", 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6815", 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6814", 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6812/2649989609", 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6813/2649989609", 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6815/2649989609", 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6814/2649989609", 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:08:34.326280+0000", 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:09:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:08:26.617762+0000", 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:08:22.802322+0000", 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:09:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:13.289005+0000", 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:08:18.919687+0000", 2024-03-24T12:09:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:08:34.676346+0000", 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 317 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:09:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:08:36.317364+0000", 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:08:17.515035+0000", 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.0", 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.13", 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: 0 2024-03-24T12:09:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [ 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.b", 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.4", 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 5 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.9", 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.1d", 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.6", 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:09:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.13", 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.18", 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 4, 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.19", 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 6 2024-03-24T12:09:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [ 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "10.3", 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "10.6", 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "10.7", 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: 3 2024-03-24T12:09:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "10.9", 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "10.a", 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: 0 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:09:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": { 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/3538503390": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1277626303": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3407451558": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3753895773": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/842059688": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1610565583": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/2730170969": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3257265142": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1459540658": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/629947599": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/520335082": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:09:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3351953651": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/4027721305": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/4028269665": "2024-03-25T11:47:36.035658+0000" 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-03-24T12:09:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:09:17.568 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:09:17.568 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:17 smithi193 ceph-mon[33593]: pgmap v776: 168 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 883 op/s; 15539/217206 objects degraded (7.154%); 20 MiB/s, 322 objects/s recovering 2024-03-24T12:09:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:17 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:09:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:17 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:09:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:17 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 15539/217206 objects degraded (7.154%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:17 smithi193 ceph-mon[33593]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T12:09:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3358595160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:17.600 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:17.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:17.701 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:17.729 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:17.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:17.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:09:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:09:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:17.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:17.855 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:17.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:17.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:17.988 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:17.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:18.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:09:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:09:18.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:18.123 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:18.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:18.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:18.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:18.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:18.350 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:18.379 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:18.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:18.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:18.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:18.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:18.511 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3543507671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:09:18.511 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:18 smithi193 ceph-mon[33593]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T12:09:18.548 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:09:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3543507671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:09:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:18 smithi064 ceph-mon[30223]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T12:09:18.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3543507671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:09:18.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:18 smithi064 ceph-mon[42243]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T12:09:18.607 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:18.637 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:18.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:18.657 INFO:tasks.thrashosds.thrasher:j is {'epoch': 365, 'fsid': '06f23456-e9d4-11ee-95cd-87774f69a715', 'created': '2024-03-24T11:46:40.657065+0000', 'modified': '2024-03-24T12:09:16.926452+0000', 'last_up_change': '2024-03-24T12:08:35.127870+0000', 'last_in_change': '2024-03-24T11:51:43.667453+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:49:56.369674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '260', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '258', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 1, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T11:52:13.871974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-24T11:52:14.117018+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T11:52:15.453744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T11:52:17.611004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T11:52:19.701416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-24T12:06:28.568197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '323', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'bc14a64f-9e6c-41f1-901e-c49fabffb53d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 354, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 2675992571}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6802', 'nonce': 2675992571}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6804', 'nonce': 2675992571}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 2675992571}]}, 'public_addr': '172.21.15.64:6801/2675992571', 'cluster_addr': '172.21.15.64:6802/2675992571', 'heartbeat_back_addr': '172.21.15.64:6804/2675992571', 'heartbeat_front_addr': '172.21.15.64:6803/2675992571', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '74587332-f0b2-4905-9888-7bdb604cc0e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2537689208984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 353, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 841667271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6806', 'nonce': 841667271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6808', 'nonce': 841667271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 841667271}]}, 'public_addr': '172.21.15.64:6805/841667271', 'cluster_addr': '172.21.15.64:6806/841667271', 'heartbeat_back_addr': '172.21.15.64:6808/841667271', 'heartbeat_front_addr': '172.21.15.64:6807/841667271', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a0c0ed85-9f33-4198-a1f2-447eb17c364b', 'up': 1, 'in': 1, 'weight': 0.3165130615234375, 'primary_affinity': 0.90252685546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 353, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 435147390}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6810', 'nonce': 435147390}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6812', 'nonce': 435147390}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 435147390}]}, 'public_addr': '172.21.15.64:6809/435147390', 'cluster_addr': '172.21.15.64:6810/435147390', 'heartbeat_back_addr': '172.21.15.64:6812/435147390', 'heartbeat_front_addr': '172.21.15.64:6811/435147390', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1396f66c-3e0c-4a01-b0a7-6f763989a479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 354, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3355318038}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6814', 'nonce': 3355318038}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6816', 'nonce': 3355318038}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3355318038}]}, 'public_addr': '172.21.15.64:6813/3355318038', 'cluster_addr': '172.21.15.64:6814/3355318038', 'heartbeat_back_addr': '172.21.15.64:6816/3355318038', 'heartbeat_front_addr': '172.21.15.64:6815/3355318038', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50d65d8b-3e90-46a4-b19d-50c3015af40b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3851339176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3851339176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3851339176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3851339176}]}, 'public_addr': '172.21.15.193:6800/3851339176', 'cluster_addr': '172.21.15.193:6801/3851339176', 'heartbeat_back_addr': '172.21.15.193:6803/3851339176', 'heartbeat_front_addr': '172.21.15.193:6802/3851339176', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '50c5d50e-7e68-487c-9f68-1abb8c927ff6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 317, 'up_from': 353, 'up_thru': 353, 'down_at': 318, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 1531860860}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 1531860860}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 1531860860}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 1531860860}]}, 'public_addr': '172.21.15.193:6804/1531860860', 'cluster_addr': '172.21.15.193:6805/1531860860', 'heartbeat_back_addr': '172.21.15.193:6807/1531860860', 'heartbeat_front_addr': '172.21.15.193:6806/1531860860', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7e74fe39-5707-4c90-ad38-1083cd7a6996', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 354, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 2408237278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 2408237278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 2408237278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 2408237278}]}, 'public_addr': '172.21.15.193:6808/2408237278', 'cluster_addr': '172.21.15.193:6809/2408237278', 'heartbeat_back_addr': '172.21.15.193:6811/2408237278', 'heartbeat_front_addr': '172.21.15.193:6810/2408237278', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e568df64-cc9a-43f8-a432-8628d1da1015', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 353, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 2649989609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 2649989609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 2649989609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 2649989609}]}, 'public_addr': '172.21.15.193:6812/2649989609', 'cluster_addr': '172.21.15.193:6813/2649989609', 'heartbeat_back_addr': '172.21.15.193:6815/2649989609', 'heartbeat_front_addr': '172.21.15.193:6814/2649989609', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:08:34.326280+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:08:26.617762+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:08:22.802322+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:13.289005+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:08:18.919687+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:08:34.676346+0000', 'dead_epoch': 317}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:08:36.317364+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:08:17.515035+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [0, 1, 7]}, {'pgid': '2.13', 'osds': [6, 3, 0]}], 'pg_upmap_items': [{'pgid': '2.b', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.6', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.3', 'osds': [3, 4]}, {'pgid': '10.6', 'osds': [0, 4]}, {'pgid': '10.7', 'osds': [6, 3]}, {'pgid': '10.9', 'osds': [4, 7]}, {'pgid': '10.a', 'osds': [6, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.64:6800/3538503390': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/1277626303': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3407451558': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3753895773': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/842059688': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/1610565583': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/2730170969': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3257265142': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/1459540658': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/629947599': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:6800/520335082': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3351953651': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:6800/4027721305': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/4028269665': '2024-03-25T11:47:36.035658+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:09:18.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:09:18.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:18.764 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:18.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:18.865 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:18.873 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:18.897 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:18.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:18.998 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:19.039 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:19.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:19.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:19.171 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:19.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:19.273 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:19.310 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:19.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:19 smithi193 ceph-mon[33593]: pgmap v778: 168 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 13082/202845 objects degraded (6.449%); 33 MiB/s, 531 objects/s recovering 2024-03-24T12:09:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:19 smithi193 ceph-mon[33593]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T12:09:19.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:19.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:19.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:19.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:19 smithi064 ceph-mon[30223]: pgmap v778: 168 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 13082/202845 objects degraded (6.449%); 33 MiB/s, 531 objects/s recovering 2024-03-24T12:09:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:19 smithi064 ceph-mon[30223]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T12:09:19.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:19 smithi064 ceph-mon[42243]: pgmap v778: 168 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 162 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 13082/202845 objects degraded (6.449%); 33 MiB/s, 531 objects/s recovering 2024-03-24T12:09:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:19 smithi064 ceph-mon[42243]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T12:09:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:19.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:19.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:19.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:19.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:19.822 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:19.863 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:19.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:19.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:20.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:20.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:20.132 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:20.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:20.159 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:20.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:20.277 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:20.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:20.351 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:20 smithi064 ceph-mon[30223]: 10.5 scrub starts 2024-03-24T12:09:20.351 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:20 smithi064 ceph-mon[42243]: 10.5 scrub starts 2024-03-24T12:09:20.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:20.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:20.415 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:20.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:20.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:20.547 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:20.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:20 smithi193 ceph-mon[33593]: 10.5 scrub starts 2024-03-24T12:09:20.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:20.675 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:20.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:20.776 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:20.829 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:20.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:20.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:20.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:20.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:21.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:21.114 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:21.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:21.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:21.258 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:21.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:21.359 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:21.398 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:21.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:21.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[30223]: pgmap v781: 168 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 163 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1.07k op/s; 12441/200292 objects degraded (6.211%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:09:21.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T12:09:21.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T12:09:21.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[30223]: 1.14 scrub starts 2024-03-24T12:09:21.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[30223]: 1.14 scrub ok 2024-03-24T12:09:21.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:09:21.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[42243]: pgmap v781: 168 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 163 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1.07k op/s; 12441/200292 objects degraded (6.211%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:09:21.413 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T12:09:21.414 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T12:09:21.414 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[42243]: 1.14 scrub starts 2024-03-24T12:09:21.414 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[42243]: 1.14 scrub ok 2024-03-24T12:09:21.414 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:21 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:09:21.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:21.536 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:21.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:21.550 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:21 smithi193 ceph-mon[33593]: pgmap v781: 168 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 163 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1.07k op/s; 12441/200292 objects degraded (6.211%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:09:21.550 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:21 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T12:09:21.550 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:21 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T12:09:21.550 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:21 smithi193 ceph-mon[33593]: 1.14 scrub starts 2024-03-24T12:09:21.551 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:21 smithi193 ceph-mon[33593]: 1.14 scrub ok 2024-03-24T12:09:21.551 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:21 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:09:21.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:21.672 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:21.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:21.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:21.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:21.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:21.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:21.941 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:21.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:22.043 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:22.079 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:22.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:22.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:09:22.180 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:22.222 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:22.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:22.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:22.366 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:22.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:22.468 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:22.506 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:22.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:22 smithi064 ceph-mon[30223]: 10.5 scrub ok 2024-03-24T12:09:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:22 smithi064 ceph-mon[30223]: 1.b scrub starts 2024-03-24T12:09:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:22 smithi064 ceph-mon[30223]: 1.b scrub ok 2024-03-24T12:09:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:22 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 12441/200292 objects degraded (6.211%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:22.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:22 smithi064 ceph-mon[42243]: 10.5 scrub ok 2024-03-24T12:09:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:22 smithi064 ceph-mon[42243]: 1.b scrub starts 2024-03-24T12:09:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:22 smithi064 ceph-mon[42243]: 1.b scrub ok 2024-03-24T12:09:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:22 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 12441/200292 objects degraded (6.211%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:22 smithi193 ceph-mon[33593]: 10.5 scrub ok 2024-03-24T12:09:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:22 smithi193 ceph-mon[33593]: 1.b scrub starts 2024-03-24T12:09:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:22 smithi193 ceph-mon[33593]: 1.b scrub ok 2024-03-24T12:09:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:22 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 12441/200292 objects degraded (6.211%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:22.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:22.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:22.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:22.789 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:22.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:22.890 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:22.936 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:22.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:23.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:23.069 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:23.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:23.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:23.206 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:23.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:23.307 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:23.350 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:23.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:23.390 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:23.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:23.509 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:23.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:23.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[30223]: pgmap v782: 168 pgs: 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 163 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 976 op/s; 11002/199638 objects degraded (5.511%); 33 MiB/s, 530 objects/s recovering 2024-03-24T12:09:23.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[30223]: 6.0 scrub starts 2024-03-24T12:09:23.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[30223]: 6.0 scrub ok 2024-03-24T12:09:23.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:09:23.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:23.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:23.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[42243]: pgmap v782: 168 pgs: 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 163 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 976 op/s; 11002/199638 objects degraded (5.511%); 33 MiB/s, 530 objects/s recovering 2024-03-24T12:09:23.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[42243]: 6.0 scrub starts 2024-03-24T12:09:23.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[42243]: 6.0 scrub ok 2024-03-24T12:09:23.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:09:23.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:23.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:23 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:23.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:23 smithi193 ceph-mon[33593]: pgmap v782: 168 pgs: 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 163 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 976 op/s; 11002/199638 objects degraded (5.511%); 33 MiB/s, 530 objects/s recovering 2024-03-24T12:09:23.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:23 smithi193 ceph-mon[33593]: 6.0 scrub starts 2024-03-24T12:09:23.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:23 smithi193 ceph-mon[33593]: 6.0 scrub ok 2024-03-24T12:09:23.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:23 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:09:23.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:23 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:23.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:23 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:23.610 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:23.655 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:23.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:23.756 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:227864 10.c 10:3a50be3f:::benchmark_data_smithi064_174908_object83447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:23.753814+0000", 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.022988795999999999, 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023025482, 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:23.779 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 227864 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.753814+0000", 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.753814+0000", 2024-03-24T12:09:23.780 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.753817+0000", 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.481e-06 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.753824+0000", 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6879999999999999e-06 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.753836+0000", 2024-03-24T12:09:23.781 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2104e-05 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.753845+0000", 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.755e-06 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.756070+0000", 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002225126 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:23.782 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.756189+0000", 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011885800000000001 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.756304+0000", 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000114517 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.759786+0000", 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034821819999999999 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:23.783 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.763019+0000", 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032334270000000001 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.15714.0:227877 10.8 e368/320)", 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:23.776717+0000", 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 8.5469000000000004e-05, 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021459499999999999, 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "queued for pg", 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.784 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.776717+0000", 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.776717+0000", 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.776720+0000", 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6589999999999999e-06 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:23.785 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.776731+0000", 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1331e-05 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.776751+0000", 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9400999999999999e-05 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:23.776761+0000", 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0635999999999999e-05 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:23.786 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:23.787 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T12:09:23.787 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 2 2024-03-24T12:09:23.787 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:09:23.800 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:23.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:23.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:23.948 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:23.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:24.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:24.086 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:24.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:24.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:24.242 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:24.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:24.343 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:24.387 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:24.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:24.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[30223]: pgmap v783: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 759 op/s; 8475/189984 objects degraded (4.461%); 39 MiB/s, 619 objects/s recovering 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[30223]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:09:24.461 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[42243]: pgmap v783: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 759 op/s; 8475/189984 objects degraded (4.461%); 39 MiB/s, 619 objects/s recovering 2024-03-24T12:09:24.462 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[42243]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T12:09:24.462 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:09:24.462 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:09:24.462 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:24 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:24.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:24.514 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:24.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:24 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:09:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:24 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:09:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:24 smithi193 ceph-mon[33593]: pgmap v783: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 759 op/s; 8475/189984 objects degraded (4.461%); 39 MiB/s, 619 objects/s recovering 2024-03-24T12:09:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:24 smithi193 ceph-mon[33593]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T12:09:24.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:24 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:09:24.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:24 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:09:24.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:24 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:24.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:24.657 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:24.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:24.759 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:24.795 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:09:24.798 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:09:24.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:24.799 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:24.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:24.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:24.930 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:24.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:25.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:25.059 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:25.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:25.161 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:25.195 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:25.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:25.274 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:09:25.297 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:25.331 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:25.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:25.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:25 smithi064 ceph-mon[30223]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-24T12:09:25.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:25 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/208769472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:25.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:25 smithi064 ceph-mon[30223]: from='client.25729 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:25.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:25 smithi064 ceph-mon[30223]: from='client.25732 v1:172.21.15.64:0/4100110856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:09:25.393 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:25 smithi064 ceph-mon[42243]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-24T12:09:25.393 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:25 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/208769472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:25.393 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:25 smithi064 ceph-mon[42243]: from='client.25729 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:25.394 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:25 smithi064 ceph-mon[42243]: from='client.25732 v1:172.21.15.64:0/4100110856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:09:25.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:25.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:25.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:25.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:25 smithi193 ceph-mon[33593]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-24T12:09:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:25 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/208769472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:25 smithi193 ceph-mon[33593]: from='client.25729 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:25 smithi193 ceph-mon[33593]: from='client.25732 v1:172.21.15.64:0/4100110856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:09:25.608 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:25.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:25.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:25.937 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:25.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:26.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:26.076 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:26.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:26.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:26.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:26.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:26.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:26.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:26.348 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:09:26.366 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:26.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:26.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:26.504 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:26.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:26 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:09:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:26 smithi064 ceph-mon[30223]: from='client.25729 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:26 smithi064 ceph-mon[30223]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T12:09:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:26 smithi064 ceph-mon[30223]: pgmap v786: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 781 B/s rd, 0 B/s wr, 617 op/s; 8475/189051 objects degraded (4.483%); 39 MiB/s, 620 objects/s recovering 2024-03-24T12:09:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:26 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:09:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:26 smithi064 ceph-mon[42243]: from='client.25729 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:26 smithi064 ceph-mon[42243]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T12:09:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:26 smithi064 ceph-mon[42243]: pgmap v786: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 781 B/s rd, 0 B/s wr, 617 op/s; 8475/189051 objects degraded (4.483%); 39 MiB/s, 620 objects/s recovering 2024-03-24T12:09:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:26 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:09:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:26 smithi193 ceph-mon[33593]: from='client.25729 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:26 smithi193 ceph-mon[33593]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T12:09:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:26 smithi193 ceph-mon[33593]: pgmap v786: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 781 B/s rd, 0 B/s wr, 617 op/s; 8475/189051 objects degraded (4.483%); 39 MiB/s, 620 objects/s recovering 2024-03-24T12:09:26.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:26.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:26.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:26.771 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:26.787 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:26.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:26.888 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:26.925 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:26.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:27.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:27.051 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:27.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:27.153 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:27.193 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:27.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:27.295 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:27.323 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:27.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:27.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:27.457 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:27.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:27.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:09:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:27.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[30223]: 4.3 deep-scrub starts 2024-03-24T12:09:27.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[30223]: 4.3 deep-scrub ok 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 8475/189051 objects degraded (4.483%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[42243]: 4.3 deep-scrub starts 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[42243]: 4.3 deep-scrub ok 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 8475/189051 objects degraded (4.483%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:27.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:27.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:27.585 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:27 smithi193 ceph-mon[33593]: 4.3 deep-scrub starts 2024-03-24T12:09:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:27 smithi193 ceph-mon[33593]: 4.3 deep-scrub ok 2024-03-24T12:09:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:27 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T12:09:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:27 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T12:09:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:27 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 8475/189051 objects degraded (4.483%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:27.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:27.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:27.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:27.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:27.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:27.863 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:27.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:27.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:27.990 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:27.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:27.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:28.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:09:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:09:28.093 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:28.126 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:28.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:28.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:28.263 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:28.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:28.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:28.394 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:28.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:28.496 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:28.537 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:28.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:28.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:28 smithi064 ceph-mon[30223]: pgmap v787: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 5814/176199 objects degraded (3.300%); 43 MiB/s, 686 objects/s recovering 2024-03-24T12:09:28.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:28.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:28 smithi064 ceph-mon[42243]: pgmap v787: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 5814/176199 objects degraded (3.300%); 43 MiB/s, 686 objects/s recovering 2024-03-24T12:09:28.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:28 smithi193 ceph-mon[33593]: pgmap v787: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 5814/176199 objects degraded (3.300%); 43 MiB/s, 686 objects/s recovering 2024-03-24T12:09:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:28.640 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:28.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:28.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:28.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:28.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:28.820 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:28.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:28.922 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:28.966 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:28.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:29.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:29.155 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:29.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:29.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:29.300 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:29.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:29.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:29.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:29.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:29.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:29.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:29 smithi064 ceph-mon[30223]: 1.11 scrub starts 2024-03-24T12:09:29.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:29 smithi064 ceph-mon[30223]: 1.11 scrub ok 2024-03-24T12:09:29.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:29 smithi064 ceph-mon[42243]: 1.11 scrub starts 2024-03-24T12:09:29.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:29 smithi064 ceph-mon[42243]: 1.11 scrub ok 2024-03-24T12:09:29.578 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:29.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:29 smithi193 ceph-mon[33593]: 1.11 scrub starts 2024-03-24T12:09:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:29 smithi193 ceph-mon[33593]: 1.11 scrub ok 2024-03-24T12:09:29.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:29.721 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:29.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:29.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:29.860 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:29.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:29.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:29.991 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:29.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:30.093 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:30.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:30.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:30.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:30.271 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:30.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:30.373 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:30.402 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:30.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:30.503 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":784,"stamp":"2024-03-24T12:09:23.739940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4137716807,"num_objects":63328,"num_object_clones":0,"num_object_copies":189984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63328,"num_whiteouts":0,"num_read":1417,"num_read_kb":1195,"num_write":226471,"num_write_kb":9243200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19493,"num_bytes_recovered":1274439260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79888,"ondisk_log_size":79888,"up":504,"acting":501,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16483208,"kb_used_data":11591292,"kb_used_omap":18,"kb_used_meta":4891693,"kb_avail":733478008,"statfs":{"total":767960285184,"available":751081480192,"internally_reserved":0,"allocated":11869483008,"data_stored":11864473352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":5009093851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,8,5,3,7,2,4,4,0,11],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-280952832,"num_objects":-4287,"num_object_clones":0,"num_object_copies":-12861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4287,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4063,"num_bytes_recovered":266272768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.247735"},"pg_stats":[{"pgid":"10.6","version":"367'13699","reported_seq":24573,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:19.237456+0000","last_change":"2024-03-24T12:09:18.996729+0000","last_active":"2024-03-24T12:09:19.237456+0000","last_peered":"2024-03-24T12:09:19.237456+0000","last_clean":"2024-03-24T12:09:19.237456+0000","last_became_active":"2024-03-24T12:09:18.952687+0000","last_became_peered":"2024-03-24T12:09:18.952687+0000","last_unstale":"2024-03-24T12:09:19.237456+0000","last_undegraded":"2024-03-24T12:09:19.237456+0000","last_fullsized":"2024-03-24T12:09:19.237456+0000","mapping_epoch":366,"log_start":"343'9326","ondisk_log_start":"343'9326","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266534912,"num_objects":4067,"num_object_clones":0,"num_object_copies":12201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13699,"num_write_kb":568512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5381,"num_bytes_recovered":352649216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'130","reported_seq":544,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960443+0000","last_change":"2024-03-24T12:07:37.002106+0000","last_active":"2024-03-24T12:09:18.960443+0000","last_peered":"2024-03-24T12:09:18.960443+0000","last_clean":"2024-03-24T12:09:18.960443+0000","last_became_active":"2024-03-24T12:03:19.822426+0000","last_became_peered":"2024-03-24T12:03:19.822426+0000","last_unstale":"2024-03-24T12:09:18.960443+0000","last_undegraded":"2024-03-24T12:09:18.960443+0000","last_fullsized":"2024-03-24T12:09:18.960443+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:37.001978+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:37.001978+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191378+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":896,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960245+0000","last_change":"2024-03-24T12:08:46.189879+0000","last_active":"2024-03-24T12:09:18.960245+0000","last_peered":"2024-03-24T12:09:18.960245+0000","last_clean":"2024-03-24T12:09:18.960245+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:09:18.960245+0000","last_undegraded":"2024-03-24T12:09:18.960245+0000","last_fullsized":"2024-03-24T12:09:18.960245+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:08:46.189592+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:57.625770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":597,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.947934+0000","last_change":"2024-03-24T12:08:36.154379+0000","last_active":"2024-03-24T12:09:18.947934+0000","last_peered":"2024-03-24T12:09:18.947934+0000","last_clean":"2024-03-24T12:09:18.947934+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:09:18.947934+0000","last_undegraded":"2024-03-24T12:09:18.947934+0000","last_fullsized":"2024-03-24T12:09:18.947934+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:05:17.253630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.253630+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":154,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944858+0000","last_change":"2024-03-24T12:08:32.282623+0000","last_active":"2024-03-24T12:09:18.944858+0000","last_peered":"2024-03-24T12:09:18.944858+0000","last_clean":"2024-03-24T12:09:18.944858+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:09:18.944858+0000","last_undegraded":"2024-03-24T12:09:18.944858+0000","last_fullsized":"2024-03-24T12:09:18.944858+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:32.282513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:32.282513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:43.785500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":581,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948262+0000","last_change":"2024-03-24T12:08:36.168091+0000","last_active":"2024-03-24T12:09:18.948262+0000","last_peered":"2024-03-24T12:09:18.948262+0000","last_clean":"2024-03-24T12:09:18.948262+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:09:18.948262+0000","last_undegraded":"2024-03-24T12:09:18.948262+0000","last_fullsized":"2024-03-24T12:09:18.948262+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.095464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.095464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"367'14482","reported_seq":14744,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:22.162995+0000","last_change":"2024-03-24T12:09:20.604425+0000","last_active":"2024-03-24T12:09:22.162995+0000","last_peered":"2024-03-24T12:09:22.162995+0000","last_clean":"2024-03-24T12:09:22.162995+0000","last_became_active":"2024-03-24T12:06:30.016602+0000","last_became_peered":"2024-03-24T12:06:30.016602+0000","last_unstale":"2024-03-24T12:09:22.162995+0000","last_undegraded":"2024-03-24T12:09:22.162995+0000","last_fullsized":"2024-03-24T12:09:22.162995+0000","mapping_epoch":320,"log_start":"332'4400","ondisk_log_start":"332'4400","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'14378","last_scrub_stamp":"2024-03-24T12:09:20.604215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:09:20.604215+0000","objects_scrubbed":4208,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:28.135524+0000","scrub_duration":1324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266469376,"num_objects":4066,"num_object_clones":0,"num_object_copies":12198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14482,"num_write_kb":593536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'130","reported_seq":543,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952489+0000","last_change":"2024-03-24T12:07:58.303998+0000","last_active":"2024-03-24T12:09:18.952489+0000","last_peered":"2024-03-24T12:09:18.952489+0000","last_clean":"2024-03-24T12:09:18.952489+0000","last_became_active":"2024-03-24T12:03:19.822938+0000","last_became_peered":"2024-03-24T12:03:19.822938+0000","last_unstale":"2024-03-24T12:09:18.952489+0000","last_undegraded":"2024-03-24T12:09:18.952489+0000","last_fullsized":"2024-03-24T12:09:18.952489+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:58.303527+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.303527+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":694,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944428+0000","last_change":"2024-03-24T12:07:52.313886+0000","last_active":"2024-03-24T12:09:18.944428+0000","last_peered":"2024-03-24T12:09:18.944428+0000","last_clean":"2024-03-24T12:09:18.944428+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:09:18.944428+0000","last_undegraded":"2024-03-24T12:09:18.944428+0000","last_fullsized":"2024-03-24T12:09:18.944428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:52.313429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:52.313429+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:20.009793+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":697,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959571+0000","last_change":"2024-03-24T12:08:46.248251+0000","last_active":"2024-03-24T12:09:18.959571+0000","last_peered":"2024-03-24T12:09:18.959571+0000","last_clean":"2024-03-24T12:09:18.959571+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:09:18.959571+0000","last_undegraded":"2024-03-24T12:09:18.959571+0000","last_fullsized":"2024-03-24T12:09:18.959571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:46.248131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:08:46.248131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:56.816672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":153,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952190+0000","last_change":"2024-03-24T12:07:49.261571+0000","last_active":"2024-03-24T12:09:18.952190+0000","last_peered":"2024-03-24T12:09:18.952190+0000","last_clean":"2024-03-24T12:09:18.952190+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:09:18.952190+0000","last_undegraded":"2024-03-24T12:09:18.952190+0000","last_fullsized":"2024-03-24T12:09:18.952190+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:07:49.261483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":659,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945210+0000","last_change":"2024-03-24T12:07:43.508366+0000","last_active":"2024-03-24T12:09:18.945210+0000","last_peered":"2024-03-24T12:09:18.945210+0000","last_clean":"2024-03-24T12:09:18.945210+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:09:18.945210+0000","last_undegraded":"2024-03-24T12:09:18.945210+0000","last_fullsized":"2024-03-24T12:09:18.945210+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:43.508257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:43.508257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"367'14380","reported_seq":14485,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:23.456685+0000","last_change":"2024-03-24T12:06:30.020617+0000","last_active":"2024-03-24T12:09:23.456685+0000","last_peered":"2024-03-24T12:09:23.456685+0000","last_clean":"2024-03-24T12:09:23.456685+0000","last_became_active":"2024-03-24T12:06:30.020223+0000","last_became_peered":"2024-03-24T12:06:30.020223+0000","last_unstale":"2024-03-24T12:09:23.456685+0000","last_undegraded":"2024-03-24T12:09:23.456685+0000","last_fullsized":"2024-03-24T12:09:23.456685+0000","mapping_epoch":320,"log_start":"347'9900","ondisk_log_start":"347'9900","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253100032,"num_objects":3862,"num_object_clones":0,"num_object_copies":11586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14380,"num_write_kb":583744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'130","reported_seq":730,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948778+0000","last_change":"2024-03-24T12:08:36.158844+0000","last_active":"2024-03-24T12:09:18.948778+0000","last_peered":"2024-03-24T12:09:18.948778+0000","last_clean":"2024-03-24T12:09:18.948778+0000","last_became_active":"2024-03-24T12:08:36.158383+0000","last_became_peered":"2024-03-24T12:08:36.158383+0000","last_unstale":"2024-03-24T12:09:18.948778+0000","last_undegraded":"2024-03-24T12:09:18.948778+0000","last_fullsized":"2024-03-24T12:09:18.948778+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:02.370579+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:02.370579+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"296'44","reported_seq":739,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948899+0000","last_change":"2024-03-24T12:07:37.639999+0000","last_active":"2024-03-24T12:09:18.948899+0000","last_peered":"2024-03-24T12:09:18.948899+0000","last_clean":"2024-03-24T12:09:18.948899+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:09:18.948899+0000","last_undegraded":"2024-03-24T12:09:18.948899+0000","last_fullsized":"2024-03-24T12:09:18.948899+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:07:37.639868+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:07:37.639868+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":51,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948084+0000","last_change":"2024-03-24T12:08:36.164911+0000","last_active":"2024-03-24T12:09:18.948084+0000","last_peered":"2024-03-24T12:09:18.948084+0000","last_clean":"2024-03-24T12:09:18.948084+0000","last_became_active":"2024-03-24T12:08:36.164337+0000","last_became_peered":"2024-03-24T12:08:36.164337+0000","last_unstale":"2024-03-24T12:09:18.948084+0000","last_undegraded":"2024-03-24T12:09:18.948084+0000","last_fullsized":"2024-03-24T12:09:18.948084+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:00.489543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:05:00.489543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":162,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:19.669041+0000","last_change":"2024-03-24T12:09:19.669080+0000","last_active":"2024-03-24T12:09:19.669041+0000","last_peered":"2024-03-24T12:09:19.669041+0000","last_clean":"2024-03-24T12:09:19.669041+0000","last_became_active":"2024-03-24T12:08:36.155105+0000","last_became_peered":"2024-03-24T12:08:36.155105+0000","last_unstale":"2024-03-24T12:09:19.669041+0000","last_undegraded":"2024-03-24T12:09:19.669041+0000","last_fullsized":"2024-03-24T12:09:19.669041+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:19.668946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:09:19.668946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:43.853782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":675,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959346+0000","last_change":"2024-03-24T12:07:35.957089+0000","last_active":"2024-03-24T12:09:18.959346+0000","last_peered":"2024-03-24T12:09:18.959346+0000","last_clean":"2024-03-24T12:09:18.959346+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:09:18.959346+0000","last_undegraded":"2024-03-24T12:09:18.959346+0000","last_fullsized":"2024-03-24T12:09:18.959346+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:35.956974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:07:35.956974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"367'14179","reported_seq":16833,"reported_epoch":367,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-24T12:09:21.238653+0000","last_change":"2024-03-24T12:09:16.532488+0000","last_active":"2024-03-24T12:09:21.238653+0000","last_peered":"2024-03-24T12:09:21.238653+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:08:37.137388+0000","last_became_peered":"2024-03-24T12:08:37.137388+0000","last_unstale":"2024-03-24T12:09:21.238653+0000","last_undegraded":"2024-03-24T12:08:36.926012+0000","last_fullsized":"2024-03-24T12:08:36.913570+0000","mapping_epoch":354,"log_start":"347'10000","ondisk_log_start":"347'10000","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256049152,"num_objects":3907,"num_object_clones":0,"num_object_copies":11721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14179,"num_write_kb":578752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1272,"num_bytes_recovered":83361792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":3907}],"blocked_by":[],"up_primary":5,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":749,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952700+0000","last_change":"2024-03-24T12:07:41.143174+0000","last_active":"2024-03-24T12:09:18.952700+0000","last_peered":"2024-03-24T12:09:18.952700+0000","last_clean":"2024-03-24T12:09:18.952700+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:09:18.952700+0000","last_undegraded":"2024-03-24T12:09:18.952700+0000","last_fullsized":"2024-03-24T12:09:18.952700+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:07:41.143037+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.143037+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":462,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952108+0000","last_change":"2024-03-24T12:09:04.230431+0000","last_active":"2024-03-24T12:09:18.952108+0000","last_peered":"2024-03-24T12:09:18.952108+0000","last_clean":"2024-03-24T12:09:18.952108+0000","last_became_active":"2024-03-24T12:09:04.230111+0000","last_became_peered":"2024-03-24T12:09:04.230111+0000","last_unstale":"2024-03-24T12:09:18.952108+0000","last_undegraded":"2024-03-24T12:09:18.952108+0000","last_fullsized":"2024-03-24T12:09:18.952108+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:08:42.842178+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.842178+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:05.812492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":417,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952682+0000","last_change":"2024-03-24T12:07:48.250602+0000","last_active":"2024-03-24T12:09:18.952682+0000","last_peered":"2024-03-24T12:09:18.952682+0000","last_clean":"2024-03-24T12:09:18.952682+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:09:18.952682+0000","last_undegraded":"2024-03-24T12:09:18.952682+0000","last_fullsized":"2024-03-24T12:09:18.952682+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_clean_scrub_stamp":"2024-03-24T12:07:48.250404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":153,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945206+0000","last_change":"2024-03-24T12:08:34.445597+0000","last_active":"2024-03-24T12:09:18.945206+0000","last_peered":"2024-03-24T12:09:18.945206+0000","last_clean":"2024-03-24T12:09:18.945206+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:09:18.945206+0000","last_undegraded":"2024-03-24T12:09:18.945206+0000","last_fullsized":"2024-03-24T12:09:18.945206+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:34.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.445477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:55.101912+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":589,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948164+0000","last_change":"2024-03-24T12:08:36.158456+0000","last_active":"2024-03-24T12:09:18.948164+0000","last_peered":"2024-03-24T12:09:18.948164+0000","last_clean":"2024-03-24T12:09:18.948164+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:09:18.948164+0000","last_undegraded":"2024-03-24T12:09:18.948164+0000","last_fullsized":"2024-03-24T12:09:18.948164+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:19.243242+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.243242+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.712908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"367'16506","reported_seq":16834,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.883672+0000","last_change":"2024-03-24T12:08:31.853008+0000","last_active":"2024-03-24T12:09:21.883672+0000","last_peered":"2024-03-24T12:09:21.883672+0000","last_clean":"2024-03-24T12:09:21.883672+0000","last_became_active":"2024-03-24T12:06:30.018755+0000","last_became_peered":"2024-03-24T12:06:30.018755+0000","last_unstale":"2024-03-24T12:09:21.883672+0000","last_undegraded":"2024-03-24T12:09:21.883672+0000","last_fullsized":"2024-03-24T12:09:21.883672+0000","mapping_epoch":320,"log_start":"343'11902","ondisk_log_start":"343'11902","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13218","last_scrub_stamp":"2024-03-24T12:08:31.852627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:08:31.852627+0000","objects_scrubbed":7225,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:58.458495+0000","scrub_duration":2505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251985920,"num_objects":3845,"num_object_clones":0,"num_object_copies":11535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13949,"num_write_kb":569408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":703,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952406+0000","last_change":"2024-03-24T12:07:56.328634+0000","last_active":"2024-03-24T12:09:18.952406+0000","last_peered":"2024-03-24T12:09:18.952406+0000","last_clean":"2024-03-24T12:09:18.952406+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:09:18.952406+0000","last_undegraded":"2024-03-24T12:09:18.952406+0000","last_fullsized":"2024-03-24T12:09:18.952406+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:56.328393+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.328393+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":416,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943227+0000","last_change":"2024-03-24T12:07:28.052211+0000","last_active":"2024-03-24T12:09:18.943227+0000","last_peered":"2024-03-24T12:09:18.943227+0000","last_clean":"2024-03-24T12:09:18.943227+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:09:18.943227+0000","last_undegraded":"2024-03-24T12:09:18.943227+0000","last_fullsized":"2024-03-24T12:09:18.943227+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:28.052079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:07:28.052079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":899,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.946551+0000","last_change":"2024-03-24T12:08:36.789595+0000","last_active":"2024-03-24T12:09:18.946551+0000","last_peered":"2024-03-24T12:09:18.946551+0000","last_clean":"2024-03-24T12:09:18.946551+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:09:18.946551+0000","last_undegraded":"2024-03-24T12:09:18.946551+0000","last_fullsized":"2024-03-24T12:09:18.946551+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:08:36.789473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.789473+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:06.594359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":672,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.953622+0000","last_change":"2024-03-24T12:08:36.161454+0000","last_active":"2024-03-24T12:09:18.953622+0000","last_peered":"2024-03-24T12:09:18.953622+0000","last_clean":"2024-03-24T12:09:18.953622+0000","last_became_active":"2024-03-24T12:08:36.161088+0000","last_became_peered":"2024-03-24T12:08:36.161088+0000","last_unstale":"2024-03-24T12:09:18.953622+0000","last_undegraded":"2024-03-24T12:09:18.953622+0000","last_fullsized":"2024-03-24T12:09:18.953622+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:03:47.909970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:47.909970+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":974,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959410+0000","last_change":"2024-03-24T12:08:44.223168+0000","last_active":"2024-03-24T12:09:18.959410+0000","last_peered":"2024-03-24T12:09:18.959410+0000","last_clean":"2024-03-24T12:09:18.959410+0000","last_became_active":"2024-03-24T12:08:36.161668+0000","last_became_peered":"2024-03-24T12:08:36.161668+0000","last_unstale":"2024-03-24T12:09:18.959410+0000","last_undegraded":"2024-03-24T12:09:18.959410+0000","last_fullsized":"2024-03-24T12:09:18.959410+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:44.222917+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:08:44.222917+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:04.176917+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"367'14378","reported_seq":14483,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.517380+0000","last_change":"2024-03-24T12:06:30.019783+0000","last_active":"2024-03-24T12:09:21.517380+0000","last_peered":"2024-03-24T12:09:21.517380+0000","last_clean":"2024-03-24T12:09:21.517380+0000","last_became_active":"2024-03-24T12:06:30.019501+0000","last_became_peered":"2024-03-24T12:06:30.019501+0000","last_unstale":"2024-03-24T12:09:21.517380+0000","last_undegraded":"2024-03-24T12:09:21.517380+0000","last_fullsized":"2024-03-24T12:09:21.517380+0000","mapping_epoch":320,"log_start":"347'10000","ondisk_log_start":"347'10000","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263979008,"num_objects":4028,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14378,"num_write_kb":588992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":735,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959680+0000","last_change":"2024-03-24T12:08:43.275820+0000","last_active":"2024-03-24T12:09:18.959680+0000","last_peered":"2024-03-24T12:09:18.959680+0000","last_clean":"2024-03-24T12:09:18.959680+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:09:18.959680+0000","last_undegraded":"2024-03-24T12:09:18.959680+0000","last_fullsized":"2024-03-24T12:09:18.959680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:43.275664+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.275664+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:02.607555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":689,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959001+0000","last_change":"2024-03-24T12:08:42.224518+0000","last_active":"2024-03-24T12:09:18.959001+0000","last_peered":"2024-03-24T12:09:18.959001+0000","last_clean":"2024-03-24T12:09:18.959001+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:09:18.959001+0000","last_undegraded":"2024-03-24T12:09:18.959001+0000","last_fullsized":"2024-03-24T12:09:18.959001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:42.224320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.224320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:57.377549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":884,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.953648+0000","last_change":"2024-03-24T12:08:36.173303+0000","last_active":"2024-03-24T12:09:18.953648+0000","last_peered":"2024-03-24T12:09:18.953648+0000","last_clean":"2024-03-24T12:09:18.953648+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:09:18.953648+0000","last_undegraded":"2024-03-24T12:09:18.953648+0000","last_fullsized":"2024-03-24T12:09:18.953648+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:05.860814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.860814+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":589,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949862+0000","last_change":"2024-03-24T12:08:36.167496+0000","last_active":"2024-03-24T12:09:18.949862+0000","last_peered":"2024-03-24T12:09:18.949862+0000","last_clean":"2024-03-24T12:09:18.949862+0000","last_became_active":"2024-03-24T12:08:36.166897+0000","last_became_peered":"2024-03-24T12:08:36.166897+0000","last_unstale":"2024-03-24T12:09:18.949862+0000","last_undegraded":"2024-03-24T12:09:18.949862+0000","last_fullsized":"2024-03-24T12:09:18.949862+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.219541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.219541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":589,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949452+0000","last_change":"2024-03-24T12:08:36.171738+0000","last_active":"2024-03-24T12:09:18.949452+0000","last_peered":"2024-03-24T12:09:18.949452+0000","last_clean":"2024-03-24T12:09:18.949452+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:09:18.949452+0000","last_undegraded":"2024-03-24T12:09:18.949452+0000","last_fullsized":"2024-03-24T12:09:18.949452+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:11.242568+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.242568+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.712933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"367'14235","reported_seq":14418,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.609468+0000","last_change":"2024-03-24T12:07:45.462425+0000","last_active":"2024-03-24T12:09:21.609468+0000","last_peered":"2024-03-24T12:09:21.609468+0000","last_clean":"2024-03-24T12:09:21.609468+0000","last_became_active":"2024-03-24T12:06:30.016735+0000","last_became_peered":"2024-03-24T12:06:30.016735+0000","last_unstale":"2024-03-24T12:09:21.609468+0000","last_undegraded":"2024-03-24T12:09:21.609468+0000","last_fullsized":"2024-03-24T12:09:21.609468+0000","mapping_epoch":320,"log_start":"346'9766","ondisk_log_start":"346'9766","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7238","last_scrub_stamp":"2024-03-24T12:07:45.462060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.462060+0000","objects_scrubbed":7162,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261816320,"num_objects":3995,"num_object_clones":0,"num_object_copies":11985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14235,"num_write_kb":583360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"83'130","reported_seq":554,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:20.672017+0000","last_change":"2024-03-24T12:09:20.672043+0000","last_active":"2024-03-24T12:09:20.672017+0000","last_peered":"2024-03-24T12:09:20.672017+0000","last_clean":"2024-03-24T12:09:20.672017+0000","last_became_active":"2024-03-24T12:08:36.153926+0000","last_became_peered":"2024-03-24T12:08:36.153926+0000","last_unstale":"2024-03-24T12:09:20.672017+0000","last_undegraded":"2024-03-24T12:09:20.672017+0000","last_fullsized":"2024-03-24T12:09:20.672017+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:09:20.671955+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:20.649114+0000","last_clean_scrub_stamp":"2024-03-24T12:09:20.671955+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:30.292057+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":696,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952518+0000","last_change":"2024-03-24T12:06:46.690666+0000","last_active":"2024-03-24T12:09:18.952518+0000","last_peered":"2024-03-24T12:09:18.952518+0000","last_clean":"2024-03-24T12:09:18.952518+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:09:18.952518+0000","last_undegraded":"2024-03-24T12:09:18.952518+0000","last_fullsized":"2024-03-24T12:09:18.952518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:06:46.690548+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:06:46.690548+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":681,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959616+0000","last_change":"2024-03-24T12:06:55.146672+0000","last_active":"2024-03-24T12:09:18.959616+0000","last_peered":"2024-03-24T12:09:18.959616+0000","last_clean":"2024-03-24T12:09:18.959616+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:09:18.959616+0000","last_undegraded":"2024-03-24T12:09:18.959616+0000","last_fullsized":"2024-03-24T12:09:18.959616+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:55.146558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:06:55.146558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":585,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949164+0000","last_change":"2024-03-24T12:08:36.163502+0000","last_active":"2024-03-24T12:09:18.949164+0000","last_peered":"2024-03-24T12:09:18.949164+0000","last_clean":"2024-03-24T12:09:18.949164+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:09:18.949164+0000","last_undegraded":"2024-03-24T12:09:18.949164+0000","last_fullsized":"2024-03-24T12:09:18.949164+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:15.334211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:15.334211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":663,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943297+0000","last_change":"2024-03-24T12:08:36.153899+0000","last_active":"2024-03-24T12:09:18.943297+0000","last_peered":"2024-03-24T12:09:18.943297+0000","last_clean":"2024-03-24T12:09:18.943297+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:09:18.943297+0000","last_undegraded":"2024-03-24T12:09:18.943297+0000","last_fullsized":"2024-03-24T12:09:18.943297+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:06.995768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:06.995768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"367'14060","reported_seq":14255,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.619765+0000","last_change":"2024-03-24T12:07:58.231671+0000","last_active":"2024-03-24T12:09:21.619765+0000","last_peered":"2024-03-24T12:09:21.619765+0000","last_clean":"2024-03-24T12:09:21.619765+0000","last_became_active":"2024-03-24T12:06:30.015853+0000","last_became_peered":"2024-03-24T12:06:30.015853+0000","last_unstale":"2024-03-24T12:09:21.619765+0000","last_undegraded":"2024-03-24T12:09:21.619765+0000","last_fullsized":"2024-03-24T12:09:21.619765+0000","mapping_epoch":320,"log_start":"344'9551","ondisk_log_start":"344'9551","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'8176","last_scrub_stamp":"2024-03-24T12:07:58.231548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.231548+0000","objects_scrubbed":8101,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:26.835989+0000","scrub_duration":2379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261095424,"num_objects":3984,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14060,"num_write_kb":577408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":873,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.942973+0000","last_change":"2024-03-24T12:07:34.996457+0000","last_active":"2024-03-24T12:09:18.942973+0000","last_peered":"2024-03-24T12:09:18.942973+0000","last_clean":"2024-03-24T12:09:18.942973+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:09:18.942973+0000","last_undegraded":"2024-03-24T12:09:18.942973+0000","last_fullsized":"2024-03-24T12:09:18.942973+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:07:34.996314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:07:34.996314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":719,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952762+0000","last_change":"2024-03-24T12:08:36.162868+0000","last_active":"2024-03-24T12:09:18.952762+0000","last_peered":"2024-03-24T12:09:18.952762+0000","last_clean":"2024-03-24T12:09:18.952762+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:09:18.952762+0000","last_undegraded":"2024-03-24T12:09:18.952762+0000","last_fullsized":"2024-03-24T12:09:18.952762+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:05:29.699282+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:05:29.699282+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":679,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.950017+0000","last_change":"2024-03-24T12:07:30.557017+0000","last_active":"2024-03-24T12:09:18.950017+0000","last_peered":"2024-03-24T12:09:18.950017+0000","last_clean":"2024-03-24T12:09:18.950017+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:09:18.950017+0000","last_undegraded":"2024-03-24T12:09:18.950017+0000","last_fullsized":"2024-03-24T12:09:18.950017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.556903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.556903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":684,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960316+0000","last_change":"2024-03-24T12:07:45.151754+0000","last_active":"2024-03-24T12:09:18.960316+0000","last_peered":"2024-03-24T12:09:18.960316+0000","last_clean":"2024-03-24T12:09:18.960316+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:09:18.960316+0000","last_undegraded":"2024-03-24T12:09:18.960316+0000","last_fullsized":"2024-03-24T12:09:18.960316+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:45.151622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.151622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191403+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":709,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943500+0000","last_change":"2024-03-24T12:08:36.201644+0000","last_active":"2024-03-24T12:09:18.943500+0000","last_peered":"2024-03-24T12:09:18.943500+0000","last_clean":"2024-03-24T12:09:18.943500+0000","last_became_active":"2024-03-24T12:08:36.146325+0000","last_became_peered":"2024-03-24T12:08:36.146325+0000","last_unstale":"2024-03-24T12:09:18.943500+0000","last_undegraded":"2024-03-24T12:09:18.943500+0000","last_fullsized":"2024-03-24T12:09:18.943500+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:36.201565+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.201565+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:50.129847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"367'13903","reported_seq":24846,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.556017+0000","last_change":"2024-03-24T12:09:17.977005+0000","last_active":"2024-03-24T12:09:21.556017+0000","last_peered":"2024-03-24T12:09:21.556017+0000","last_clean":"2024-03-24T12:09:21.556017+0000","last_became_active":"2024-03-24T12:09:17.945785+0000","last_became_peered":"2024-03-24T12:09:17.945785+0000","last_unstale":"2024-03-24T12:09:21.556017+0000","last_undegraded":"2024-03-24T12:09:21.556017+0000","last_fullsized":"2024-03-24T12:09:21.556017+0000","mapping_epoch":365,"log_start":"347'9600","ondisk_log_start":"347'9600","created":320,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253689856,"num_objects":3871,"num_object_clones":0,"num_object_copies":11613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13903,"num_write_kb":568768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5413,"num_bytes_recovered":354680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":657,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.953548+0000","last_change":"2024-03-24T12:06:44.708778+0000","last_active":"2024-03-24T12:09:18.953548+0000","last_peered":"2024-03-24T12:09:18.953548+0000","last_clean":"2024-03-24T12:09:18.953548+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:09:18.953548+0000","last_undegraded":"2024-03-24T12:09:18.953548+0000","last_fullsized":"2024-03-24T12:09:18.953548+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:44.708677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:06:44.708677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":721,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.942898+0000","last_change":"2024-03-24T12:06:54.925013+0000","last_active":"2024-03-24T12:09:18.942898+0000","last_peered":"2024-03-24T12:09:18.942898+0000","last_clean":"2024-03-24T12:09:18.942898+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:09:18.942898+0000","last_undegraded":"2024-03-24T12:09:18.942898+0000","last_fullsized":"2024-03-24T12:09:18.942898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:06:54.924751+0000","last_deep_scrub":"195'26","last_deep_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.924751+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":432,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949473+0000","last_change":"2024-03-24T12:07:40.603920+0000","last_active":"2024-03-24T12:09:18.949473+0000","last_peered":"2024-03-24T12:09:18.949473+0000","last_clean":"2024-03-24T12:09:18.949473+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:09:18.949473+0000","last_undegraded":"2024-03-24T12:09:18.949473+0000","last_fullsized":"2024-03-24T12:09:18.949473+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:07:40.603768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.603768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":675,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952278+0000","last_change":"2024-03-24T12:06:39.791204+0000","last_active":"2024-03-24T12:09:18.952278+0000","last_peered":"2024-03-24T12:09:18.952278+0000","last_clean":"2024-03-24T12:09:18.952278+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:09:18.952278+0000","last_undegraded":"2024-03-24T12:09:18.952278+0000","last_fullsized":"2024-03-24T12:09:18.952278+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:39.791081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.791081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":718,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952142+0000","last_change":"2024-03-24T12:08:36.163557+0000","last_active":"2024-03-24T12:09:18.952142+0000","last_peered":"2024-03-24T12:09:18.952142+0000","last_clean":"2024-03-24T12:09:18.952142+0000","last_became_active":"2024-03-24T12:08:36.163164+0000","last_became_peered":"2024-03-24T12:08:36.163164+0000","last_unstale":"2024-03-24T12:09:18.952142+0000","last_undegraded":"2024-03-24T12:09:18.952142+0000","last_fullsized":"2024-03-24T12:09:18.952142+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:18.897352+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:18.897352+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"367'13953","reported_seq":14138,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.568393+0000","last_change":"2024-03-24T12:07:46.785479+0000","last_active":"2024-03-24T12:09:21.568393+0000","last_peered":"2024-03-24T12:09:21.568393+0000","last_clean":"2024-03-24T12:09:21.568393+0000","last_became_active":"2024-03-24T12:06:30.017829+0000","last_became_peered":"2024-03-24T12:06:30.017829+0000","last_unstale":"2024-03-24T12:09:21.568393+0000","last_undegraded":"2024-03-24T12:09:21.568393+0000","last_fullsized":"2024-03-24T12:09:21.568393+0000","mapping_epoch":320,"log_start":"345'9508","ondisk_log_start":"345'9508","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7161","last_scrub_stamp":"2024-03-24T12:07:46.785035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:46.785035+0000","objects_scrubbed":7079,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259784728,"num_objects":3965,"num_object_clones":0,"num_object_copies":11895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13953,"num_write_kb":573313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":673,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949013+0000","last_change":"2024-03-24T12:08:36.162113+0000","last_active":"2024-03-24T12:09:18.949013+0000","last_peered":"2024-03-24T12:09:18.949013+0000","last_clean":"2024-03-24T12:09:18.949013+0000","last_became_active":"2024-03-24T12:08:36.161658+0000","last_became_peered":"2024-03-24T12:08:36.161658+0000","last_unstale":"2024-03-24T12:09:18.949013+0000","last_undegraded":"2024-03-24T12:09:18.949013+0000","last_fullsized":"2024-03-24T12:09:18.949013+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:24.576146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:05:24.576146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"241'29","reported_seq":714,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944262+0000","last_change":"2024-03-24T12:08:36.156053+0000","last_active":"2024-03-24T12:09:18.944262+0000","last_peered":"2024-03-24T12:09:18.944262+0000","last_clean":"2024-03-24T12:09:18.944262+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:09:18.944262+0000","last_undegraded":"2024-03-24T12:09:18.944262+0000","last_fullsized":"2024-03-24T12:09:18.944262+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-24T12:05:09.620298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:09.620298+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":690,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960971+0000","last_change":"2024-03-24T12:07:39.003839+0000","last_active":"2024-03-24T12:09:18.960971+0000","last_peered":"2024-03-24T12:09:18.960971+0000","last_clean":"2024-03-24T12:09:18.960971+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:09:18.960971+0000","last_undegraded":"2024-03-24T12:09:18.960971+0000","last_fullsized":"2024-03-24T12:09:18.960971+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:39.003694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:07:39.003694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":422,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.947940+0000","last_change":"2024-03-24T12:06:40.600601+0000","last_active":"2024-03-24T12:09:18.947940+0000","last_peered":"2024-03-24T12:09:18.947940+0000","last_clean":"2024-03-24T12:09:18.947940+0000","last_became_active":"2024-03-24T12:05:44.738926+0000","last_became_peered":"2024-03-24T12:05:44.738926+0000","last_unstale":"2024-03-24T12:09:18.947940+0000","last_undegraded":"2024-03-24T12:09:18.947940+0000","last_fullsized":"2024-03-24T12:09:18.947940+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:40.600197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:40.600197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":730,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949568+0000","last_change":"2024-03-24T12:08:35.901951+0000","last_active":"2024-03-24T12:09:18.949568+0000","last_peered":"2024-03-24T12:09:18.949568+0000","last_clean":"2024-03-24T12:09:18.949568+0000","last_became_active":"2024-03-24T12:03:15.295100+0000","last_became_peered":"2024-03-24T12:03:15.295100+0000","last_unstale":"2024-03-24T12:09:18.949568+0000","last_undegraded":"2024-03-24T12:09:18.949568+0000","last_fullsized":"2024-03-24T12:09:18.949568+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:35.901858+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:35.901858+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:49.096478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"367'14118","reported_seq":14301,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:23.460952+0000","last_change":"2024-03-24T12:08:34.971600+0000","last_active":"2024-03-24T12:09:23.460952+0000","last_peered":"2024-03-24T12:09:23.460952+0000","last_clean":"2024-03-24T12:09:23.460952+0000","last_became_active":"2024-03-24T12:06:30.015684+0000","last_became_peered":"2024-03-24T12:06:30.015684+0000","last_unstale":"2024-03-24T12:09:23.460952+0000","last_undegraded":"2024-03-24T12:09:23.460952+0000","last_fullsized":"2024-03-24T12:09:23.460952+0000","mapping_epoch":320,"log_start":"346'9600","ondisk_log_start":"346'9600","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'11035","last_scrub_stamp":"2024-03-24T12:08:34.971218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.971218+0000","objects_scrubbed":6993,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:47.107246+0000","scrub_duration":2351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14118,"num_write_kb":574720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":551,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949326+0000","last_change":"2024-03-24T12:08:36.165575+0000","last_active":"2024-03-24T12:09:18.949326+0000","last_peered":"2024-03-24T12:09:18.949326+0000","last_clean":"2024-03-24T12:09:18.949326+0000","last_became_active":"2024-03-24T12:08:36.165265+0000","last_became_peered":"2024-03-24T12:08:36.165265+0000","last_unstale":"2024-03-24T12:09:18.949326+0000","last_undegraded":"2024-03-24T12:09:18.949326+0000","last_fullsized":"2024-03-24T12:09:18.949326+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:28.232417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:28.232417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":723,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945334+0000","last_change":"2024-03-24T12:06:39.802730+0000","last_active":"2024-03-24T12:09:18.945334+0000","last_peered":"2024-03-24T12:09:18.945334+0000","last_clean":"2024-03-24T12:09:18.945334+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:09:18.945334+0000","last_undegraded":"2024-03-24T12:09:18.945334+0000","last_fullsized":"2024-03-24T12:09:18.945334+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:06:39.802577+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.802577+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":510,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944199+0000","last_change":"2024-03-24T12:07:34.290389+0000","last_active":"2024-03-24T12:09:18.944199+0000","last_peered":"2024-03-24T12:09:18.944199+0000","last_clean":"2024-03-24T12:09:18.944199+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:09:18.944199+0000","last_undegraded":"2024-03-24T12:09:18.944199+0000","last_fullsized":"2024-03-24T12:09:18.944199+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:34.290275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:34.290275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":681,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944761+0000","last_change":"2024-03-24T12:07:39.389030+0000","last_active":"2024-03-24T12:09:18.944761+0000","last_peered":"2024-03-24T12:09:18.944761+0000","last_clean":"2024-03-24T12:09:18.944761+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:09:18.944761+0000","last_undegraded":"2024-03-24T12:09:18.944761+0000","last_fullsized":"2024-03-24T12:09:18.944761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:39.387609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:07:39.387609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":679,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959349+0000","last_change":"2024-03-24T12:08:38.243974+0000","last_active":"2024-03-24T12:09:18.959349+0000","last_peered":"2024-03-24T12:09:18.959349+0000","last_clean":"2024-03-24T12:09:18.959349+0000","last_became_active":"2024-03-24T12:08:36.173357+0000","last_became_peered":"2024-03-24T12:08:36.173357+0000","last_unstale":"2024-03-24T12:09:18.959349+0000","last_undegraded":"2024-03-24T12:09:18.959349+0000","last_fullsized":"2024-03-24T12:09:18.959349+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:38.243821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:38.243821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:01.303017+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":722,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948956+0000","last_change":"2024-03-24T12:06:54.603424+0000","last_active":"2024-03-24T12:09:18.948956+0000","last_peered":"2024-03-24T12:09:18.948956+0000","last_clean":"2024-03-24T12:09:18.948956+0000","last_became_active":"2024-03-24T12:03:15.297422+0000","last_became_peered":"2024-03-24T12:03:15.297422+0000","last_unstale":"2024-03-24T12:09:18.948956+0000","last_undegraded":"2024-03-24T12:09:18.948956+0000","last_fullsized":"2024-03-24T12:09:18.948956+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:54.603313+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.603313+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"367'13908","reported_seq":14094,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.634164+0000","last_change":"2024-03-24T12:08:31.898942+0000","last_active":"2024-03-24T12:09:21.634164+0000","last_peered":"2024-03-24T12:09:21.634164+0000","last_clean":"2024-03-24T12:09:21.634164+0000","last_became_active":"2024-03-24T12:06:30.017471+0000","last_became_peered":"2024-03-24T12:06:30.017471+0000","last_unstale":"2024-03-24T12:09:21.634164+0000","last_undegraded":"2024-03-24T12:09:21.634164+0000","last_fullsized":"2024-03-24T12:09:21.634164+0000","mapping_epoch":320,"log_start":"344'9463","ondisk_log_start":"344'9463","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'10717","last_scrub_stamp":"2024-03-24T12:08:31.898767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:08:31.898767+0000","objects_scrubbed":7350,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.390776+0000","scrub_duration":2107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266469376,"num_objects":4066,"num_object_clones":0,"num_object_copies":12198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13908,"num_write_kb":575168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":683,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960693+0000","last_change":"2024-03-24T12:08:43.232781+0000","last_active":"2024-03-24T12:09:18.960693+0000","last_peered":"2024-03-24T12:09:18.960693+0000","last_clean":"2024-03-24T12:09:18.960693+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:09:18.960693+0000","last_undegraded":"2024-03-24T12:09:18.960693+0000","last_fullsized":"2024-03-24T12:09:18.960693+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:43.232649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.232649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:07.262241+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":938,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949425+0000","last_change":"2024-03-24T12:08:32.823527+0000","last_active":"2024-03-24T12:09:18.949425+0000","last_peered":"2024-03-24T12:09:18.949425+0000","last_clean":"2024-03-24T12:09:18.949425+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:09:18.949425+0000","last_undegraded":"2024-03-24T12:09:18.949425+0000","last_fullsized":"2024-03-24T12:09:18.949425+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:08:32.823398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:32.823398+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:44.902148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":962,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959521+0000","last_change":"2024-03-24T12:08:49.185014+0000","last_active":"2024-03-24T12:09:18.959521+0000","last_peered":"2024-03-24T12:09:18.959521+0000","last_clean":"2024-03-24T12:09:18.959521+0000","last_became_active":"2024-03-24T12:03:15.298475+0000","last_became_peered":"2024-03-24T12:03:15.298475+0000","last_unstale":"2024-03-24T12:09:18.959521+0000","last_undegraded":"2024-03-24T12:09:18.959521+0000","last_fullsized":"2024-03-24T12:09:18.959521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:49.184886+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:03:16.845832+0000","last_clean_scrub_stamp":"2024-03-24T12:08:49.184886+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:54.517988+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":591,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948733+0000","last_change":"2024-03-24T12:08:36.169985+0000","last_active":"2024-03-24T12:09:18.948733+0000","last_peered":"2024-03-24T12:09:18.948733+0000","last_clean":"2024-03-24T12:09:18.948733+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:09:18.948733+0000","last_undegraded":"2024-03-24T12:09:18.948733+0000","last_fullsized":"2024-03-24T12:09:18.948733+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.356759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.356759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":685,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943077+0000","last_change":"2024-03-24T12:08:43.113674+0000","last_active":"2024-03-24T12:09:18.943077+0000","last_peered":"2024-03-24T12:09:18.943077+0000","last_clean":"2024-03-24T12:09:18.943077+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:09:18.943077+0000","last_undegraded":"2024-03-24T12:09:18.943077+0000","last_fullsized":"2024-03-24T12:09:18.943077+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:08:43.113510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.113510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:59.354499+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":546,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949294+0000","last_change":"2024-03-24T12:08:36.162637+0000","last_active":"2024-03-24T12:09:18.949294+0000","last_peered":"2024-03-24T12:09:18.949294+0000","last_clean":"2024-03-24T12:09:18.949294+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:09:18.949294+0000","last_undegraded":"2024-03-24T12:09:18.949294+0000","last_fullsized":"2024-03-24T12:09:18.949294+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.463242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.463242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"367'13998","reported_seq":15848,"reported_epoch":367,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-24T12:09:21.554301+0000","last_change":"2024-03-24T12:09:17.687057+0000","last_active":"2024-03-24T12:09:21.554301+0000","last_peered":"2024-03-24T12:09:21.554301+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:08:37.144663+0000","last_became_peered":"2024-03-24T12:08:37.144663+0000","last_unstale":"2024-03-24T12:09:21.554301+0000","last_undegraded":"2024-03-24T12:08:36.938871+0000","last_fullsized":"2024-03-24T12:08:36.916554+0000","mapping_epoch":354,"log_start":"347'9700","ondisk_log_start":"347'9700","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249954304,"num_objects":3814,"num_object_clones":0,"num_object_copies":11442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13998,"num_write_kb":569984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":57081856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":3814}],"blocked_by":[],"up_primary":5,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948539+0000","last_change":"2024-03-24T12:08:36.162550+0000","last_active":"2024-03-24T12:09:18.948539+0000","last_peered":"2024-03-24T12:09:18.948539+0000","last_clean":"2024-03-24T12:09:18.948539+0000","last_became_active":"2024-03-24T12:08:36.162188+0000","last_became_peered":"2024-03-24T12:08:36.162188+0000","last_unstale":"2024-03-24T12:09:18.948539+0000","last_undegraded":"2024-03-24T12:09:18.948539+0000","last_fullsized":"2024-03-24T12:09:18.948539+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.646040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":719,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959558+0000","last_change":"2024-03-24T12:08:41.237306+0000","last_active":"2024-03-24T12:09:18.959558+0000","last_peered":"2024-03-24T12:09:18.959558+0000","last_clean":"2024-03-24T12:09:18.959558+0000","last_became_active":"2024-03-24T12:08:36.172879+0000","last_became_peered":"2024-03-24T12:08:36.172879+0000","last_unstale":"2024-03-24T12:09:18.959558+0000","last_undegraded":"2024-03-24T12:09:18.959558+0000","last_fullsized":"2024-03-24T12:09:18.959558+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:08:41.237149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:41.237149+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:42.468495+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944080+0000","last_change":"2024-03-24T12:05:44.739463+0000","last_active":"2024-03-24T12:09:18.944080+0000","last_peered":"2024-03-24T12:09:18.944080+0000","last_clean":"2024-03-24T12:09:18.944080+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:09:18.944080+0000","last_undegraded":"2024-03-24T12:09:18.944080+0000","last_fullsized":"2024-03-24T12:09:18.944080+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.610545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.610545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":415,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:20.179197+0000","last_change":"2024-03-24T12:06:55.648298+0000","last_active":"2024-03-24T12:09:20.179197+0000","last_peered":"2024-03-24T12:09:20.179197+0000","last_clean":"2024-03-24T12:09:20.179197+0000","last_became_active":"2024-03-24T12:05:44.736437+0000","last_became_peered":"2024-03-24T12:05:44.736437+0000","last_unstale":"2024-03-24T12:09:20.179197+0000","last_undegraded":"2024-03-24T12:09:20.179197+0000","last_fullsized":"2024-03-24T12:09:20.179197+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:06:55.648166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:55.648166+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":223,"num_read_kb":218,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":670,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960884+0000","last_change":"2024-03-24T12:07:28.069295+0000","last_active":"2024-03-24T12:09:18.960884+0000","last_peered":"2024-03-24T12:09:18.960884+0000","last_clean":"2024-03-24T12:09:18.960884+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:09:18.960884+0000","last_undegraded":"2024-03-24T12:09:18.960884+0000","last_fullsized":"2024-03-24T12:09:18.960884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:28.068587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:28.068587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191325+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":961,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960342+0000","last_change":"2024-03-24T12:06:56.127437+0000","last_active":"2024-03-24T12:09:18.960342+0000","last_peered":"2024-03-24T12:09:18.960342+0000","last_clean":"2024-03-24T12:09:18.960342+0000","last_became_active":"2024-03-24T12:03:15.298007+0000","last_became_peered":"2024-03-24T12:03:15.298007+0000","last_unstale":"2024-03-24T12:09:18.960342+0000","last_undegraded":"2024-03-24T12:09:18.960342+0000","last_fullsized":"2024-03-24T12:09:18.960342+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:56.127305+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:56.127305+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191251+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"367'14142","reported_seq":16457,"reported_epoch":367,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-24T12:09:21.637174+0000","last_change":"2024-03-24T12:09:17.176566+0000","last_active":"2024-03-24T12:09:21.637174+0000","last_peered":"2024-03-24T12:09:21.637174+0000","last_clean":"2024-03-24T12:06:28.998031+0000","last_became_active":"2024-03-24T12:08:37.151019+0000","last_became_peered":"2024-03-24T12:08:37.151019+0000","last_unstale":"2024-03-24T12:09:21.637174+0000","last_undegraded":"2024-03-24T12:08:36.935750+0000","last_fullsized":"2024-03-24T12:08:36.917236+0000","mapping_epoch":354,"log_start":"346'9645","ondisk_log_start":"346'9645","created":320,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254148608,"num_objects":3878,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14142,"num_write_kb":576640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":72220672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":3878}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":658,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945624+0000","last_change":"2024-03-24T12:06:47.944634+0000","last_active":"2024-03-24T12:09:18.945624+0000","last_peered":"2024-03-24T12:09:18.945624+0000","last_clean":"2024-03-24T12:09:18.945624+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:09:18.945624+0000","last_undegraded":"2024-03-24T12:09:18.945624+0000","last_fullsized":"2024-03-24T12:09:18.945624+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:47.944504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:06:47.944504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"354'44","reported_seq":776,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948555+0000","last_change":"2024-03-24T12:08:36.153332+0000","last_active":"2024-03-24T12:09:18.948555+0000","last_peered":"2024-03-24T12:09:18.948555+0000","last_clean":"2024-03-24T12:09:18.948555+0000","last_became_active":"2024-03-24T12:08:36.152866+0000","last_became_peered":"2024-03-24T12:08:36.152866+0000","last_unstale":"2024-03-24T12:09:18.948555+0000","last_undegraded":"2024-03-24T12:09:18.948555+0000","last_fullsized":"2024-03-24T12:09:18.948555+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-24T12:04:07.131550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:07.131550+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":555,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948116+0000","last_change":"2024-03-24T12:08:36.170514+0000","last_active":"2024-03-24T12:09:18.948116+0000","last_peered":"2024-03-24T12:09:18.948116+0000","last_clean":"2024-03-24T12:09:18.948116+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:09:18.948116+0000","last_undegraded":"2024-03-24T12:09:18.948116+0000","last_fullsized":"2024-03-24T12:09:18.948116+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.196749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.196749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":654,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945657+0000","last_change":"2024-03-24T12:07:33.984169+0000","last_active":"2024-03-24T12:09:18.945657+0000","last_peered":"2024-03-24T12:09:18.945657+0000","last_clean":"2024-03-24T12:09:18.945657+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:09:18.945657+0000","last_undegraded":"2024-03-24T12:09:18.945657+0000","last_fullsized":"2024-03-24T12:09:18.945657+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:33.984042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:33.984042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":598,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949848+0000","last_change":"2024-03-24T12:08:36.157710+0000","last_active":"2024-03-24T12:09:18.949848+0000","last_peered":"2024-03-24T12:09:18.949848+0000","last_clean":"2024-03-24T12:09:18.949848+0000","last_became_active":"2024-03-24T12:08:36.155758+0000","last_became_peered":"2024-03-24T12:08:36.155758+0000","last_unstale":"2024-03-24T12:09:18.949848+0000","last_undegraded":"2024-03-24T12:09:18.949848+0000","last_fullsized":"2024-03-24T12:09:18.949848+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:47.152371+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:47.152371+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.712893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"367'14089","reported_seq":14266,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.646382+0000","last_change":"2024-03-24T12:08:50.231238+0000","last_active":"2024-03-24T12:09:21.646382+0000","last_peered":"2024-03-24T12:09:21.646382+0000","last_clean":"2024-03-24T12:09:21.646382+0000","last_became_active":"2024-03-24T12:06:30.017875+0000","last_became_peered":"2024-03-24T12:06:30.017875+0000","last_unstale":"2024-03-24T12:09:21.646382+0000","last_undegraded":"2024-03-24T12:09:21.646382+0000","last_fullsized":"2024-03-24T12:09:21.646382+0000","mapping_epoch":320,"log_start":"347'9800","ondisk_log_start":"347'9800","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'11995","last_scrub_stamp":"2024-03-24T12:08:50.231089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:08:50.231089+0000","objects_scrubbed":6181,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:09.714731+0000","scrub_duration":2108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263389184,"num_objects":4019,"num_object_clones":0,"num_object_copies":12057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14089,"num_write_kb":579456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":891,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944580+0000","last_change":"2024-03-24T12:08:36.673921+0000","last_active":"2024-03-24T12:09:18.944580+0000","last_peered":"2024-03-24T12:09:18.944580+0000","last_clean":"2024-03-24T12:09:18.944580+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:09:18.944580+0000","last_undegraded":"2024-03-24T12:09:18.944580+0000","last_fullsized":"2024-03-24T12:09:18.944580+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:08:36.673814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.673814+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:43.939613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":722,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959872+0000","last_change":"2024-03-24T12:08:39.239739+0000","last_active":"2024-03-24T12:09:18.959872+0000","last_peered":"2024-03-24T12:09:18.959872+0000","last_clean":"2024-03-24T12:09:18.959872+0000","last_became_active":"2024-03-24T12:08:36.174663+0000","last_became_peered":"2024-03-24T12:08:36.174663+0000","last_unstale":"2024-03-24T12:09:18.959872+0000","last_undegraded":"2024-03-24T12:09:18.959872+0000","last_fullsized":"2024-03-24T12:09:18.959872+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:08:39.239229+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:08:39.239229+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:50.536814+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":416,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944304+0000","last_change":"2024-03-24T12:06:49.982193+0000","last_active":"2024-03-24T12:09:18.944304+0000","last_peered":"2024-03-24T12:09:18.944304+0000","last_clean":"2024-03-24T12:09:18.944304+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:09:18.944304+0000","last_undegraded":"2024-03-24T12:09:18.944304+0000","last_fullsized":"2024-03-24T12:09:18.944304+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:49.981897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.981897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":669,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.734609+0000","last_change":"2024-03-24T12:09:21.734702+0000","last_active":"2024-03-24T12:09:21.734609+0000","last_peered":"2024-03-24T12:09:21.734609+0000","last_clean":"2024-03-24T12:09:21.734609+0000","last_became_active":"2024-03-24T12:08:36.145936+0000","last_became_peered":"2024-03-24T12:08:36.145936+0000","last_unstale":"2024-03-24T12:09:21.734609+0000","last_undegraded":"2024-03-24T12:09:21.734609+0000","last_fullsized":"2024-03-24T12:09:21.734609+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:21.734396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:09:21.734396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:50.267058+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":722,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949680+0000","last_change":"2024-03-24T12:07:32.603069+0000","last_active":"2024-03-24T12:09:18.949680+0000","last_peered":"2024-03-24T12:09:18.949680+0000","last_clean":"2024-03-24T12:09:18.949680+0000","last_became_active":"2024-03-24T12:03:15.298983+0000","last_became_peered":"2024-03-24T12:03:15.298983+0000","last_unstale":"2024-03-24T12:09:18.949680+0000","last_undegraded":"2024-03-24T12:09:18.949680+0000","last_fullsized":"2024-03-24T12:09:18.949680+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:32.602779+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:07:32.602779+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"367'14227","reported_seq":25144,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:21.619640+0000","last_change":"2024-03-24T12:09:18.987861+0000","last_active":"2024-03-24T12:09:21.619640+0000","last_peered":"2024-03-24T12:09:21.619640+0000","last_clean":"2024-03-24T12:09:21.619640+0000","last_became_active":"2024-03-24T12:09:18.950940+0000","last_became_peered":"2024-03-24T12:09:18.950940+0000","last_unstale":"2024-03-24T12:09:21.619640+0000","last_undegraded":"2024-03-24T12:09:21.619640+0000","last_fullsized":"2024-03-24T12:09:21.619640+0000","mapping_epoch":366,"log_start":"347'9814","ondisk_log_start":"347'9814","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256966656,"num_objects":3921,"num_object_clones":0,"num_object_copies":11763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14227,"num_write_kb":580736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5401,"num_bytes_recovered":353959936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"83'130","reported_seq":713,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945167+0000","last_change":"2024-03-24T12:07:27.057073+0000","last_active":"2024-03-24T12:09:18.945167+0000","last_peered":"2024-03-24T12:09:18.945167+0000","last_clean":"2024-03-24T12:09:18.945167+0000","last_became_active":"2024-03-24T12:03:19.827283+0000","last_became_peered":"2024-03-24T12:03:19.827283+0000","last_unstale":"2024-03-24T12:09:18.945167+0000","last_undegraded":"2024-03-24T12:09:18.945167+0000","last_fullsized":"2024-03-24T12:09:18.945167+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:27.056921+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:27.056921+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":682,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.946213+0000","last_change":"2024-03-24T12:08:32.727236+0000","last_active":"2024-03-24T12:09:18.946213+0000","last_peered":"2024-03-24T12:09:18.946213+0000","last_clean":"2024-03-24T12:09:18.946213+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:09:18.946213+0000","last_undegraded":"2024-03-24T12:09:18.946213+0000","last_fullsized":"2024-03-24T12:09:18.946213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:32.727103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:32.727103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:34.854286+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":726,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952873+0000","last_change":"2024-03-24T12:08:34.327181+0000","last_active":"2024-03-24T12:09:18.952873+0000","last_peered":"2024-03-24T12:09:18.952873+0000","last_clean":"2024-03-24T12:09:18.952873+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:09:18.952873+0000","last_undegraded":"2024-03-24T12:09:18.952873+0000","last_fullsized":"2024-03-24T12:09:18.952873+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:08:34.327037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.327037+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:02.132550+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":684,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952649+0000","last_change":"2024-03-24T12:07:45.216727+0000","last_active":"2024-03-24T12:09:18.952649+0000","last_peered":"2024-03-24T12:09:18.952649+0000","last_clean":"2024-03-24T12:09:18.952649+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:09:18.952649+0000","last_undegraded":"2024-03-24T12:09:18.952649+0000","last_fullsized":"2024-03-24T12:09:18.952649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:07:45.216581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.216581+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":654,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944224+0000","last_change":"2024-03-24T12:07:47.318834+0000","last_active":"2024-03-24T12:09:18.944224+0000","last_peered":"2024-03-24T12:09:18.944224+0000","last_clean":"2024-03-24T12:09:18.944224+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:09:18.944224+0000","last_undegraded":"2024-03-24T12:09:18.944224+0000","last_fullsized":"2024-03-24T12:09:18.944224+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:47.318040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:07:47.318040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"83'130","reported_seq":538,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943515+0000","last_change":"2024-03-24T12:06:52.949501+0000","last_active":"2024-03-24T12:09:18.943515+0000","last_peered":"2024-03-24T12:09:18.943515+0000","last_clean":"2024-03-24T12:09:18.943515+0000","last_became_active":"2024-03-24T12:03:19.828385+0000","last_became_peered":"2024-03-24T12:03:19.828385+0000","last_unstale":"2024-03-24T12:09:18.943515+0000","last_undegraded":"2024-03-24T12:09:18.943515+0000","last_fullsized":"2024-03-24T12:09:18.943515+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:52.949381+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:52.949381+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":698,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945374+0000","last_change":"2024-03-24T12:07:26.164276+0000","last_active":"2024-03-24T12:09:18.945374+0000","last_peered":"2024-03-24T12:09:18.945374+0000","last_clean":"2024-03-24T12:09:18.945374+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:09:18.945374+0000","last_undegraded":"2024-03-24T12:09:18.945374+0000","last_fullsized":"2024-03-24T12:09:18.945374+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:07:26.164151+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:07:26.164151+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":154,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943937+0000","last_change":"2024-03-24T12:06:51.959021+0000","last_active":"2024-03-24T12:09:18.943937+0000","last_peered":"2024-03-24T12:09:18.943937+0000","last_clean":"2024-03-24T12:09:18.943937+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:09:18.943937+0000","last_undegraded":"2024-03-24T12:09:18.943937+0000","last_fullsized":"2024-03-24T12:09:18.943937+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:51.958902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:51.958902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":650,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945058+0000","last_change":"2024-03-24T12:06:42.950243+0000","last_active":"2024-03-24T12:09:18.945058+0000","last_peered":"2024-03-24T12:09:18.945058+0000","last_clean":"2024-03-24T12:09:18.945058+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:09:18.945058+0000","last_undegraded":"2024-03-24T12:09:18.945058+0000","last_fullsized":"2024-03-24T12:09:18.945058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.950004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.950004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":153,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952208+0000","last_change":"2024-03-24T12:07:50.278024+0000","last_active":"2024-03-24T12:09:18.952208+0000","last_peered":"2024-03-24T12:09:18.952208+0000","last_clean":"2024-03-24T12:09:18.952208+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:09:18.952208+0000","last_undegraded":"2024-03-24T12:09:18.952208+0000","last_fullsized":"2024-03-24T12:09:18.952208+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:50.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:50.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.542018+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"83'130","reported_seq":459,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948363+0000","last_change":"2024-03-24T12:08:36.171122+0000","last_active":"2024-03-24T12:09:18.948363+0000","last_peered":"2024-03-24T12:09:18.948363+0000","last_clean":"2024-03-24T12:09:18.948363+0000","last_became_active":"2024-03-24T12:08:36.167969+0000","last_became_peered":"2024-03-24T12:08:36.167969+0000","last_unstale":"2024-03-24T12:09:18.948363+0000","last_undegraded":"2024-03-24T12:09:18.948363+0000","last_fullsized":"2024-03-24T12:09:18.948363+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:48.080209+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.080209+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.712920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":680,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944958+0000","last_change":"2024-03-24T12:06:49.874361+0000","last_active":"2024-03-24T12:09:18.944958+0000","last_peered":"2024-03-24T12:09:18.944958+0000","last_clean":"2024-03-24T12:09:18.944958+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:09:18.944958+0000","last_undegraded":"2024-03-24T12:09:18.944958+0000","last_fullsized":"2024-03-24T12:09:18.944958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:06:49.873894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.873894+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":687,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948177+0000","last_change":"2024-03-24T12:08:30.827983+0000","last_active":"2024-03-24T12:09:18.948177+0000","last_peered":"2024-03-24T12:09:18.948177+0000","last_clean":"2024-03-24T12:09:18.948177+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:09:18.948177+0000","last_undegraded":"2024-03-24T12:09:18.948177+0000","last_fullsized":"2024-03-24T12:09:18.948177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:30.827866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:08:30.827866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:48.184568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":503,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948684+0000","last_change":"2024-03-24T12:08:36.170084+0000","last_active":"2024-03-24T12:09:18.948684+0000","last_peered":"2024-03-24T12:09:18.948684+0000","last_clean":"2024-03-24T12:09:18.948684+0000","last_became_active":"2024-03-24T12:08:36.165715+0000","last_became_peered":"2024-03-24T12:08:36.165715+0000","last_unstale":"2024-03-24T12:09:18.948684+0000","last_undegraded":"2024-03-24T12:09:18.948684+0000","last_fullsized":"2024-03-24T12:09:18.948684+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:11.242845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:11.242845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":672,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960630+0000","last_change":"2024-03-24T12:07:30.073083+0000","last_active":"2024-03-24T12:09:18.960630+0000","last_peered":"2024-03-24T12:09:18.960630+0000","last_clean":"2024-03-24T12:09:18.960630+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:09:18.960630+0000","last_undegraded":"2024-03-24T12:09:18.960630+0000","last_fullsized":"2024-03-24T12:09:18.960630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.072953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.072953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191427+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"83'130","reported_seq":558,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944516+0000","last_change":"2024-03-24T12:08:36.144336+0000","last_active":"2024-03-24T12:09:18.944516+0000","last_peered":"2024-03-24T12:09:18.944516+0000","last_clean":"2024-03-24T12:09:18.944516+0000","last_became_active":"2024-03-24T12:08:36.143986+0000","last_became_peered":"2024-03-24T12:08:36.143986+0000","last_unstale":"2024-03-24T12:09:18.944516+0000","last_undegraded":"2024-03-24T12:09:18.944516+0000","last_fullsized":"2024-03-24T12:09:18.944516+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:21.650437+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:21.650437+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":697,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943397+0000","last_change":"2024-03-24T12:06:53.944441+0000","last_active":"2024-03-24T12:09:18.943397+0000","last_peered":"2024-03-24T12:09:18.943397+0000","last_clean":"2024-03-24T12:09:18.943397+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:09:18.943397+0000","last_undegraded":"2024-03-24T12:09:18.943397+0000","last_fullsized":"2024-03-24T12:09:18.943397+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:06:53.944281+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:06:53.944281+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":554,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949041+0000","last_change":"2024-03-24T12:08:36.171996+0000","last_active":"2024-03-24T12:09:18.949041+0000","last_peered":"2024-03-24T12:09:18.949041+0000","last_clean":"2024-03-24T12:09:18.949041+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:09:18.949041+0000","last_undegraded":"2024-03-24T12:09:18.949041+0000","last_fullsized":"2024-03-24T12:09:18.949041+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.284540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.284540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":775,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949725+0000","last_change":"2024-03-24T12:08:36.167205+0000","last_active":"2024-03-24T12:09:18.949725+0000","last_peered":"2024-03-24T12:09:18.949725+0000","last_clean":"2024-03-24T12:09:18.949725+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:09:18.949725+0000","last_undegraded":"2024-03-24T12:09:18.949725+0000","last_fullsized":"2024-03-24T12:09:18.949725+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:03:51.155727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.155727+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":661,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945227+0000","last_change":"2024-03-24T12:07:58.405060+0000","last_active":"2024-03-24T12:09:18.945227+0000","last_peered":"2024-03-24T12:09:18.945227+0000","last_clean":"2024-03-24T12:09:18.945227+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:09:18.945227+0000","last_undegraded":"2024-03-24T12:09:18.945227+0000","last_fullsized":"2024-03-24T12:09:18.945227+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:58.404932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.404932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"83'130","reported_seq":469,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948132+0000","last_change":"2024-03-24T12:08:36.620186+0000","last_active":"2024-03-24T12:09:18.948132+0000","last_peered":"2024-03-24T12:09:18.948132+0000","last_clean":"2024-03-24T12:09:18.948132+0000","last_became_active":"2024-03-24T12:08:36.152274+0000","last_became_peered":"2024-03-24T12:08:36.152274+0000","last_unstale":"2024-03-24T12:09:18.948132+0000","last_undegraded":"2024-03-24T12:09:18.948132+0000","last_fullsized":"2024-03-24T12:09:18.948132+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:36.620055+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.620055+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:05.250482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":579,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949973+0000","last_change":"2024-03-24T12:08:36.154109+0000","last_active":"2024-03-24T12:09:18.949973+0000","last_peered":"2024-03-24T12:09:18.949973+0000","last_clean":"2024-03-24T12:09:18.949973+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:09:18.949973+0000","last_undegraded":"2024-03-24T12:09:18.949973+0000","last_fullsized":"2024-03-24T12:09:18.949973+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":57,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948753+0000","last_change":"2024-03-24T12:08:15.996146+0000","last_active":"2024-03-24T12:09:18.948753+0000","last_peered":"2024-03-24T12:09:18.948753+0000","last_clean":"2024-03-24T12:09:18.948753+0000","last_became_active":"2024-03-24T12:08:15.995868+0000","last_became_peered":"2024-03-24T12:08:15.995868+0000","last_unstale":"2024-03-24T12:09:18.948753+0000","last_undegraded":"2024-03-24T12:09:18.948753+0000","last_fullsized":"2024-03-24T12:09:18.948753+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:45.449624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.449624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":882,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943719+0000","last_change":"2024-03-24T12:06:10.014367+0000","last_active":"2024-03-24T12:09:18.943719+0000","last_peered":"2024-03-24T12:09:18.943719+0000","last_clean":"2024-03-24T12:09:18.943719+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:09:18.943719+0000","last_undegraded":"2024-03-24T12:09:18.943719+0000","last_fullsized":"2024-03-24T12:09:18.943719+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:06:10.014224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:10.014224+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":661,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.953605+0000","last_change":"2024-03-24T12:08:36.160517+0000","last_active":"2024-03-24T12:09:18.953605+0000","last_peered":"2024-03-24T12:09:18.953605+0000","last_clean":"2024-03-24T12:09:18.953605+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:09:18.953605+0000","last_undegraded":"2024-03-24T12:09:18.953605+0000","last_fullsized":"2024-03-24T12:09:18.953605+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.041566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.041566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":468,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948729+0000","last_change":"2024-03-24T12:07:38.607052+0000","last_active":"2024-03-24T12:09:18.948729+0000","last_peered":"2024-03-24T12:09:18.948729+0000","last_clean":"2024-03-24T12:09:18.948729+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:09:18.948729+0000","last_undegraded":"2024-03-24T12:09:18.948729+0000","last_fullsized":"2024-03-24T12:09:18.948729+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:38.606914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:38.606914+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":680,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949300+0000","last_change":"2024-03-24T12:08:36.153114+0000","last_active":"2024-03-24T12:09:18.949300+0000","last_peered":"2024-03-24T12:09:18.949300+0000","last_clean":"2024-03-24T12:09:18.949300+0000","last_became_active":"2024-03-24T12:08:36.152805+0000","last_became_peered":"2024-03-24T12:08:36.152805+0000","last_unstale":"2024-03-24T12:09:18.949300+0000","last_undegraded":"2024-03-24T12:09:18.949300+0000","last_fullsized":"2024-03-24T12:09:18.949300+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:05.334379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:04:05.334379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":664,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944136+0000","last_change":"2024-03-24T12:07:29.080020+0000","last_active":"2024-03-24T12:09:18.944136+0000","last_peered":"2024-03-24T12:09:18.944136+0000","last_clean":"2024-03-24T12:09:18.944136+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:09:18.944136+0000","last_undegraded":"2024-03-24T12:09:18.944136+0000","last_fullsized":"2024-03-24T12:09:18.944136+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:29.079435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:07:29.079435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":662,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943594+0000","last_change":"2024-03-24T12:08:36.147539+0000","last_active":"2024-03-24T12:09:18.943594+0000","last_peered":"2024-03-24T12:09:18.943594+0000","last_clean":"2024-03-24T12:09:18.943594+0000","last_became_active":"2024-03-24T12:08:36.146995+0000","last_became_peered":"2024-03-24T12:08:36.146995+0000","last_unstale":"2024-03-24T12:09:18.943594+0000","last_undegraded":"2024-03-24T12:09:18.943594+0000","last_fullsized":"2024-03-24T12:09:18.943594+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:22.691279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:19.108015+0000","last_clean_scrub_stamp":"2024-03-24T12:05:22.691279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":691,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952133+0000","last_change":"2024-03-24T12:06:50.625611+0000","last_active":"2024-03-24T12:09:18.952133+0000","last_peered":"2024-03-24T12:09:18.952133+0000","last_clean":"2024-03-24T12:09:18.952133+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:09:18.952133+0000","last_undegraded":"2024-03-24T12:09:18.952133+0000","last_fullsized":"2024-03-24T12:09:18.952133+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:06:50.625428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:50.625428+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":698,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.959482+0000","last_change":"2024-03-24T12:08:40.215573+0000","last_active":"2024-03-24T12:09:18.959482+0000","last_peered":"2024-03-24T12:09:18.959482+0000","last_clean":"2024-03-24T12:09:18.959482+0000","last_became_active":"2024-03-24T12:08:36.174093+0000","last_became_peered":"2024-03-24T12:08:36.174093+0000","last_unstale":"2024-03-24T12:09:18.959482+0000","last_undegraded":"2024-03-24T12:09:18.959482+0000","last_fullsized":"2024-03-24T12:09:18.959482+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.215463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.215463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:05.250154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":305,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.947759+0000","last_change":"2024-03-24T12:08:36.170017+0000","last_active":"2024-03-24T12:09:18.947759+0000","last_peered":"2024-03-24T12:09:18.947759+0000","last_clean":"2024-03-24T12:09:18.947759+0000","last_became_active":"2024-03-24T12:08:36.166281+0000","last_became_peered":"2024-03-24T12:08:36.166281+0000","last_unstale":"2024-03-24T12:09:18.947759+0000","last_undegraded":"2024-03-24T12:09:18.947759+0000","last_fullsized":"2024-03-24T12:09:18.947759+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:10.080832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:03:10.080832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":655,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943582+0000","last_change":"2024-03-24T12:08:36.145270+0000","last_active":"2024-03-24T12:09:18.943582+0000","last_peered":"2024-03-24T12:09:18.943582+0000","last_clean":"2024-03-24T12:09:18.943582+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:09:18.943582+0000","last_undegraded":"2024-03-24T12:09:18.943582+0000","last_fullsized":"2024-03-24T12:09:18.943582+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:25.928874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.928874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":684,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.942811+0000","last_change":"2024-03-24T12:07:56.051237+0000","last_active":"2024-03-24T12:09:18.942811+0000","last_peered":"2024-03-24T12:09:18.942811+0000","last_clean":"2024-03-24T12:09:18.942811+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:09:18.942811+0000","last_undegraded":"2024-03-24T12:09:18.942811+0000","last_fullsized":"2024-03-24T12:09:18.942811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:07:56.051061+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.051061+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:23.143128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":567,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948041+0000","last_change":"2024-03-24T12:08:36.170759+0000","last_active":"2024-03-24T12:09:18.948041+0000","last_peered":"2024-03-24T12:09:18.948041+0000","last_clean":"2024-03-24T12:09:18.948041+0000","last_became_active":"2024-03-24T12:08:36.166947+0000","last_became_peered":"2024-03-24T12:08:36.166947+0000","last_unstale":"2024-03-24T12:09:18.948041+0000","last_undegraded":"2024-03-24T12:09:18.948041+0000","last_fullsized":"2024-03-24T12:09:18.948041+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:18.028433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:18.028433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":671,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948092+0000","last_change":"2024-03-24T12:06:42.538679+0000","last_active":"2024-03-24T12:09:18.948092+0000","last_peered":"2024-03-24T12:09:18.948092+0000","last_clean":"2024-03-24T12:09:18.948092+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:09:18.948092+0000","last_undegraded":"2024-03-24T12:09:18.948092+0000","last_fullsized":"2024-03-24T12:09:18.948092+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.538568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.538568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":669,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960447+0000","last_change":"2024-03-24T12:08:33.337568+0000","last_active":"2024-03-24T12:09:18.960447+0000","last_peered":"2024-03-24T12:09:18.960447+0000","last_clean":"2024-03-24T12:09:18.960447+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:09:18.960447+0000","last_undegraded":"2024-03-24T12:09:18.960447+0000","last_fullsized":"2024-03-24T12:09:18.960447+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:33.337437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:08:33.337437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:00.884474+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"83'130","reported_seq":726,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:20.291183+0000","last_change":"2024-03-24T12:09:20.291224+0000","last_active":"2024-03-24T12:09:20.291183+0000","last_peered":"2024-03-24T12:09:20.291183+0000","last_clean":"2024-03-24T12:09:20.291183+0000","last_became_active":"2024-03-24T12:03:19.819346+0000","last_became_peered":"2024-03-24T12:03:19.819346+0000","last_unstale":"2024-03-24T12:09:20.291183+0000","last_undegraded":"2024-03-24T12:09:20.291183+0000","last_fullsized":"2024-03-24T12:09:20.291183+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:09:20.291101+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:09:20.291101+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:35.522829+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":685,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944224+0000","last_change":"2024-03-24T12:06:13.055590+0000","last_active":"2024-03-24T12:09:18.944224+0000","last_peered":"2024-03-24T12:09:18.944224+0000","last_clean":"2024-03-24T12:09:18.944224+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:09:18.944224+0000","last_undegraded":"2024-03-24T12:09:18.944224+0000","last_fullsized":"2024-03-24T12:09:18.944224+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:06:13.055489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:13.055489+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":678,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949808+0000","last_change":"2024-03-24T12:08:36.157924+0000","last_active":"2024-03-24T12:09:18.949808+0000","last_peered":"2024-03-24T12:09:18.949808+0000","last_clean":"2024-03-24T12:09:18.949808+0000","last_became_active":"2024-03-24T12:08:36.157290+0000","last_became_peered":"2024-03-24T12:08:36.157290+0000","last_unstale":"2024-03-24T12:09:18.949808+0000","last_undegraded":"2024-03-24T12:09:18.949808+0000","last_fullsized":"2024-03-24T12:09:18.949808+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.372798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.372798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":673,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960356+0000","last_change":"2024-03-24T12:07:55.049337+0000","last_active":"2024-03-24T12:09:18.960356+0000","last_peered":"2024-03-24T12:09:18.960356+0000","last_clean":"2024-03-24T12:09:18.960356+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:09:18.960356+0000","last_undegraded":"2024-03-24T12:09:18.960356+0000","last_fullsized":"2024-03-24T12:09:18.960356+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:55.049210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:07:55.049210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":646,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944323+0000","last_change":"2024-03-24T12:06:18.013886+0000","last_active":"2024-03-24T12:09:18.944323+0000","last_peered":"2024-03-24T12:09:18.944323+0000","last_clean":"2024-03-24T12:09:18.944323+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:09:18.944323+0000","last_undegraded":"2024-03-24T12:09:18.944323+0000","last_fullsized":"2024-03-24T12:09:18.944323+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:18.013768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:18.013768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":705,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945581+0000","last_change":"2024-03-24T12:07:51.268007+0000","last_active":"2024-03-24T12:09:18.945581+0000","last_peered":"2024-03-24T12:09:18.945581+0000","last_clean":"2024-03-24T12:09:18.945581+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:09:18.945581+0000","last_undegraded":"2024-03-24T12:09:18.945581+0000","last_fullsized":"2024-03-24T12:09:18.945581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:51.267854+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.267854+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":519,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948041+0000","last_change":"2024-03-24T12:08:36.161709+0000","last_active":"2024-03-24T12:09:18.948041+0000","last_peered":"2024-03-24T12:09:18.948041+0000","last_clean":"2024-03-24T12:09:18.948041+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:09:18.948041+0000","last_undegraded":"2024-03-24T12:09:18.948041+0000","last_fullsized":"2024-03-24T12:09:18.948041+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.160982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.160982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":890,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.950127+0000","last_change":"2024-03-24T12:07:51.843434+0000","last_active":"2024-03-24T12:09:18.950127+0000","last_peered":"2024-03-24T12:09:18.950127+0000","last_clean":"2024-03-24T12:09:18.950127+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:09:18.950127+0000","last_undegraded":"2024-03-24T12:09:18.950127+0000","last_fullsized":"2024-03-24T12:09:18.950127+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:07:51.843305+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.843305+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:20.495680+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":687,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960795+0000","last_change":"2024-03-24T12:08:42.237732+0000","last_active":"2024-03-24T12:09:18.960795+0000","last_peered":"2024-03-24T12:09:18.960795+0000","last_clean":"2024-03-24T12:09:18.960795+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:09:18.960795+0000","last_undegraded":"2024-03-24T12:09:18.960795+0000","last_fullsized":"2024-03-24T12:09:18.960795+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:08:42.237615+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.237615+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:50.512255+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":716,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.950104+0000","last_change":"2024-03-24T12:08:36.157338+0000","last_active":"2024-03-24T12:09:18.950104+0000","last_peered":"2024-03-24T12:09:18.950104+0000","last_clean":"2024-03-24T12:09:18.950104+0000","last_became_active":"2024-03-24T12:08:36.156563+0000","last_became_peered":"2024-03-24T12:08:36.156563+0000","last_unstale":"2024-03-24T12:09:18.950104+0000","last_undegraded":"2024-03-24T12:09:18.950104+0000","last_fullsized":"2024-03-24T12:09:18.950104+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:05:25.602806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.602806+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":659,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943858+0000","last_change":"2024-03-24T12:06:40.021255+0000","last_active":"2024-03-24T12:09:18.943858+0000","last_peered":"2024-03-24T12:09:18.943858+0000","last_clean":"2024-03-24T12:09:18.943858+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:09:18.943858+0000","last_undegraded":"2024-03-24T12:09:18.943858+0000","last_fullsized":"2024-03-24T12:09:18.943858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:06:40.021114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:40.021114+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":672,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945569+0000","last_change":"2024-03-24T12:07:36.333159+0000","last_active":"2024-03-24T12:09:18.945569+0000","last_peered":"2024-03-24T12:09:18.945569+0000","last_clean":"2024-03-24T12:09:18.945569+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:09:18.945569+0000","last_undegraded":"2024-03-24T12:09:18.945569+0000","last_fullsized":"2024-03-24T12:09:18.945569+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:36.333044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:07:36.333044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":267,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949766+0000","last_change":"2024-03-24T12:08:36.167702+0000","last_active":"2024-03-24T12:09:18.949766+0000","last_peered":"2024-03-24T12:09:18.949766+0000","last_clean":"2024-03-24T12:09:18.949766+0000","last_became_active":"2024-03-24T12:08:36.166973+0000","last_became_peered":"2024-03-24T12:08:36.166973+0000","last_unstale":"2024-03-24T12:09:18.949766+0000","last_undegraded":"2024-03-24T12:09:18.949766+0000","last_fullsized":"2024-03-24T12:09:18.949766+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:04:14.188600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":616,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943838+0000","last_change":"2024-03-24T12:07:42.113052+0000","last_active":"2024-03-24T12:09:18.943838+0000","last_peered":"2024-03-24T12:09:18.943838+0000","last_clean":"2024-03-24T12:09:18.943838+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:09:18.943838+0000","last_undegraded":"2024-03-24T12:09:18.943838+0000","last_fullsized":"2024-03-24T12:09:18.943838+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:07:42.112569+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:07:42.112569+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":678,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948993+0000","last_change":"2024-03-24T12:06:57.653082+0000","last_active":"2024-03-24T12:09:18.948993+0000","last_peered":"2024-03-24T12:09:18.948993+0000","last_clean":"2024-03-24T12:09:18.948993+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:09:18.948993+0000","last_undegraded":"2024-03-24T12:09:18.948993+0000","last_fullsized":"2024-03-24T12:09:18.948993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:57.653004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:06:57.653004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":425,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.953587+0000","last_change":"2024-03-24T12:07:53.354298+0000","last_active":"2024-03-24T12:09:18.953587+0000","last_peered":"2024-03-24T12:09:18.953587+0000","last_clean":"2024-03-24T12:09:18.953587+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:09:18.953587+0000","last_undegraded":"2024-03-24T12:09:18.953587+0000","last_fullsized":"2024-03-24T12:09:18.953587+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:07:53.354213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":665,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952634+0000","last_change":"2024-03-24T12:08:36.167032+0000","last_active":"2024-03-24T12:09:18.952634+0000","last_peered":"2024-03-24T12:09:18.952634+0000","last_clean":"2024-03-24T12:09:18.952634+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:09:18.952634+0000","last_undegraded":"2024-03-24T12:09:18.952634+0000","last_fullsized":"2024-03-24T12:09:18.952634+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:53.809021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:53.809021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":716,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943390+0000","last_change":"2024-03-24T12:07:41.142569+0000","last_active":"2024-03-24T12:09:18.943390+0000","last_peered":"2024-03-24T12:09:18.943390+0000","last_clean":"2024-03-24T12:09:18.943390+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:09:18.943390+0000","last_undegraded":"2024-03-24T12:09:18.943390+0000","last_fullsized":"2024-03-24T12:09:18.943390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:07:41.142421+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.142421+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":671,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944033+0000","last_change":"2024-03-24T12:07:32.002134+0000","last_active":"2024-03-24T12:09:18.944033+0000","last_peered":"2024-03-24T12:09:18.944033+0000","last_clean":"2024-03-24T12:09:18.944033+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:09:18.944033+0000","last_undegraded":"2024-03-24T12:09:18.944033+0000","last_fullsized":"2024-03-24T12:09:18.944033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:07:32.001718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:32.001718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":661,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952309+0000","last_change":"2024-03-24T12:06:42.722854+0000","last_active":"2024-03-24T12:09:18.952309+0000","last_peered":"2024-03-24T12:09:18.952309+0000","last_clean":"2024-03-24T12:09:18.952309+0000","last_became_active":"2024-03-24T12:05:44.736383+0000","last_became_peered":"2024-03-24T12:05:44.736383+0000","last_unstale":"2024-03-24T12:09:18.952309+0000","last_undegraded":"2024-03-24T12:09:18.952309+0000","last_fullsized":"2024-03-24T12:09:18.952309+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.722733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.722733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":664,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944842+0000","last_change":"2024-03-24T12:08:39.416538+0000","last_active":"2024-03-24T12:09:18.944842+0000","last_peered":"2024-03-24T12:09:18.944842+0000","last_clean":"2024-03-24T12:09:18.944842+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:09:18.944842+0000","last_undegraded":"2024-03-24T12:09:18.944842+0000","last_fullsized":"2024-03-24T12:09:18.944842+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:39.416426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:39.416426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:54.813785+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":692,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.951972+0000","last_change":"2024-03-24T12:07:40.191735+0000","last_active":"2024-03-24T12:09:18.951972+0000","last_peered":"2024-03-24T12:09:18.951972+0000","last_clean":"2024-03-24T12:09:18.951972+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:09:18.951972+0000","last_undegraded":"2024-03-24T12:09:18.951972+0000","last_fullsized":"2024-03-24T12:09:18.951972+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:40.191513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.191513+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":692,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960576+0000","last_change":"2024-03-24T12:08:40.240985+0000","last_active":"2024-03-24T12:09:18.960576+0000","last_peered":"2024-03-24T12:09:18.960576+0000","last_clean":"2024-03-24T12:09:18.960576+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:09:18.960576+0000","last_undegraded":"2024-03-24T12:09:18.960576+0000","last_fullsized":"2024-03-24T12:09:18.960576+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.240855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.240855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:41.963342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":580,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.948593+0000","last_change":"2024-03-24T12:08:36.170723+0000","last_active":"2024-03-24T12:09:18.948593+0000","last_peered":"2024-03-24T12:09:18.948593+0000","last_clean":"2024-03-24T12:09:18.948593+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:09:18.948593+0000","last_undegraded":"2024-03-24T12:09:18.948593+0000","last_fullsized":"2024-03-24T12:09:18.948593+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.099551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.099551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":653,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944335+0000","last_change":"2024-03-24T12:07:46.478165+0000","last_active":"2024-03-24T12:09:18.944335+0000","last_peered":"2024-03-24T12:09:18.944335+0000","last_clean":"2024-03-24T12:09:18.944335+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:09:18.944335+0000","last_undegraded":"2024-03-24T12:09:18.944335+0000","last_fullsized":"2024-03-24T12:09:18.944335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:46.478049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:07:46.478049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":682,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.952248+0000","last_change":"2024-03-24T12:06:56.609759+0000","last_active":"2024-03-24T12:09:18.952248+0000","last_peered":"2024-03-24T12:09:18.952248+0000","last_clean":"2024-03-24T12:09:18.952248+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:09:18.952248+0000","last_undegraded":"2024-03-24T12:09:18.952248+0000","last_fullsized":"2024-03-24T12:09:18.952248+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:06:56.609621+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:06:56.609621+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":696,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960734+0000","last_change":"2024-03-24T12:09:16.215114+0000","last_active":"2024-03-24T12:09:18.960734+0000","last_peered":"2024-03-24T12:09:18.960734+0000","last_clean":"2024-03-24T12:09:18.960734+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:09:18.960734+0000","last_undegraded":"2024-03-24T12:09:18.960734+0000","last_fullsized":"2024-03-24T12:09:18.960734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:16.214991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:09:16.214991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:32.017588+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":680,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.960169+0000","last_change":"2024-03-24T12:07:40.020936+0000","last_active":"2024-03-24T12:09:18.960169+0000","last_peered":"2024-03-24T12:09:18.960169+0000","last_clean":"2024-03-24T12:09:18.960169+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:09:18.960169+0000","last_undegraded":"2024-03-24T12:09:18.960169+0000","last_fullsized":"2024-03-24T12:09:18.960169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:40.020836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:17.067728+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.020836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:19.191301+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":586,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.949575+0000","last_change":"2024-03-24T12:08:36.164485+0000","last_active":"2024-03-24T12:09:18.949575+0000","last_peered":"2024-03-24T12:09:18.949575+0000","last_clean":"2024-03-24T12:09:18.949575+0000","last_became_active":"2024-03-24T12:08:36.164166+0000","last_became_peered":"2024-03-24T12:08:36.164166+0000","last_unstale":"2024-03-24T12:09:18.949575+0000","last_undegraded":"2024-03-24T12:09:18.949575+0000","last_fullsized":"2024-03-24T12:09:18.949575+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.117875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.117875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":697,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.951565+0000","last_change":"2024-03-24T12:07:51.280566+0000","last_active":"2024-03-24T12:09:18.951565+0000","last_peered":"2024-03-24T12:09:18.951565+0000","last_clean":"2024-03-24T12:09:18.951565+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:09:18.951565+0000","last_undegraded":"2024-03-24T12:09:18.951565+0000","last_fullsized":"2024-03-24T12:09:18.951565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:07:51.280424+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.280424+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":676,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.945295+0000","last_change":"2024-03-24T12:08:40.183479+0000","last_active":"2024-03-24T12:09:18.945295+0000","last_peered":"2024-03-24T12:09:18.945295+0000","last_clean":"2024-03-24T12:09:18.945295+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:09:18.945295+0000","last_undegraded":"2024-03-24T12:09:18.945295+0000","last_fullsized":"2024-03-24T12:09:18.945295+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.183311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.183311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:55.086388+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":145,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.947849+0000","last_change":"2024-03-24T12:08:36.153334+0000","last_active":"2024-03-24T12:09:18.947849+0000","last_peered":"2024-03-24T12:09:18.947849+0000","last_clean":"2024-03-24T12:09:18.947849+0000","last_became_active":"2024-03-24T12:08:36.152717+0000","last_became_peered":"2024-03-24T12:08:36.152717+0000","last_unstale":"2024-03-24T12:09:18.947849+0000","last_undegraded":"2024-03-24T12:09:18.947849+0000","last_fullsized":"2024-03-24T12:09:18.947849+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.683700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.683700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":660,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.951955+0000","last_change":"2024-03-24T12:08:36.165633+0000","last_active":"2024-03-24T12:09:18.951955+0000","last_peered":"2024-03-24T12:09:18.951955+0000","last_clean":"2024-03-24T12:09:18.951955+0000","last_became_active":"2024-03-24T12:08:36.165334+0000","last_became_peered":"2024-03-24T12:08:36.165334+0000","last_unstale":"2024-03-24T12:09:18.951955+0000","last_undegraded":"2024-03-24T12:09:18.951955+0000","last_fullsized":"2024-03-24T12:09:18.951955+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.241824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.241824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":743,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944464+0000","last_change":"2024-03-24T12:08:47.101220+0000","last_active":"2024-03-24T12:09:18.944464+0000","last_peered":"2024-03-24T12:09:18.944464+0000","last_clean":"2024-03-24T12:09:18.944464+0000","last_became_active":"2024-03-24T12:08:36.154117+0000","last_became_peered":"2024-03-24T12:08:36.154117+0000","last_unstale":"2024-03-24T12:09:18.944464+0000","last_undegraded":"2024-03-24T12:09:18.944464+0000","last_fullsized":"2024-03-24T12:09:18.944464+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:47.101082+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:08:47.101082+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:07.268703+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":666,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.944912+0000","last_change":"2024-03-24T12:07:30.248462+0000","last_active":"2024-03-24T12:09:18.944912+0000","last_peered":"2024-03-24T12:09:18.944912+0000","last_clean":"2024-03-24T12:09:18.944912+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:09:18.944912+0000","last_undegraded":"2024-03-24T12:09:18.944912+0000","last_fullsized":"2024-03-24T12:09:18.944912+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.248337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.248337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":136,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.942720+0000","last_change":"2024-03-24T12:08:33.256620+0000","last_active":"2024-03-24T12:09:18.942720+0000","last_peered":"2024-03-24T12:09:18.942720+0000","last_clean":"2024-03-24T12:09:18.942720+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:09:18.942720+0000","last_undegraded":"2024-03-24T12:09:18.942720+0000","last_fullsized":"2024-03-24T12:09:18.942720+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:33.255322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:08:33.255322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:51.307702+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":653,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T12:09:18.943621+0000","last_change":"2024-03-24T12:07:48.265595+0000","last_active":"2024-03-24T12:09:18.943621+0000","last_peered":"2024-03-24T12:09:18.943621+0000","last_clean":"2024-03-24T12:09:18.943621+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:09:18.943621+0000","last_undegraded":"2024-03-24T12:09:18.943621+0000","last_fullsized":"2024-03-24T12:09:18.943621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:48.265295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:48.265295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4137222168,"num_objects":63130,"num_object_clones":0,"num_object_copies":189390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":225700,"num_write_kb":9242497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19440,"num_bytes_recovered":1273954304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11771392000,"data_stored":11771379784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76282,"ondisk_log_size":76282,"up":48,"acting":45,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":488689,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":80,"num_write":49,"num_write_kb":658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":483028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":223,"num_read_kb":218,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005265,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1679416,"kb_used_data":1022128,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":92065736,"statfs":{"total":95995035648,"available":94275313664,"internally_reserved":0,"allocated":1046659072,"data_stored":1046041588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":673053130},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332312,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2415180,"kb_used_data":1802308,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":91329972,"statfs":{"total":95995035648,"available":93521891328,"internally_reserved":0,"allocated":1845563392,"data_stored":1844925738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":627571142},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1459140,"kb_used_data":769912,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92286012,"statfs":{"total":95995035648,"available":94500876288,"internally_reserved":0,"allocated":788389888,"data_stored":787763609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738614,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3290500,"kb_used_data":2501308,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":90454652,"statfs":{"total":95995035648,"available":92625563648,"internally_reserved":0,"allocated":2561339392,"data_stored":2560709010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411564,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1331868,"kb_used_data":777300,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92413284,"statfs":{"total":95995035648,"available":94631202816,"internally_reserved":0,"allocated":795955200,"data_stored":795365659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117222,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2024660,"kb_used_data":1289612,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":91720492,"statfs":{"total":95995035648,"available":93921783808,"internally_reserved":0,"allocated":1320562688,"data_stored":1319917323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":752679367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920993,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3163208,"kb_used_data":2522432,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":90581944,"statfs":{"total":95995035648,"available":92755910656,"internally_reserved":0,"allocated":2582970368,"data_stored":2582335956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,0,0,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455498,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1119236,"kb_used_data":906292,"kb_used_omap":7,"kb_used_meta":212920,"kb_avail":92625916,"statfs":{"total":95995035648,"available":94848937984,"internally_reserved":0,"allocated":928043008,"data_stored":927414469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":218030192},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,11],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2525630464,"data_stored":2525626392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":777191424,"data_stored":777191424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":783548416,"data_stored":783548416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1310982144,"data_stored":1310982144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2579501056,"data_stored":2579496984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":944439296,"data_stored":944439296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1813385216,"data_stored":1813381144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1036713984,"data_stored":1036713984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:09:30.511 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [7, 0, 3, 6, 1, 4] 2024-03-24T12:09:30.511 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.0', '7', '0', '3', '6', '1', '4'] 2024-03-24T12:09:30.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 1.0 7 0 3 6 1 4 2024-03-24T12:09:30.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:30.512 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:30 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:09:30.512 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:30 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:09:30.512 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:30 smithi193 ceph-mon[33593]: pgmap v788: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 5814/173316 objects degraded (3.355%); 33 MiB/s, 527 objects/s recovering 2024-03-24T12:09:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:30.553 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:30.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:30.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:30.688 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:30.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:30.790 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:30 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:09:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:30 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:09:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:30 smithi064 ceph-mon[30223]: pgmap v788: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 5814/173316 objects degraded (3.355%); 33 MiB/s, 527 objects/s recovering 2024-03-24T12:09:30.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:30 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:09:30.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:30 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:09:30.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:30 smithi064 ceph-mon[42243]: pgmap v788: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 5814/173316 objects degraded (3.355%); 33 MiB/s, 527 objects/s recovering 2024-03-24T12:09:30.828 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:30.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:30.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:30.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:30.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:31.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:31.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:31.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:31.235 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:31.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:31.281 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:09:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:09:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T12:09:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T12:09:31.282 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:09:31.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:31.371 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:31.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:31.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:31.508 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:31.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:31 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T12:09:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:31 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T12:09:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:31 smithi064 ceph-mon[30223]: 6.10 deep-scrub starts 2024-03-24T12:09:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:31 smithi064 ceph-mon[30223]: 6.10 deep-scrub ok 2024-03-24T12:09:31.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:31 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T12:09:31.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:31 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T12:09:31.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:31 smithi064 ceph-mon[42243]: 6.10 deep-scrub starts 2024-03-24T12:09:31.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:31 smithi064 ceph-mon[42243]: 6.10 deep-scrub ok 2024-03-24T12:09:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:31 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T12:09:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:31 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T12:09:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:31 smithi193 ceph-mon[33593]: 6.10 deep-scrub starts 2024-03-24T12:09:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:31 smithi193 ceph-mon[33593]: 6.10 deep-scrub ok 2024-03-24T12:09:31.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:31.643 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:31.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:31.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:31.775 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:31.803 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:31.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:31.904 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:31.949 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:31.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:32.051 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:32.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:32.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:32.120 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:09:32.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:32.233 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:32.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:32.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:32.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:32 smithi193 ceph-mon[33593]: pgmap v789: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 736 op/s; 4961/172326 objects degraded (2.879%); 22 MiB/s, 345 objects/s recovering 2024-03-24T12:09:32.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:32 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 5814/173316 objects degraded (3.355%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:32.371 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:32.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:32.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:32.529 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:32.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:32.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:32 smithi064 ceph-mon[30223]: pgmap v789: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 736 op/s; 4961/172326 objects degraded (2.879%); 22 MiB/s, 345 objects/s recovering 2024-03-24T12:09:32.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:32 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 5814/173316 objects degraded (3.355%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:32.539 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:32 smithi064 ceph-mon[42243]: pgmap v789: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 736 op/s; 4961/172326 objects degraded (2.879%); 22 MiB/s, 345 objects/s recovering 2024-03-24T12:09:32.539 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:32 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 5814/173316 objects degraded (3.355%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:32.577 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:32.631 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:32.684 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:32.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:32.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:32.826 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:32.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:32.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:32.952 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:32.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:33.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:33.078 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:33.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:33.180 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:33.217 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:33.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:33.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:09:33.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:33.354 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:33.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:33.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:33.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:33.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:33.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:33 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1126619782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:33.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:33 smithi064 ceph-mon[30223]: 10.4 scrub starts 2024-03-24T12:09:33.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:33 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1126619782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:33.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:33 smithi064 ceph-mon[42243]: 10.4 scrub starts 2024-03-24T12:09:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:33 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1126619782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:33 smithi193 ceph-mon[33593]: 10.4 scrub starts 2024-03-24T12:09:33.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:33.625 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:33.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:33.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:33.762 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:09:33.762 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:09:33.762 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:09:33.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:09:33.764 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:09:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:33.765 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:33.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:33.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:33.900 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:33.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:34.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:34.037 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:34.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:34.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:34.178 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:34.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:34.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:34.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:34.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:34.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:34.453 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:34.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:34.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:34 smithi064 ceph-mon[30223]: pgmap v790: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s; 3213/160041 objects degraded (2.008%); 27 MiB/s, 432 objects/s recovering 2024-03-24T12:09:34.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:34 smithi064 ceph-mon[30223]: 10.4 scrub ok 2024-03-24T12:09:34.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:34 smithi064 ceph-mon[42243]: pgmap v790: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s; 3213/160041 objects degraded (2.008%); 27 MiB/s, 432 objects/s recovering 2024-03-24T12:09:34.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:34 smithi064 ceph-mon[42243]: 10.4 scrub ok 2024-03-24T12:09:34.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:34 smithi193 ceph-mon[33593]: pgmap v790: 168 pgs: 3 active+undersized+degraded+remapped+backfilling, 165 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s; 3213/160041 objects degraded (2.008%); 27 MiB/s, 432 objects/s recovering 2024-03-24T12:09:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:34 smithi193 ceph-mon[33593]: 10.4 scrub ok 2024-03-24T12:09:34.599 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:34.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:34.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:34.734 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:34.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:34.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:34.873 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:34.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:34.974 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:35.013 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:35.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:35.115 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:35.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:35.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:35.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:35.280 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:35.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:35.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:35.417 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:35.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:35.519 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:35.556 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:35.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:35.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:35.710 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:35.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:35.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:35.724 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:35.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:35.862 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:35.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:35.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:35.997 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:35.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:35.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:36.099 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:36.136 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:36.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:36.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:36.287 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:36.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:36.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:36.429 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:36.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:36.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:36.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:36.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:36.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:36.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:09:36.665 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:36.690 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:36.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:36.779 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:36 smithi064 ceph-mon[30223]: pgmap v791: 168 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1000 B/s rd, 0 B/s wr, 1.05k op/s; 3213/156789 objects degraded (2.049%); 26 MiB/s, 412 objects/s recovering 2024-03-24T12:09:36.780 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:36 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T12:09:36.780 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:36 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T12:09:36.780 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:36 smithi064 ceph-mon[42243]: pgmap v791: 168 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1000 B/s rd, 0 B/s wr, 1.05k op/s; 3213/156789 objects degraded (2.049%); 26 MiB/s, 412 objects/s recovering 2024-03-24T12:09:36.780 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:36 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T12:09:36.780 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:36 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T12:09:36.792 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:36.853 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:36.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:36.854 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:36 smithi193 ceph-mon[33593]: pgmap v791: 168 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1000 B/s rd, 0 B/s wr, 1.05k op/s; 3213/156789 objects degraded (2.049%); 26 MiB/s, 412 objects/s recovering 2024-03-24T12:09:36.854 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:36 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T12:09:36.854 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:36 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T12:09:36.951 INFO:teuthology.orchestra.run.smithi064.stderr:set 1.0 pg_upmap_items mapping to [7->0,3->6,1->4] 2024-03-24T12:09:36.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:36.994 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:36.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:37.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:37.125 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:37.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:37.227 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:37.256 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:37.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:37.358 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:37.389 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:37.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:37.427 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:09:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:37.491 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:37.516 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:37.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:37.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:37.641 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:37.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:37.677 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:09:37.677 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:09:37.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:37.766 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:37.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:09:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:09:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T12:09:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T12:09:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4007523413' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [7, 0, 3, 6, 1, 4]}]: dispatch 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 3213/156789 objects degraded (2.049%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4007523413' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [7, 0, 3, 6, 1, 4]}]': finished 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[30223]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4007523413' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [7, 0, 3, 6, 1, 4]}]: dispatch 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 3213/156789 objects degraded (2.049%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4007523413' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [7, 0, 3, 6, 1, 4]}]': finished 2024-03-24T12:09:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:37 smithi064 ceph-mon[42243]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T12:09:37.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:09:37.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:09:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:09:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T12:09:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T12:09:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4007523413' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [7, 0, 3, 6, 1, 4]}]: dispatch 2024-03-24T12:09:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 3213/156789 objects degraded (2.049%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:09:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4007523413' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [7, 0, 3, 6, 1, 4]}]': finished 2024-03-24T12:09:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:37 smithi193 ceph-mon[33593]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T12:09:37.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:37.891 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:37.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:37.993 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:38.026 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:38.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:38.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:38.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:38.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:38.277 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:38.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:38.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:09:38.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:38.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:38.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:38.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:38.529 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:38.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:38.537 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:38.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:38.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:38.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:38.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:38.778 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:38.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:38.880 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:38.904 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:38.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:38.933 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:39.006 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:39.044 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:39.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:39.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:39.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:39.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:39.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:39.300 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: pgmap v793: 168 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 827/143553 objects degraded (0.576%); 27 MiB/s, 432 objects/s recovering 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: 5.1a deep-scrub starts 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: 5.1a deep-scrub ok 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: 5.12 scrub starts 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[30223]: 5.12 scrub ok 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: pgmap v793: 168 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 827/143553 objects degraded (0.576%); 27 MiB/s, 432 objects/s recovering 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T12:09:39.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T12:09:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: 5.1a deep-scrub starts 2024-03-24T12:09:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: 5.1a deep-scrub ok 2024-03-24T12:09:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: 5.12 scrub starts 2024-03-24T12:09:39.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:38 smithi064 ceph-mon[42243]: 5.12 scrub ok 2024-03-24T12:09:39.317 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:39.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:39.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: pgmap v793: 168 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 827/143553 objects degraded (0.576%); 27 MiB/s, 432 objects/s recovering 2024-03-24T12:09:39.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T12:09:39.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T12:09:39.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:39.331 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T12:09:39.331 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: 5.1a deep-scrub starts 2024-03-24T12:09:39.331 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: 5.1a deep-scrub ok 2024-03-24T12:09:39.331 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: 5.12 scrub starts 2024-03-24T12:09:39.331 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:38 smithi193 ceph-mon[33593]: 5.12 scrub ok 2024-03-24T12:09:39.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:39.460 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:39.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:39.561 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:39.588 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:39.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:39.690 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:39.721 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:39.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:39.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:39.862 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:39.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:39.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:39.989 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:09:39.998 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:39.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:40.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:40.138 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:40.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:40.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:39 smithi193 ceph-mon[33593]: 2.17 deep-scrub starts 2024-03-24T12:09:40.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:39 smithi193 ceph-mon[33593]: 2.17 deep-scrub ok 2024-03-24T12:09:40.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:39 smithi193 ceph-mon[33593]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T12:09:40.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:39 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:09:40.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:39 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:09:40.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:39 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/464180056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:09:40.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:39 smithi193 ceph-mon[33593]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:09:40.240 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:40.275 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:40.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[30223]: 2.17 deep-scrub starts 2024-03-24T12:09:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[30223]: 2.17 deep-scrub ok 2024-03-24T12:09:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[30223]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T12:09:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:09:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:09:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/464180056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:09:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[30223]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:09:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[42243]: 2.17 deep-scrub starts 2024-03-24T12:09:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[42243]: 2.17 deep-scrub ok 2024-03-24T12:09:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[42243]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T12:09:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:09:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:09:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/464180056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:09:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:39 smithi064 ceph-mon[42243]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:09:40.377 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:40.413 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:40.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:40.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:40.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:40.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:40.695 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:40.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:40.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:40.829 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:40.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:40.930 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:40.969 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:40.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:41.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:41.104 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:41.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:41.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:41.214 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:40 smithi193 ceph-mon[33593]: pgmap v796: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 167 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 281/140865 objects degraded (0.199%); 28 MiB/s, 449 objects/s recovering 2024-03-24T12:09:41.214 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:40 smithi193 ceph-mon[33593]: 1.f deep-scrub starts 2024-03-24T12:09:41.214 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:40 smithi193 ceph-mon[33593]: 1.f deep-scrub ok 2024-03-24T12:09:41.215 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:40 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:41.215 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:40 smithi193 ceph-mon[33593]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:09:41.215 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:40 smithi193 ceph-mon[33593]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T12:09:41.215 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:40 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:09:41.215 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:40 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:09:41.235 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:41.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:41.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[30223]: pgmap v796: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 167 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 281/140865 objects degraded (0.199%); 28 MiB/s, 449 objects/s recovering 2024-03-24T12:09:41.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[30223]: 1.f deep-scrub starts 2024-03-24T12:09:41.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[30223]: 1.f deep-scrub ok 2024-03-24T12:09:41.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[30223]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[30223]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[42243]: pgmap v796: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 167 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 281/140865 objects degraded (0.199%); 28 MiB/s, 449 objects/s recovering 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[42243]: 1.f deep-scrub starts 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[42243]: 1.f deep-scrub ok 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[42243]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[42243]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T12:09:41.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:09:41.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:40 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:09:41.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:41.373 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:41.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:41.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:41.516 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:41.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:41.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:41.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:41.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:41.754 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:41.793 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:41.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:41.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:41.915 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:09:41.915 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:09:41.937 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:41.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:42.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:42.075 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:42.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:42.103 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:09:42.149 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:09:42.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:42.184 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:09:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:42.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:42.221 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:09:42.256 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:09:42.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:41 smithi064 ceph-mon[30223]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T12:09:42.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:41 smithi064 ceph-mon[30223]: Health check update: Degraded data redundancy: 281/140865 objects degraded (0.199%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-24T12:09:42.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:41 smithi064 ceph-mon[30223]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T12:09:42.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:41 smithi064 ceph-mon[42243]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T12:09:42.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:41 smithi064 ceph-mon[42243]: Health check update: Degraded data redundancy: 281/140865 objects degraded (0.199%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-24T12:09:42.256 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:41 smithi064 ceph-mon[42243]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T12:09:42.275 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:41 smithi193 ceph-mon[33593]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T12:09:42.275 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:41 smithi193 ceph-mon[33593]: Health check update: Degraded data redundancy: 281/140865 objects degraded (0.199%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-24T12:09:42.275 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:41 smithi193 ceph-mon[33593]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T12:09:42.295 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:09:42.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:42.337 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:09:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:42.355 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:42.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:42.378 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:09:42.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:09:42.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:09:42.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:09:42.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:09:42.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:09:42.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:09:42.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:09:42.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:09:42.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:42.492 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:42.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:42.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:42.629 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:42.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:42.648 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T12:09:42.731 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:42.785 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:42.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:42.883 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:09:42.883 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:09:42.883 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-24T12:09:42.884 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-24T12:09:42.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:09:42.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:42.964 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:42.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:42.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:42 smithi193 ceph-mon[33593]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 167 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 223 op/s; 281/140865 objects degraded (0.199%); 8.4 MiB/s, 134 objects/s recovering 2024-03-24T12:09:42.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2373258328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:42.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:42.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:43.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:42 smithi064 ceph-mon[30223]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 167 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 223 op/s; 281/140865 objects degraded (0.199%); 8.4 MiB/s, 134 objects/s recovering 2024-03-24T12:09:43.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2373258328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:43.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:43.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:43.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:42 smithi064 ceph-mon[42243]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 167 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 223 op/s; 281/140865 objects degraded (0.199%); 8.4 MiB/s, 134 objects/s recovering 2024-03-24T12:09:43.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2373258328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:43.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:43.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:43.053 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:43.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:43.102 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:43.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:43.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:43.245 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:43.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:43.346 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:43.396 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:43.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:43.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:09:43.498 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:43.536 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:43.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:43.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:43.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:43.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:43.842 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:43.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:43.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:43.981 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:43.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:44.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:44.118 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:44.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:44.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:44.260 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:44.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:44.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:44 smithi064 ceph-mon[30223]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 281/140865 objects degraded (0.199%), 1 pg degraded, 1 pg undersized) 2024-03-24T12:09:44.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:44 smithi064 ceph-mon[42243]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 281/140865 objects degraded (0.199%), 1 pg degraded, 1 pg undersized) 2024-03-24T12:09:44.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:44 smithi193 ceph-mon[33593]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 281/140865 objects degraded (0.199%), 1 pg degraded, 1 pg undersized) 2024-03-24T12:09:44.361 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:44.397 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:44.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:44.498 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:44.544 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:44.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:44.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:44.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:44.733 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:44.782 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:44.822 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:44.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:44.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:44.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:44.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:45.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:45.089 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:45.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:45.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:45.099 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:09:45.099 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:09:45.100 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T12:09:45.100 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T12:09:45.100 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:09:45.180 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:45 smithi064 ceph-mon[30223]: pgmap v801: 168 pgs: 168 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 179 objects/s recovering 2024-03-24T12:09:45.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:45 smithi064 ceph-mon[42243]: pgmap v801: 168 pgs: 168 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 179 objects/s recovering 2024-03-24T12:09:45.192 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:45.226 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:45.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:45.328 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:45 smithi193 ceph-mon[33593]: pgmap v801: 168 pgs: 168 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 722 op/s; 11 MiB/s, 179 objects/s recovering 2024-03-24T12:09:45.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:45.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:45.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:45.502 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:45.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:45.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:09:45.608 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:45.640 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:45.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:45.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:45.791 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:45.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:45.893 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:45.954 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:45.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:46.055 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:46.093 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:46.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:46.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:46.226 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:46.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:46.328 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:46.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T12:09:46.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:46 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:09:46.330 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:46 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:09:46.361 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:46 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:09:46.361 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:46 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:09:46.361 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:46 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:09:46.362 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:46 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:09:46.402 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:46.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:46.504 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:46.555 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:46.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:46.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:46.695 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:46.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:46.763 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:09:46.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:46.837 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:46.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:46.924 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:09:46.927 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:09:46.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:46.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:47.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:47.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:47.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:47.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:47.244 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:47 smithi193 ceph-mon[33593]: pgmap v802: 168 pgs: 168 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-24T12:09:47.244 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:47 smithi193 ceph-mon[33593]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T12:09:47.244 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:47 smithi193 ceph-mon[33593]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T12:09:47.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:47.282 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:47.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:47 smithi064 ceph-mon[30223]: pgmap v802: 168 pgs: 168 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-24T12:09:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:47 smithi064 ceph-mon[30223]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T12:09:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:47 smithi064 ceph-mon[30223]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T12:09:47.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:47 smithi064 ceph-mon[42243]: pgmap v802: 168 pgs: 168 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-24T12:09:47.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:47 smithi064 ceph-mon[42243]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T12:09:47.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:47 smithi064 ceph-mon[42243]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T12:09:47.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:47.422 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:47.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:47.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:47.558 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:09:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:09:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:47.564 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:47.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:47.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:47.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:47.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:47.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:47.841 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:47.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:47.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:47.980 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:47.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:48.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:09:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:09:48.073 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2024-03-24T12:09:46.053346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2753006663,"num_objects":42199,"num_object_clones":0,"num_object_copies":126597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42199,"num_whiteouts":0,"num_read":1437,"num_read_kb":1215,"num_write":247600,"num_write_kb":9243200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26690,"num_bytes_recovered":1746101852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80225,"ondisk_log_size":80225,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13174036,"kb_used_data":8236392,"kb_used_omap":18,"kb_used_meta":4937517,"kb_avail":736787180,"statfs":{"total":767960285184,"available":754470072320,"internally_reserved":0,"allocated":8434065408,"data_stored":8428763094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":5056017627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-311689216,"num_objects":-4756,"num_object_clones":0,"num_object_copies":-14268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4756,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4756,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":18284544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.079498"},"pg_stats":[{"pgid":"10.6","version":"375'15281","reported_seq":26172,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:44.262701+0000","last_change":"2024-03-24T12:09:18.996729+0000","last_active":"2024-03-24T12:09:44.262701+0000","last_peered":"2024-03-24T12:09:44.262701+0000","last_clean":"2024-03-24T12:09:44.262701+0000","last_became_active":"2024-03-24T12:09:18.952687+0000","last_became_peered":"2024-03-24T12:09:18.952687+0000","last_unstale":"2024-03-24T12:09:44.262701+0000","last_undegraded":"2024-03-24T12:09:44.262701+0000","last_fullsized":"2024-03-24T12:09:44.262701+0000","mapping_epoch":366,"log_start":"355'10926","ondisk_log_start":"355'10926","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.744161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162856960,"num_objects":2485,"num_object_clones":0,"num_object_copies":7455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15281,"num_write_kb":568512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5381,"num_bytes_recovered":352649216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'130","reported_seq":558,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989325+0000","last_change":"2024-03-24T12:07:37.002106+0000","last_active":"2024-03-24T12:09:40.989325+0000","last_peered":"2024-03-24T12:09:40.989325+0000","last_clean":"2024-03-24T12:09:40.989325+0000","last_became_active":"2024-03-24T12:03:19.822426+0000","last_became_peered":"2024-03-24T12:03:19.822426+0000","last_unstale":"2024-03-24T12:09:40.989325+0000","last_undegraded":"2024-03-24T12:09:40.989325+0000","last_fullsized":"2024-03-24T12:09:40.989325+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:37.001978+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:37.001978+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":914,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989069+0000","last_change":"2024-03-24T12:08:46.189879+0000","last_active":"2024-03-24T12:09:40.989069+0000","last_peered":"2024-03-24T12:09:40.989069+0000","last_clean":"2024-03-24T12:09:40.989069+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:09:40.989069+0000","last_undegraded":"2024-03-24T12:09:40.989069+0000","last_fullsized":"2024-03-24T12:09:40.989069+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:08:46.189592+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:57.625770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":613,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.952104+0000","last_change":"2024-03-24T12:08:36.154379+0000","last_active":"2024-03-24T12:09:41.952104+0000","last_peered":"2024-03-24T12:09:41.952104+0000","last_clean":"2024-03-24T12:09:41.952104+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:09:41.952104+0000","last_undegraded":"2024-03-24T12:09:41.952104+0000","last_fullsized":"2024-03-24T12:09:41.952104+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:05:17.253630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.253630+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":168,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981457+0000","last_change":"2024-03-24T12:08:32.282623+0000","last_active":"2024-03-24T12:09:40.981457+0000","last_peered":"2024-03-24T12:09:40.981457+0000","last_clean":"2024-03-24T12:09:40.981457+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:09:40.981457+0000","last_undegraded":"2024-03-24T12:09:40.981457+0000","last_fullsized":"2024-03-24T12:09:40.981457+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:32.282513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:32.282513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:43.785500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":597,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953125+0000","last_change":"2024-03-24T12:08:36.168091+0000","last_active":"2024-03-24T12:09:41.953125+0000","last_peered":"2024-03-24T12:09:41.953125+0000","last_clean":"2024-03-24T12:09:41.953125+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:09:41.953125+0000","last_undegraded":"2024-03-24T12:09:41.953125+0000","last_fullsized":"2024-03-24T12:09:41.953125+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:58.095464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:03:58.095464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"375'15817","reported_seq":16095,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:42.168344+0000","last_change":"2024-03-24T12:09:20.604425+0000","last_active":"2024-03-24T12:09:42.168344+0000","last_peered":"2024-03-24T12:09:42.168344+0000","last_clean":"2024-03-24T12:09:42.168344+0000","last_became_active":"2024-03-24T12:06:30.016602+0000","last_became_peered":"2024-03-24T12:06:30.016602+0000","last_unstale":"2024-03-24T12:09:42.168344+0000","last_undegraded":"2024-03-24T12:09:42.168344+0000","last_fullsized":"2024-03-24T12:09:42.168344+0000","mapping_epoch":320,"log_start":"337'5800","ondisk_log_start":"337'5800","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'14378","last_scrub_stamp":"2024-03-24T12:09:20.604215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:09:20.604215+0000","objects_scrubbed":4208,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:28.135524+0000","scrub_duration":1324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15817,"num_write_kb":593536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'130","reported_seq":557,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997717+0000","last_change":"2024-03-24T12:07:58.303998+0000","last_active":"2024-03-24T12:09:40.997717+0000","last_peered":"2024-03-24T12:09:40.997717+0000","last_clean":"2024-03-24T12:09:40.997717+0000","last_became_active":"2024-03-24T12:03:19.822938+0000","last_became_peered":"2024-03-24T12:03:19.822938+0000","last_unstale":"2024-03-24T12:09:40.997717+0000","last_undegraded":"2024-03-24T12:09:40.997717+0000","last_fullsized":"2024-03-24T12:09:40.997717+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:58.303527+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.303527+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":710,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.959120+0000","last_change":"2024-03-24T12:07:52.313886+0000","last_active":"2024-03-24T12:09:41.959120+0000","last_peered":"2024-03-24T12:09:41.959120+0000","last_clean":"2024-03-24T12:09:41.959120+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:09:41.959120+0000","last_undegraded":"2024-03-24T12:09:41.959120+0000","last_fullsized":"2024-03-24T12:09:41.959120+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:52.313429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:52.313429+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863747+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":713,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.957159+0000","last_change":"2024-03-24T12:08:46.248251+0000","last_active":"2024-03-24T12:09:41.957159+0000","last_peered":"2024-03-24T12:09:41.957159+0000","last_clean":"2024-03-24T12:09:41.957159+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:09:41.957159+0000","last_undegraded":"2024-03-24T12:09:41.957159+0000","last_fullsized":"2024-03-24T12:09:41.957159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:46.248131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:08:46.248131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:56.816672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":167,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.995977+0000","last_change":"2024-03-24T12:07:49.261571+0000","last_active":"2024-03-24T12:09:40.995977+0000","last_peered":"2024-03-24T12:09:40.995977+0000","last_clean":"2024-03-24T12:09:40.995977+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:09:40.995977+0000","last_undegraded":"2024-03-24T12:09:40.995977+0000","last_fullsized":"2024-03-24T12:09:40.995977+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:07:49.261483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954263+0000","last_change":"2024-03-24T12:07:43.508366+0000","last_active":"2024-03-24T12:09:41.954263+0000","last_peered":"2024-03-24T12:09:41.954263+0000","last_clean":"2024-03-24T12:09:41.954263+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:09:41.954263+0000","last_undegraded":"2024-03-24T12:09:41.954263+0000","last_fullsized":"2024-03-24T12:09:41.954263+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:43.508257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:43.508257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863630+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"375'15692","reported_seq":15857,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:43.495320+0000","last_change":"2024-03-24T12:09:34.084159+0000","last_active":"2024-03-24T12:09:43.495320+0000","last_peered":"2024-03-24T12:09:43.495320+0000","last_clean":"2024-03-24T12:09:43.495320+0000","last_became_active":"2024-03-24T12:06:30.020223+0000","last_became_peered":"2024-03-24T12:06:30.020223+0000","last_unstale":"2024-03-24T12:09:43.495320+0000","last_undegraded":"2024-03-24T12:09:43.495320+0000","last_fullsized":"2024-03-24T12:09:43.495320+0000","mapping_epoch":320,"log_start":"355'11200","ondisk_log_start":"355'11200","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'15123","last_scrub_stamp":"2024-03-24T12:09:34.083776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:09:34.083776+0000","objects_scrubbed":3155,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:51.940721+0000","scrub_duration":1282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167116800,"num_objects":2550,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15692,"num_write_kb":583744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'130","reported_seq":752,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.994481+0000","last_change":"2024-03-24T12:09:39.787366+0000","last_active":"2024-03-24T12:09:40.994481+0000","last_peered":"2024-03-24T12:09:40.994481+0000","last_clean":"2024-03-24T12:09:40.994481+0000","last_became_active":"2024-03-24T12:08:36.158383+0000","last_became_peered":"2024-03-24T12:08:36.158383+0000","last_unstale":"2024-03-24T12:09:40.994481+0000","last_undegraded":"2024-03-24T12:09:40.994481+0000","last_fullsized":"2024-03-24T12:09:40.994481+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:09:39.787240+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:09:39.787240+0000","last_clean_scrub_stamp":"2024-03-24T12:09:39.787240+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:55.167890+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"296'44","reported_seq":753,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.991492+0000","last_change":"2024-03-24T12:07:37.639999+0000","last_active":"2024-03-24T12:09:40.991492+0000","last_peered":"2024-03-24T12:09:40.991492+0000","last_clean":"2024-03-24T12:09:40.991492+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:09:40.991492+0000","last_undegraded":"2024-03-24T12:09:40.991492+0000","last_fullsized":"2024-03-24T12:09:40.991492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:07:37.639868+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:07:37.639868+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":67,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.952953+0000","last_change":"2024-03-24T12:08:36.164911+0000","last_active":"2024-03-24T12:09:41.952953+0000","last_peered":"2024-03-24T12:09:41.952953+0000","last_clean":"2024-03-24T12:09:41.952953+0000","last_became_active":"2024-03-24T12:08:36.164337+0000","last_became_peered":"2024-03-24T12:08:36.164337+0000","last_unstale":"2024-03-24T12:09:41.952953+0000","last_undegraded":"2024-03-24T12:09:41.952953+0000","last_fullsized":"2024-03-24T12:09:41.952953+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:00.489543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:05:00.489543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":178,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.958839+0000","last_change":"2024-03-24T12:09:19.669080+0000","last_active":"2024-03-24T12:09:41.958839+0000","last_peered":"2024-03-24T12:09:41.958839+0000","last_clean":"2024-03-24T12:09:41.958839+0000","last_became_active":"2024-03-24T12:08:36.155105+0000","last_became_peered":"2024-03-24T12:08:36.155105+0000","last_unstale":"2024-03-24T12:09:41.958839+0000","last_undegraded":"2024-03-24T12:09:41.958839+0000","last_fullsized":"2024-03-24T12:09:41.958839+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:19.668946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:09:19.668946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:43.853782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":690,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.985621+0000","last_change":"2024-03-24T12:07:35.957089+0000","last_active":"2024-03-24T12:09:40.985621+0000","last_peered":"2024-03-24T12:09:40.985621+0000","last_clean":"2024-03-24T12:09:40.985621+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:09:40.985621+0000","last_undegraded":"2024-03-24T12:09:40.985621+0000","last_fullsized":"2024-03-24T12:09:40.985621+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:35.956974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:07:35.956974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.060843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"375'15593","reported_seq":22857,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:44.226099+0000","last_change":"2024-03-24T12:09:37.984160+0000","last_active":"2024-03-24T12:09:44.226099+0000","last_peered":"2024-03-24T12:09:44.226099+0000","last_clean":"2024-03-24T12:09:44.226099+0000","last_became_active":"2024-03-24T12:09:37.956409+0000","last_became_peered":"2024-03-24T12:09:37.956409+0000","last_unstale":"2024-03-24T12:09:44.226099+0000","last_undegraded":"2024-03-24T12:09:44.226099+0000","last_fullsized":"2024-03-24T12:09:44.226099+0000","mapping_epoch":370,"log_start":"355'11200","ondisk_log_start":"355'11200","created":320,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.744116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":7479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15593,"num_write_kb":578752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":233701376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":763,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997310+0000","last_change":"2024-03-24T12:07:41.143174+0000","last_active":"2024-03-24T12:09:40.997310+0000","last_peered":"2024-03-24T12:09:40.997310+0000","last_clean":"2024-03-24T12:09:40.997310+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:09:40.997310+0000","last_undegraded":"2024-03-24T12:09:40.997310+0000","last_fullsized":"2024-03-24T12:09:40.997310+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:07:41.143037+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.143037+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":476,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.000187+0000","last_change":"2024-03-24T12:09:04.230431+0000","last_active":"2024-03-24T12:09:41.000187+0000","last_peered":"2024-03-24T12:09:41.000187+0000","last_clean":"2024-03-24T12:09:41.000187+0000","last_became_active":"2024-03-24T12:09:04.230111+0000","last_became_peered":"2024-03-24T12:09:04.230111+0000","last_unstale":"2024-03-24T12:09:41.000187+0000","last_undegraded":"2024-03-24T12:09:41.000187+0000","last_fullsized":"2024-03-24T12:09:41.000187+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:08:42.842178+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.842178+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:05.812492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":431,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997403+0000","last_change":"2024-03-24T12:07:48.250602+0000","last_active":"2024-03-24T12:09:40.997403+0000","last_peered":"2024-03-24T12:09:40.997403+0000","last_clean":"2024-03-24T12:09:40.997403+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:09:40.997403+0000","last_undegraded":"2024-03-24T12:09:40.997403+0000","last_fullsized":"2024-03-24T12:09:40.997403+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_clean_scrub_stamp":"2024-03-24T12:07:48.250404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":167,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.983312+0000","last_change":"2024-03-24T12:08:34.445597+0000","last_active":"2024-03-24T12:09:40.983312+0000","last_peered":"2024-03-24T12:09:40.983312+0000","last_clean":"2024-03-24T12:09:40.983312+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:09:40.983312+0000","last_undegraded":"2024-03-24T12:09:40.983312+0000","last_fullsized":"2024-03-24T12:09:40.983312+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:34.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.445477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:55.101912+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":605,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.952399+0000","last_change":"2024-03-24T12:08:36.158456+0000","last_active":"2024-03-24T12:09:41.952399+0000","last_peered":"2024-03-24T12:09:41.952399+0000","last_clean":"2024-03-24T12:09:41.952399+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:09:41.952399+0000","last_undegraded":"2024-03-24T12:09:41.952399+0000","last_fullsized":"2024-03-24T12:09:41.952399+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:19.243242+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.243242+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"375'17761","reported_seq":18105,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.957001+0000","last_change":"2024-03-24T12:08:31.853008+0000","last_active":"2024-03-24T12:09:41.957001+0000","last_peered":"2024-03-24T12:09:41.957001+0000","last_clean":"2024-03-24T12:09:41.957001+0000","last_became_active":"2024-03-24T12:06:30.018755+0000","last_became_peered":"2024-03-24T12:06:30.018755+0000","last_unstale":"2024-03-24T12:09:41.957001+0000","last_undegraded":"2024-03-24T12:09:41.957001+0000","last_fullsized":"2024-03-24T12:09:41.957001+0000","mapping_epoch":320,"log_start":"350'13202","ondisk_log_start":"350'13202","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13218","last_scrub_stamp":"2024-03-24T12:08:31.852627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:08:31.852627+0000","objects_scrubbed":7225,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:58.458495+0000","scrub_duration":2505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15204,"num_write_kb":569408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":717,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997538+0000","last_change":"2024-03-24T12:07:56.328634+0000","last_active":"2024-03-24T12:09:40.997538+0000","last_peered":"2024-03-24T12:09:40.997538+0000","last_clean":"2024-03-24T12:09:40.997538+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:09:40.997538+0000","last_undegraded":"2024-03-24T12:09:40.997538+0000","last_fullsized":"2024-03-24T12:09:40.997538+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:56.328393+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.328393+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":430,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981261+0000","last_change":"2024-03-24T12:07:28.052211+0000","last_active":"2024-03-24T12:09:40.981261+0000","last_peered":"2024-03-24T12:09:40.981261+0000","last_clean":"2024-03-24T12:09:40.981261+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:09:40.981261+0000","last_undegraded":"2024-03-24T12:09:40.981261+0000","last_fullsized":"2024-03-24T12:09:40.981261+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:28.052079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:07:28.052079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":919,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.947829+0000","last_change":"2024-03-24T12:08:36.789595+0000","last_active":"2024-03-24T12:09:41.947829+0000","last_peered":"2024-03-24T12:09:41.947829+0000","last_clean":"2024-03-24T12:09:41.947829+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:09:41.947829+0000","last_undegraded":"2024-03-24T12:09:41.947829+0000","last_fullsized":"2024-03-24T12:09:41.947829+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:08:36.789473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.789473+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:06.594359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":686,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.998892+0000","last_change":"2024-03-24T12:08:36.161454+0000","last_active":"2024-03-24T12:09:40.998892+0000","last_peered":"2024-03-24T12:09:40.998892+0000","last_clean":"2024-03-24T12:09:40.998892+0000","last_became_active":"2024-03-24T12:08:36.161088+0000","last_became_peered":"2024-03-24T12:08:36.161088+0000","last_unstale":"2024-03-24T12:09:40.998892+0000","last_undegraded":"2024-03-24T12:09:40.998892+0000","last_fullsized":"2024-03-24T12:09:40.998892+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:03:47.909970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:03:47.909970+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":988,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.985535+0000","last_change":"2024-03-24T12:08:44.223168+0000","last_active":"2024-03-24T12:09:40.985535+0000","last_peered":"2024-03-24T12:09:40.985535+0000","last_clean":"2024-03-24T12:09:40.985535+0000","last_became_active":"2024-03-24T12:08:36.161668+0000","last_became_peered":"2024-03-24T12:08:36.161668+0000","last_unstale":"2024-03-24T12:09:40.985535+0000","last_undegraded":"2024-03-24T12:09:40.985535+0000","last_fullsized":"2024-03-24T12:09:40.985535+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:44.222917+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:08:44.222917+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:04.176917+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"374'15682","reported_seq":15801,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.547189+0000","last_change":"2024-03-24T12:06:30.019783+0000","last_active":"2024-03-24T12:09:41.547189+0000","last_peered":"2024-03-24T12:09:41.547189+0000","last_clean":"2024-03-24T12:09:41.547189+0000","last_became_active":"2024-03-24T12:06:30.019501+0000","last_became_peered":"2024-03-24T12:06:30.019501+0000","last_unstale":"2024-03-24T12:09:41.547189+0000","last_undegraded":"2024-03-24T12:09:41.547189+0000","last_fullsized":"2024-03-24T12:09:41.547189+0000","mapping_epoch":320,"log_start":"355'11300","ondisk_log_start":"355'11300","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.519369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178520064,"num_objects":2724,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15682,"num_write_kb":588992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":751,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.957407+0000","last_change":"2024-03-24T12:08:43.275820+0000","last_active":"2024-03-24T12:09:41.957407+0000","last_peered":"2024-03-24T12:09:41.957407+0000","last_clean":"2024-03-24T12:09:41.957407+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:09:41.957407+0000","last_undegraded":"2024-03-24T12:09:41.957407+0000","last_fullsized":"2024-03-24T12:09:41.957407+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:43.275664+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.275664+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:02.607555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":706,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955949+0000","last_change":"2024-03-24T12:08:42.224518+0000","last_active":"2024-03-24T12:09:41.955949+0000","last_peered":"2024-03-24T12:09:41.955949+0000","last_clean":"2024-03-24T12:09:41.955949+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:09:41.955949+0000","last_undegraded":"2024-03-24T12:09:41.955949+0000","last_fullsized":"2024-03-24T12:09:41.955949+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:42.224320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.224320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:57.377549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":902,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.998826+0000","last_change":"2024-03-24T12:08:36.173303+0000","last_active":"2024-03-24T12:09:40.998826+0000","last_peered":"2024-03-24T12:09:40.998826+0000","last_clean":"2024-03-24T12:09:40.998826+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:09:40.998826+0000","last_undegraded":"2024-03-24T12:09:40.998826+0000","last_fullsized":"2024-03-24T12:09:40.998826+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:05:05.860814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:05.860814+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":605,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953543+0000","last_change":"2024-03-24T12:08:36.167496+0000","last_active":"2024-03-24T12:09:41.953543+0000","last_peered":"2024-03-24T12:09:41.953543+0000","last_clean":"2024-03-24T12:09:41.953543+0000","last_became_active":"2024-03-24T12:08:36.166897+0000","last_became_peered":"2024-03-24T12:08:36.166897+0000","last_unstale":"2024-03-24T12:09:41.953543+0000","last_undegraded":"2024-03-24T12:09:41.953543+0000","last_fullsized":"2024-03-24T12:09:41.953543+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.219541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.219541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":605,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954168+0000","last_change":"2024-03-24T12:08:36.171738+0000","last_active":"2024-03-24T12:09:41.954168+0000","last_peered":"2024-03-24T12:09:41.954168+0000","last_clean":"2024-03-24T12:09:41.954168+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:09:41.954168+0000","last_undegraded":"2024-03-24T12:09:41.954168+0000","last_fullsized":"2024-03-24T12:09:41.954168+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:05:11.242568+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:05:11.242568+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"374'15557","reported_seq":15755,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.670737+0000","last_change":"2024-03-24T12:07:45.462425+0000","last_active":"2024-03-24T12:09:41.670737+0000","last_peered":"2024-03-24T12:09:41.670737+0000","last_clean":"2024-03-24T12:09:41.670737+0000","last_became_active":"2024-03-24T12:06:30.016735+0000","last_became_peered":"2024-03-24T12:06:30.016735+0000","last_unstale":"2024-03-24T12:09:41.670737+0000","last_undegraded":"2024-03-24T12:09:41.670737+0000","last_fullsized":"2024-03-24T12:09:41.670737+0000","mapping_epoch":320,"log_start":"352'11066","ondisk_log_start":"352'11066","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7238","last_scrub_stamp":"2024-03-24T12:07:45.462060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.462060+0000","objects_scrubbed":7162,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175177728,"num_objects":2673,"num_object_clones":0,"num_object_copies":8019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15557,"num_write_kb":583360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"83'130","reported_seq":570,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.958637+0000","last_change":"2024-03-24T12:09:20.672043+0000","last_active":"2024-03-24T12:09:41.958637+0000","last_peered":"2024-03-24T12:09:41.958637+0000","last_clean":"2024-03-24T12:09:41.958637+0000","last_became_active":"2024-03-24T12:08:36.153926+0000","last_became_peered":"2024-03-24T12:08:36.153926+0000","last_unstale":"2024-03-24T12:09:41.958637+0000","last_undegraded":"2024-03-24T12:09:41.958637+0000","last_fullsized":"2024-03-24T12:09:41.958637+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:09:20.671955+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:20.649114+0000","last_clean_scrub_stamp":"2024-03-24T12:09:20.671955+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:30.292057+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":710,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997864+0000","last_change":"2024-03-24T12:06:46.690666+0000","last_active":"2024-03-24T12:09:40.997864+0000","last_peered":"2024-03-24T12:09:40.997864+0000","last_clean":"2024-03-24T12:09:40.997864+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:09:40.997864+0000","last_undegraded":"2024-03-24T12:09:40.997864+0000","last_fullsized":"2024-03-24T12:09:40.997864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:06:46.690548+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:06:46.690548+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":704,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.985885+0000","last_change":"2024-03-24T12:09:26.059856+0000","last_active":"2024-03-24T12:09:40.985885+0000","last_peered":"2024-03-24T12:09:40.985885+0000","last_clean":"2024-03-24T12:09:40.985885+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:09:40.985885+0000","last_undegraded":"2024-03-24T12:09:40.985885+0000","last_fullsized":"2024-03-24T12:09:40.985885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:26.059376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:09:26.059376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:41.805225+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":601,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954290+0000","last_change":"2024-03-24T12:08:36.163502+0000","last_active":"2024-03-24T12:09:41.954290+0000","last_peered":"2024-03-24T12:09:41.954290+0000","last_clean":"2024-03-24T12:09:41.954290+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:09:41.954290+0000","last_undegraded":"2024-03-24T12:09:41.954290+0000","last_fullsized":"2024-03-24T12:09:41.954290+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:15.334211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:15.334211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":677,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981258+0000","last_change":"2024-03-24T12:08:36.153899+0000","last_active":"2024-03-24T12:09:40.981258+0000","last_peered":"2024-03-24T12:09:40.981258+0000","last_clean":"2024-03-24T12:09:40.981258+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:09:40.981258+0000","last_undegraded":"2024-03-24T12:09:40.981258+0000","last_fullsized":"2024-03-24T12:09:40.981258+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:06.995768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:06.995768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"374'15324","reported_seq":15533,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.620750+0000","last_change":"2024-03-24T12:07:58.231671+0000","last_active":"2024-03-24T12:09:41.620750+0000","last_peered":"2024-03-24T12:09:41.620750+0000","last_clean":"2024-03-24T12:09:41.620750+0000","last_became_active":"2024-03-24T12:06:30.015853+0000","last_became_peered":"2024-03-24T12:06:30.015853+0000","last_unstale":"2024-03-24T12:09:41.620750+0000","last_undegraded":"2024-03-24T12:09:41.620750+0000","last_fullsized":"2024-03-24T12:09:41.620750+0000","mapping_epoch":320,"log_start":"351'10851","ondisk_log_start":"351'10851","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'8176","last_scrub_stamp":"2024-03-24T12:07:58.231548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.231548+0000","objects_scrubbed":8101,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178257920,"num_objects":2720,"num_object_clones":0,"num_object_copies":8160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15324,"num_write_kb":577408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":891,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981689+0000","last_change":"2024-03-24T12:07:34.996457+0000","last_active":"2024-03-24T12:09:40.981689+0000","last_peered":"2024-03-24T12:09:40.981689+0000","last_clean":"2024-03-24T12:09:40.981689+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:09:40.981689+0000","last_undegraded":"2024-03-24T12:09:40.981689+0000","last_fullsized":"2024-03-24T12:09:40.981689+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:07:34.996314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:07:34.996314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":733,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997108+0000","last_change":"2024-03-24T12:08:36.162868+0000","last_active":"2024-03-24T12:09:40.997108+0000","last_peered":"2024-03-24T12:09:40.997108+0000","last_clean":"2024-03-24T12:09:40.997108+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:09:40.997108+0000","last_undegraded":"2024-03-24T12:09:40.997108+0000","last_fullsized":"2024-03-24T12:09:40.997108+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:05:29.699282+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:05:29.699282+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":694,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.994921+0000","last_change":"2024-03-24T12:07:30.557017+0000","last_active":"2024-03-24T12:09:40.994921+0000","last_peered":"2024-03-24T12:09:40.994921+0000","last_clean":"2024-03-24T12:09:40.994921+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:09:40.994921+0000","last_undegraded":"2024-03-24T12:09:40.994921+0000","last_fullsized":"2024-03-24T12:09:40.994921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.556903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.556903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":698,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.990259+0000","last_change":"2024-03-24T12:07:45.151754+0000","last_active":"2024-03-24T12:09:40.990259+0000","last_peered":"2024-03-24T12:09:40.990259+0000","last_clean":"2024-03-24T12:09:40.990259+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:09:40.990259+0000","last_undegraded":"2024-03-24T12:09:40.990259+0000","last_fullsized":"2024-03-24T12:09:40.990259+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:45.151622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.151622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":723,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981366+0000","last_change":"2024-03-24T12:08:36.201644+0000","last_active":"2024-03-24T12:09:40.981366+0000","last_peered":"2024-03-24T12:09:40.981366+0000","last_clean":"2024-03-24T12:09:40.981366+0000","last_became_active":"2024-03-24T12:08:36.146325+0000","last_became_peered":"2024-03-24T12:08:36.146325+0000","last_unstale":"2024-03-24T12:09:40.981366+0000","last_undegraded":"2024-03-24T12:09:40.981366+0000","last_fullsized":"2024-03-24T12:09:40.981366+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:36.201565+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.201565+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:50.129847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"374'15168","reported_seq":26124,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.597266+0000","last_change":"2024-03-24T12:09:17.977005+0000","last_active":"2024-03-24T12:09:41.597266+0000","last_peered":"2024-03-24T12:09:41.597266+0000","last_clean":"2024-03-24T12:09:41.597266+0000","last_became_active":"2024-03-24T12:09:17.945785+0000","last_became_peered":"2024-03-24T12:09:17.945785+0000","last_unstale":"2024-03-24T12:09:41.597266+0000","last_undegraded":"2024-03-24T12:09:41.597266+0000","last_fullsized":"2024-03-24T12:09:41.597266+0000","mapping_epoch":365,"log_start":"351'10800","ondisk_log_start":"351'10800","created":320,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4368,"log_dups_size":0,"ondisk_log_size":4368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.519298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170786816,"num_objects":2606,"num_object_clones":0,"num_object_copies":7818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15168,"num_write_kb":568768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5413,"num_bytes_recovered":354680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":671,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997212+0000","last_change":"2024-03-24T12:06:44.708778+0000","last_active":"2024-03-24T12:09:40.997212+0000","last_peered":"2024-03-24T12:09:40.997212+0000","last_clean":"2024-03-24T12:09:40.997212+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:09:40.997212+0000","last_undegraded":"2024-03-24T12:09:40.997212+0000","last_fullsized":"2024-03-24T12:09:40.997212+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:44.708677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:06:44.708677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":735,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.980785+0000","last_change":"2024-03-24T12:06:54.925013+0000","last_active":"2024-03-24T12:09:40.980785+0000","last_peered":"2024-03-24T12:09:40.980785+0000","last_clean":"2024-03-24T12:09:40.980785+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:09:40.980785+0000","last_undegraded":"2024-03-24T12:09:40.980785+0000","last_fullsized":"2024-03-24T12:09:40.980785+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:06:54.924751+0000","last_deep_scrub":"195'26","last_deep_scrub_stamp":"2024-03-24T12:00:09.391234+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.924751+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":446,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.991170+0000","last_change":"2024-03-24T12:07:40.603920+0000","last_active":"2024-03-24T12:09:40.991170+0000","last_peered":"2024-03-24T12:09:40.991170+0000","last_clean":"2024-03-24T12:09:40.991170+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:09:40.991170+0000","last_undegraded":"2024-03-24T12:09:40.991170+0000","last_fullsized":"2024-03-24T12:09:40.991170+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:07:40.603768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.603768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":689,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.996064+0000","last_change":"2024-03-24T12:06:39.791204+0000","last_active":"2024-03-24T12:09:40.996064+0000","last_peered":"2024-03-24T12:09:40.996064+0000","last_clean":"2024-03-24T12:09:40.996064+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:09:40.996064+0000","last_undegraded":"2024-03-24T12:09:40.996064+0000","last_fullsized":"2024-03-24T12:09:40.996064+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:39.791081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:06:39.791081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":732,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.996859+0000","last_change":"2024-03-24T12:08:36.163557+0000","last_active":"2024-03-24T12:09:40.996859+0000","last_peered":"2024-03-24T12:09:40.996859+0000","last_clean":"2024-03-24T12:09:40.996859+0000","last_became_active":"2024-03-24T12:08:36.163164+0000","last_became_peered":"2024-03-24T12:08:36.163164+0000","last_unstale":"2024-03-24T12:09:40.996859+0000","last_undegraded":"2024-03-24T12:09:40.996859+0000","last_fullsized":"2024-03-24T12:09:40.996859+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:18.897352+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:18.897352+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"374'15216","reported_seq":15415,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.676476+0000","last_change":"2024-03-24T12:07:46.785479+0000","last_active":"2024-03-24T12:09:41.676476+0000","last_peered":"2024-03-24T12:09:41.676476+0000","last_clean":"2024-03-24T12:09:41.676476+0000","last_became_active":"2024-03-24T12:06:30.017829+0000","last_became_peered":"2024-03-24T12:06:30.017829+0000","last_unstale":"2024-03-24T12:09:41.676476+0000","last_undegraded":"2024-03-24T12:09:41.676476+0000","last_fullsized":"2024-03-24T12:09:41.676476+0000","mapping_epoch":320,"log_start":"350'10708","ondisk_log_start":"350'10708","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7161","last_scrub_stamp":"2024-03-24T12:07:46.785035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:07:46.785035+0000","objects_scrubbed":7079,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177012760,"num_objects":2702,"num_object_clones":0,"num_object_copies":8106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15216,"num_write_kb":573313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":687,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.991658+0000","last_change":"2024-03-24T12:08:36.162113+0000","last_active":"2024-03-24T12:09:40.991658+0000","last_peered":"2024-03-24T12:09:40.991658+0000","last_clean":"2024-03-24T12:09:40.991658+0000","last_became_active":"2024-03-24T12:08:36.161658+0000","last_became_peered":"2024-03-24T12:08:36.161658+0000","last_unstale":"2024-03-24T12:09:40.991658+0000","last_undegraded":"2024-03-24T12:09:40.991658+0000","last_fullsized":"2024-03-24T12:09:40.991658+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:24.576146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:05:24.576146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"369'37","reported_seq":746,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.958983+0000","last_change":"2024-03-24T12:09:25.695502+0000","last_active":"2024-03-24T12:09:41.958983+0000","last_peered":"2024-03-24T12:09:41.958983+0000","last_clean":"2024-03-24T12:09:41.958983+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:09:41.958983+0000","last_undegraded":"2024-03-24T12:09:41.958983+0000","last_fullsized":"2024-03-24T12:09:41.958983+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'37","last_scrub_stamp":"2024-03-24T12:09:25.695317+0000","last_deep_scrub":"369'37","last_deep_scrub_stamp":"2024-03-24T12:09:25.695317+0000","last_clean_scrub_stamp":"2024-03-24T12:09:25.695317+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:50.847818+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":712,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989743+0000","last_change":"2024-03-24T12:09:30.041137+0000","last_active":"2024-03-24T12:09:40.989743+0000","last_peered":"2024-03-24T12:09:40.989743+0000","last_clean":"2024-03-24T12:09:40.989743+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:09:40.989743+0000","last_undegraded":"2024-03-24T12:09:40.989743+0000","last_fullsized":"2024-03-24T12:09:40.989743+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:30.041008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:09:30.041008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:36.464797+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":436,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.987960+0000","last_change":"2024-03-24T12:06:40.600601+0000","last_active":"2024-03-24T12:09:40.987960+0000","last_peered":"2024-03-24T12:09:40.987960+0000","last_clean":"2024-03-24T12:09:40.987960+0000","last_became_active":"2024-03-24T12:05:44.738926+0000","last_became_peered":"2024-03-24T12:05:44.738926+0000","last_unstale":"2024-03-24T12:09:40.987960+0000","last_undegraded":"2024-03-24T12:09:40.987960+0000","last_fullsized":"2024-03-24T12:09:40.987960+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:40.600197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:40.600197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":744,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.991390+0000","last_change":"2024-03-24T12:08:35.901951+0000","last_active":"2024-03-24T12:09:40.991390+0000","last_peered":"2024-03-24T12:09:40.991390+0000","last_clean":"2024-03-24T12:09:40.991390+0000","last_became_active":"2024-03-24T12:03:15.295100+0000","last_became_peered":"2024-03-24T12:03:15.295100+0000","last_unstale":"2024-03-24T12:09:40.991390+0000","last_undegraded":"2024-03-24T12:09:40.991390+0000","last_fullsized":"2024-03-24T12:09:40.991390+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:35.901858+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:35.901858+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:49.096478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"375'15435","reported_seq":15634,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:43.492180+0000","last_change":"2024-03-24T12:08:34.971600+0000","last_active":"2024-03-24T12:09:43.492180+0000","last_peered":"2024-03-24T12:09:43.492180+0000","last_clean":"2024-03-24T12:09:43.492180+0000","last_became_active":"2024-03-24T12:06:30.015684+0000","last_became_peered":"2024-03-24T12:06:30.015684+0000","last_unstale":"2024-03-24T12:09:43.492180+0000","last_undegraded":"2024-03-24T12:09:43.492180+0000","last_fullsized":"2024-03-24T12:09:43.492180+0000","mapping_epoch":320,"log_start":"351'10900","ondisk_log_start":"351'10900","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'11035","last_scrub_stamp":"2024-03-24T12:08:34.971218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.971218+0000","objects_scrubbed":6993,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:47.107246+0000","scrub_duration":2351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165478400,"num_objects":2525,"num_object_clones":0,"num_object_copies":7575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15435,"num_write_kb":574720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":567,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955377+0000","last_change":"2024-03-24T12:08:36.165575+0000","last_active":"2024-03-24T12:09:41.955377+0000","last_peered":"2024-03-24T12:09:41.955377+0000","last_clean":"2024-03-24T12:09:41.955377+0000","last_became_active":"2024-03-24T12:08:36.165265+0000","last_became_peered":"2024-03-24T12:08:36.165265+0000","last_unstale":"2024-03-24T12:09:41.955377+0000","last_undegraded":"2024-03-24T12:09:41.955377+0000","last_fullsized":"2024-03-24T12:09:41.955377+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:28.232417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:28.232417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":748,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954550+0000","last_change":"2024-03-24T12:09:35.807760+0000","last_active":"2024-03-24T12:09:41.954550+0000","last_peered":"2024-03-24T12:09:41.954550+0000","last_clean":"2024-03-24T12:09:41.954550+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:09:41.954550+0000","last_undegraded":"2024-03-24T12:09:41.954550+0000","last_fullsized":"2024-03-24T12:09:41.954550+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:09:35.807608+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:09:35.807608+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:44.198467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":526,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955768+0000","last_change":"2024-03-24T12:07:34.290389+0000","last_active":"2024-03-24T12:09:41.955768+0000","last_peered":"2024-03-24T12:09:41.955768+0000","last_clean":"2024-03-24T12:09:41.955768+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:09:41.955768+0000","last_undegraded":"2024-03-24T12:09:41.955768+0000","last_fullsized":"2024-03-24T12:09:41.955768+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:34.290275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:34.290275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863495+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":697,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953981+0000","last_change":"2024-03-24T12:07:39.389030+0000","last_active":"2024-03-24T12:09:41.953981+0000","last_peered":"2024-03-24T12:09:41.953981+0000","last_clean":"2024-03-24T12:09:41.953981+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:09:41.953981+0000","last_undegraded":"2024-03-24T12:09:41.953981+0000","last_fullsized":"2024-03-24T12:09:41.953981+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:39.387609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:07:39.387609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863659+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.956343+0000","last_change":"2024-03-24T12:08:38.243974+0000","last_active":"2024-03-24T12:09:41.956343+0000","last_peered":"2024-03-24T12:09:41.956343+0000","last_clean":"2024-03-24T12:09:41.956343+0000","last_became_active":"2024-03-24T12:08:36.173357+0000","last_became_peered":"2024-03-24T12:08:36.173357+0000","last_unstale":"2024-03-24T12:09:41.956343+0000","last_undegraded":"2024-03-24T12:09:41.956343+0000","last_fullsized":"2024-03-24T12:09:41.956343+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:38.243821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:38.243821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:01.303017+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":736,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.987889+0000","last_change":"2024-03-24T12:06:54.603424+0000","last_active":"2024-03-24T12:09:40.987889+0000","last_peered":"2024-03-24T12:09:40.987889+0000","last_clean":"2024-03-24T12:09:40.987889+0000","last_became_active":"2024-03-24T12:03:15.297422+0000","last_became_peered":"2024-03-24T12:03:15.297422+0000","last_unstale":"2024-03-24T12:09:40.987889+0000","last_undegraded":"2024-03-24T12:09:40.987889+0000","last_fullsized":"2024-03-24T12:09:40.987889+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:54.603313+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:54.603313+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"374'15213","reported_seq":15412,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.670246+0000","last_change":"2024-03-24T12:08:31.898942+0000","last_active":"2024-03-24T12:09:41.670246+0000","last_peered":"2024-03-24T12:09:41.670246+0000","last_clean":"2024-03-24T12:09:41.670246+0000","last_became_active":"2024-03-24T12:06:30.017471+0000","last_became_peered":"2024-03-24T12:06:30.017471+0000","last_unstale":"2024-03-24T12:09:41.670246+0000","last_undegraded":"2024-03-24T12:09:41.670246+0000","last_fullsized":"2024-03-24T12:09:41.670246+0000","mapping_epoch":320,"log_start":"351'10763","ondisk_log_start":"351'10763","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'10717","last_scrub_stamp":"2024-03-24T12:08:31.898767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:08:31.898767+0000","objects_scrubbed":7350,"log_size":4450,"log_dups_size":0,"ondisk_log_size":4450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.390776+0000","scrub_duration":2107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15213,"num_write_kb":575168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":697,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.992246+0000","last_change":"2024-03-24T12:08:43.232781+0000","last_active":"2024-03-24T12:09:40.992246+0000","last_peered":"2024-03-24T12:09:40.992246+0000","last_clean":"2024-03-24T12:09:40.992246+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:09:40.992246+0000","last_undegraded":"2024-03-24T12:09:40.992246+0000","last_fullsized":"2024-03-24T12:09:40.992246+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:43.232649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.232649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:07.262241+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":956,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.992073+0000","last_change":"2024-03-24T12:08:32.823527+0000","last_active":"2024-03-24T12:09:40.992073+0000","last_peered":"2024-03-24T12:09:40.992073+0000","last_clean":"2024-03-24T12:09:40.992073+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:09:40.992073+0000","last_undegraded":"2024-03-24T12:09:40.992073+0000","last_fullsized":"2024-03-24T12:09:40.992073+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:08:32.823398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:32.823398+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:44.902148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":976,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.985775+0000","last_change":"2024-03-24T12:08:49.185014+0000","last_active":"2024-03-24T12:09:40.985775+0000","last_peered":"2024-03-24T12:09:40.985775+0000","last_clean":"2024-03-24T12:09:40.985775+0000","last_became_active":"2024-03-24T12:03:15.298475+0000","last_became_peered":"2024-03-24T12:03:15.298475+0000","last_unstale":"2024-03-24T12:09:40.985775+0000","last_undegraded":"2024-03-24T12:09:40.985775+0000","last_fullsized":"2024-03-24T12:09:40.985775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:49.184886+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:03:16.845832+0000","last_clean_scrub_stamp":"2024-03-24T12:08:49.184886+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:54.517988+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":607,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953381+0000","last_change":"2024-03-24T12:08:36.169985+0000","last_active":"2024-03-24T12:09:41.953381+0000","last_peered":"2024-03-24T12:09:41.953381+0000","last_clean":"2024-03-24T12:09:41.953381+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:09:41.953381+0000","last_undegraded":"2024-03-24T12:09:41.953381+0000","last_fullsized":"2024-03-24T12:09:41.953381+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.356759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.356759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":699,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981140+0000","last_change":"2024-03-24T12:08:43.113674+0000","last_active":"2024-03-24T12:09:40.981140+0000","last_peered":"2024-03-24T12:09:40.981140+0000","last_clean":"2024-03-24T12:09:40.981140+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:09:40.981140+0000","last_undegraded":"2024-03-24T12:09:40.981140+0000","last_fullsized":"2024-03-24T12:09:40.981140+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:08:43.113510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.113510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:59.354499+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":562,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954427+0000","last_change":"2024-03-24T12:08:36.162637+0000","last_active":"2024-03-24T12:09:41.954427+0000","last_peered":"2024-03-24T12:09:41.954427+0000","last_clean":"2024-03-24T12:09:41.954427+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:09:41.954427+0000","last_undegraded":"2024-03-24T12:09:41.954427+0000","last_fullsized":"2024-03-24T12:09:41.954427+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.463242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.463242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"375'15379","reported_seq":22307,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:44.230760+0000","last_change":"2024-03-24T12:09:39.001856+0000","last_active":"2024-03-24T12:09:44.230760+0000","last_peered":"2024-03-24T12:09:44.230760+0000","last_clean":"2024-03-24T12:09:44.230760+0000","last_became_active":"2024-03-24T12:09:38.959122+0000","last_became_peered":"2024-03-24T12:09:38.959122+0000","last_unstale":"2024-03-24T12:09:44.230760+0000","last_undegraded":"2024-03-24T12:09:44.230760+0000","last_fullsized":"2024-03-24T12:09:44.230760+0000","mapping_epoch":371,"log_start":"355'11000","ondisk_log_start":"355'11000","created":320,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.744356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159449088,"num_objects":2433,"num_object_clones":0,"num_object_copies":7299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15379,"num_write_kb":569984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3398,"num_bytes_recovered":222691328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":89,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955382+0000","last_change":"2024-03-24T12:08:36.162550+0000","last_active":"2024-03-24T12:09:41.955382+0000","last_peered":"2024-03-24T12:09:41.955382+0000","last_clean":"2024-03-24T12:09:41.955382+0000","last_became_active":"2024-03-24T12:08:36.162188+0000","last_became_peered":"2024-03-24T12:08:36.162188+0000","last_unstale":"2024-03-24T12:09:41.955382+0000","last_undegraded":"2024-03-24T12:09:41.955382+0000","last_fullsized":"2024-03-24T12:09:41.955382+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.646040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":736,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.957303+0000","last_change":"2024-03-24T12:08:41.237306+0000","last_active":"2024-03-24T12:09:41.957303+0000","last_peered":"2024-03-24T12:09:41.957303+0000","last_clean":"2024-03-24T12:09:41.957303+0000","last_became_active":"2024-03-24T12:08:36.172879+0000","last_became_peered":"2024-03-24T12:08:36.172879+0000","last_unstale":"2024-03-24T12:09:41.957303+0000","last_undegraded":"2024-03-24T12:09:41.957303+0000","last_fullsized":"2024-03-24T12:09:41.957303+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:08:41.237149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:41.237149+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:42.468495+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":160,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.980971+0000","last_change":"2024-03-24T12:05:44.739463+0000","last_active":"2024-03-24T12:09:40.980971+0000","last_peered":"2024-03-24T12:09:40.980971+0000","last_clean":"2024-03-24T12:09:40.980971+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:09:40.980971+0000","last_undegraded":"2024-03-24T12:09:40.980971+0000","last_fullsized":"2024-03-24T12:09:40.980971+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:17.610545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.610545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":449,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.993831+0000","last_change":"2024-03-24T12:06:55.648298+0000","last_active":"2024-03-24T12:09:40.993831+0000","last_peered":"2024-03-24T12:09:40.993831+0000","last_clean":"2024-03-24T12:09:40.993831+0000","last_became_active":"2024-03-24T12:05:44.736437+0000","last_became_peered":"2024-03-24T12:05:44.736437+0000","last_unstale":"2024-03-24T12:09:40.993831+0000","last_undegraded":"2024-03-24T12:09:40.993831+0000","last_fullsized":"2024-03-24T12:09:40.993831+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:06:55.648166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:55.648166+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":243,"num_read_kb":238,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":692,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989659+0000","last_change":"2024-03-24T12:09:39.094894+0000","last_active":"2024-03-24T12:09:40.989659+0000","last_peered":"2024-03-24T12:09:40.989659+0000","last_clean":"2024-03-24T12:09:40.989659+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:09:40.989659+0000","last_undegraded":"2024-03-24T12:09:40.989659+0000","last_fullsized":"2024-03-24T12:09:40.989659+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:39.094777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:09:39.094777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:04.218387+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":983,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989689+0000","last_change":"2024-03-24T12:09:29.088550+0000","last_active":"2024-03-24T12:09:40.989689+0000","last_peered":"2024-03-24T12:09:40.989689+0000","last_clean":"2024-03-24T12:09:40.989689+0000","last_became_active":"2024-03-24T12:03:15.298007+0000","last_became_peered":"2024-03-24T12:03:15.298007+0000","last_unstale":"2024-03-24T12:09:40.989689+0000","last_undegraded":"2024-03-24T12:09:40.989689+0000","last_fullsized":"2024-03-24T12:09:40.989689+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:09:29.088421+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:09:29.088421+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:46.367625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"374'15416","reported_seq":22505,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.670692+0000","last_change":"2024-03-24T12:09:38.980474+0000","last_active":"2024-03-24T12:09:41.670692+0000","last_peered":"2024-03-24T12:09:41.670692+0000","last_clean":"2024-03-24T12:09:41.670692+0000","last_became_active":"2024-03-24T12:09:38.957397+0000","last_became_peered":"2024-03-24T12:09:38.957397+0000","last_unstale":"2024-03-24T12:09:41.670692+0000","last_undegraded":"2024-03-24T12:09:41.670692+0000","last_fullsized":"2024-03-24T12:09:41.670692+0000","mapping_epoch":371,"log_start":"351'10945","ondisk_log_start":"351'10945","created":320,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170655744,"num_objects":2604,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15416,"num_write_kb":576640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3478,"num_bytes_recovered":227934208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":674,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.963692+0000","last_change":"2024-03-24T12:06:47.944634+0000","last_active":"2024-03-24T12:09:41.963692+0000","last_peered":"2024-03-24T12:09:41.963692+0000","last_clean":"2024-03-24T12:09:41.963692+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:09:41.963692+0000","last_undegraded":"2024-03-24T12:09:41.963692+0000","last_fullsized":"2024-03-24T12:09:41.963692+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:47.944504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:06:47.944504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"354'44","reported_seq":797,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953976+0000","last_change":"2024-03-24T12:08:36.153332+0000","last_active":"2024-03-24T12:09:41.953976+0000","last_peered":"2024-03-24T12:09:41.953976+0000","last_clean":"2024-03-24T12:09:41.953976+0000","last_became_active":"2024-03-24T12:08:36.152866+0000","last_became_peered":"2024-03-24T12:08:36.152866+0000","last_unstale":"2024-03-24T12:09:41.953976+0000","last_undegraded":"2024-03-24T12:09:41.953976+0000","last_fullsized":"2024-03-24T12:09:41.953976+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-24T12:04:07.131550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:07.131550+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737959+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":571,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.952982+0000","last_change":"2024-03-24T12:08:36.170514+0000","last_active":"2024-03-24T12:09:41.952982+0000","last_peered":"2024-03-24T12:09:41.952982+0000","last_clean":"2024-03-24T12:09:41.952982+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:09:41.952982+0000","last_undegraded":"2024-03-24T12:09:41.952982+0000","last_fullsized":"2024-03-24T12:09:41.952982+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.196749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.196749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":669,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981803+0000","last_change":"2024-03-24T12:07:33.984169+0000","last_active":"2024-03-24T12:09:40.981803+0000","last_peered":"2024-03-24T12:09:40.981803+0000","last_clean":"2024-03-24T12:09:40.981803+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:09:40.981803+0000","last_undegraded":"2024-03-24T12:09:40.981803+0000","last_fullsized":"2024-03-24T12:09:40.981803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:33.984042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:33.984042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":614,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955770+0000","last_change":"2024-03-24T12:08:36.157710+0000","last_active":"2024-03-24T12:09:41.955770+0000","last_peered":"2024-03-24T12:09:41.955770+0000","last_clean":"2024-03-24T12:09:41.955770+0000","last_became_active":"2024-03-24T12:08:36.155758+0000","last_became_peered":"2024-03-24T12:08:36.155758+0000","last_unstale":"2024-03-24T12:09:41.955770+0000","last_undegraded":"2024-03-24T12:09:41.955770+0000","last_fullsized":"2024-03-24T12:09:41.955770+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:04:47.152371+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:04:47.152371+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"374'15367","reported_seq":15558,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.661159+0000","last_change":"2024-03-24T12:08:50.231238+0000","last_active":"2024-03-24T12:09:41.661159+0000","last_peered":"2024-03-24T12:09:41.661159+0000","last_clean":"2024-03-24T12:09:41.661159+0000","last_became_active":"2024-03-24T12:06:30.017875+0000","last_became_peered":"2024-03-24T12:06:30.017875+0000","last_unstale":"2024-03-24T12:09:41.661159+0000","last_undegraded":"2024-03-24T12:09:41.661159+0000","last_fullsized":"2024-03-24T12:09:41.661159+0000","mapping_epoch":320,"log_start":"355'11100","ondisk_log_start":"355'11100","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'11995","last_scrub_stamp":"2024-03-24T12:08:50.231089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:08:50.231089+0000","objects_scrubbed":6181,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:09.714731+0000","scrub_duration":2108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179634176,"num_objects":2741,"num_object_clones":0,"num_object_copies":8223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15367,"num_write_kb":579456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":911,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.959210+0000","last_change":"2024-03-24T12:08:36.673921+0000","last_active":"2024-03-24T12:09:41.959210+0000","last_peered":"2024-03-24T12:09:41.959210+0000","last_clean":"2024-03-24T12:09:41.959210+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:09:41.959210+0000","last_undegraded":"2024-03-24T12:09:41.959210+0000","last_fullsized":"2024-03-24T12:09:41.959210+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:08:36.673814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.673814+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:43.939613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":738,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.956634+0000","last_change":"2024-03-24T12:08:39.239739+0000","last_active":"2024-03-24T12:09:41.956634+0000","last_peered":"2024-03-24T12:09:41.956634+0000","last_clean":"2024-03-24T12:09:41.956634+0000","last_became_active":"2024-03-24T12:08:36.174663+0000","last_became_peered":"2024-03-24T12:08:36.174663+0000","last_unstale":"2024-03-24T12:09:41.956634+0000","last_undegraded":"2024-03-24T12:09:41.956634+0000","last_fullsized":"2024-03-24T12:09:41.956634+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:08:39.239229+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:08:39.239229+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:50.536814+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":432,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955338+0000","last_change":"2024-03-24T12:06:49.982193+0000","last_active":"2024-03-24T12:09:41.955338+0000","last_peered":"2024-03-24T12:09:41.955338+0000","last_clean":"2024-03-24T12:09:41.955338+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:09:41.955338+0000","last_undegraded":"2024-03-24T12:09:41.955338+0000","last_fullsized":"2024-03-24T12:09:41.955338+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:49.981897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.981897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863405+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":685,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955314+0000","last_change":"2024-03-24T12:09:21.734702+0000","last_active":"2024-03-24T12:09:41.955314+0000","last_peered":"2024-03-24T12:09:41.955314+0000","last_clean":"2024-03-24T12:09:41.955314+0000","last_became_active":"2024-03-24T12:08:36.145936+0000","last_became_peered":"2024-03-24T12:08:36.145936+0000","last_unstale":"2024-03-24T12:09:41.955314+0000","last_undegraded":"2024-03-24T12:09:41.955314+0000","last_fullsized":"2024-03-24T12:09:41.955314+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:21.734396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:09:21.734396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:50.267058+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":736,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.994596+0000","last_change":"2024-03-24T12:07:32.603069+0000","last_active":"2024-03-24T12:09:40.994596+0000","last_peered":"2024-03-24T12:09:40.994596+0000","last_clean":"2024-03-24T12:09:40.994596+0000","last_became_active":"2024-03-24T12:03:15.298983+0000","last_became_peered":"2024-03-24T12:03:15.298983+0000","last_unstale":"2024-03-24T12:09:40.994596+0000","last_undegraded":"2024-03-24T12:09:40.994596+0000","last_fullsized":"2024-03-24T12:09:40.994596+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:32.602779+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:07:32.602779+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"374'15485","reported_seq":26416,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.664616+0000","last_change":"2024-03-24T12:09:18.987861+0000","last_active":"2024-03-24T12:09:41.664616+0000","last_peered":"2024-03-24T12:09:41.664616+0000","last_clean":"2024-03-24T12:09:41.664616+0000","last_became_active":"2024-03-24T12:09:18.950940+0000","last_became_peered":"2024-03-24T12:09:18.950940+0000","last_unstale":"2024-03-24T12:09:41.664616+0000","last_undegraded":"2024-03-24T12:09:41.664616+0000","last_fullsized":"2024-03-24T12:09:41.664616+0000","mapping_epoch":366,"log_start":"351'11014","ondisk_log_start":"351'11014","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:06:28.998031+0000","last_clean_scrub_stamp":"2024-03-24T12:06:28.998031+0000","objects_scrubbed":0,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174522368,"num_objects":2663,"num_object_clones":0,"num_object_copies":7989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15485,"num_write_kb":580736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5401,"num_bytes_recovered":353959936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"83'130","reported_seq":727,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981057+0000","last_change":"2024-03-24T12:07:27.057073+0000","last_active":"2024-03-24T12:09:40.981057+0000","last_peered":"2024-03-24T12:09:40.981057+0000","last_clean":"2024-03-24T12:09:40.981057+0000","last_became_active":"2024-03-24T12:03:19.827283+0000","last_became_peered":"2024-03-24T12:03:19.827283+0000","last_unstale":"2024-03-24T12:09:40.981057+0000","last_undegraded":"2024-03-24T12:09:40.981057+0000","last_fullsized":"2024-03-24T12:09:40.981057+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:27.056921+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:07:27.056921+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":707,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.947091+0000","last_change":"2024-03-24T12:09:36.374858+0000","last_active":"2024-03-24T12:09:41.947091+0000","last_peered":"2024-03-24T12:09:41.947091+0000","last_clean":"2024-03-24T12:09:41.947091+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:09:41.947091+0000","last_undegraded":"2024-03-24T12:09:41.947091+0000","last_fullsized":"2024-03-24T12:09:41.947091+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:36.374505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:09:36.374505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:05.979873+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":741,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997184+0000","last_change":"2024-03-24T12:08:34.327181+0000","last_active":"2024-03-24T12:09:40.997184+0000","last_peered":"2024-03-24T12:09:40.997184+0000","last_clean":"2024-03-24T12:09:40.997184+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:09:40.997184+0000","last_undegraded":"2024-03-24T12:09:40.997184+0000","last_fullsized":"2024-03-24T12:09:40.997184+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:08:34.327037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.327037+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:02.132550+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":698,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.996239+0000","last_change":"2024-03-24T12:07:45.216727+0000","last_active":"2024-03-24T12:09:40.996239+0000","last_peered":"2024-03-24T12:09:40.996239+0000","last_clean":"2024-03-24T12:09:40.996239+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:09:40.996239+0000","last_undegraded":"2024-03-24T12:09:40.996239+0000","last_fullsized":"2024-03-24T12:09:40.996239+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:07:45.216581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.216581+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":668,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981951+0000","last_change":"2024-03-24T12:07:47.318834+0000","last_active":"2024-03-24T12:09:40.981951+0000","last_peered":"2024-03-24T12:09:40.981951+0000","last_clean":"2024-03-24T12:09:40.981951+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:09:40.981951+0000","last_undegraded":"2024-03-24T12:09:40.981951+0000","last_fullsized":"2024-03-24T12:09:40.981951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:47.318040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:07:47.318040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"83'130","reported_seq":552,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.980553+0000","last_change":"2024-03-24T12:06:52.949501+0000","last_active":"2024-03-24T12:09:40.980553+0000","last_peered":"2024-03-24T12:09:40.980553+0000","last_clean":"2024-03-24T12:09:40.980553+0000","last_became_active":"2024-03-24T12:03:19.828385+0000","last_became_peered":"2024-03-24T12:03:19.828385+0000","last_unstale":"2024-03-24T12:09:40.980553+0000","last_undegraded":"2024-03-24T12:09:40.980553+0000","last_fullsized":"2024-03-24T12:09:40.980553+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:06:52.949381+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:06:52.949381+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":714,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954651+0000","last_change":"2024-03-24T12:07:26.164276+0000","last_active":"2024-03-24T12:09:41.954651+0000","last_peered":"2024-03-24T12:09:41.954651+0000","last_clean":"2024-03-24T12:09:41.954651+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:09:41.954651+0000","last_undegraded":"2024-03-24T12:09:41.954651+0000","last_fullsized":"2024-03-24T12:09:41.954651+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:07:26.164151+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:07:26.164151+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863466+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":178,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955544+0000","last_change":"2024-03-24T12:09:36.778664+0000","last_active":"2024-03-24T12:09:41.955544+0000","last_peered":"2024-03-24T12:09:41.955544+0000","last_clean":"2024-03-24T12:09:41.955544+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:09:41.955544+0000","last_undegraded":"2024-03-24T12:09:41.955544+0000","last_fullsized":"2024-03-24T12:09:41.955544+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:36.778553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:09:36.778553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:39.848476+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":664,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.983225+0000","last_change":"2024-03-24T12:06:42.950243+0000","last_active":"2024-03-24T12:09:40.983225+0000","last_peered":"2024-03-24T12:09:40.983225+0000","last_clean":"2024-03-24T12:09:40.983225+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:09:40.983225+0000","last_undegraded":"2024-03-24T12:09:40.983225+0000","last_fullsized":"2024-03-24T12:09:40.983225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.950004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.950004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":167,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997091+0000","last_change":"2024-03-24T12:07:50.278024+0000","last_active":"2024-03-24T12:09:40.997091+0000","last_peered":"2024-03-24T12:09:40.997091+0000","last_clean":"2024-03-24T12:09:40.997091+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:09:40.997091+0000","last_undegraded":"2024-03-24T12:09:40.997091+0000","last_fullsized":"2024-03-24T12:09:40.997091+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:50.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:50.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"83'130","reported_seq":475,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953218+0000","last_change":"2024-03-24T12:08:36.171122+0000","last_active":"2024-03-24T12:09:41.953218+0000","last_peered":"2024-03-24T12:09:41.953218+0000","last_clean":"2024-03-24T12:09:41.953218+0000","last_became_active":"2024-03-24T12:08:36.167969+0000","last_became_peered":"2024-03-24T12:08:36.167969+0000","last_unstale":"2024-03-24T12:09:41.953218+0000","last_undegraded":"2024-03-24T12:09:41.953218+0000","last_fullsized":"2024-03-24T12:09:41.953218+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:03:48.080209+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:03:48.080209+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":694,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.983186+0000","last_change":"2024-03-24T12:06:49.874361+0000","last_active":"2024-03-24T12:09:40.983186+0000","last_peered":"2024-03-24T12:09:40.983186+0000","last_clean":"2024-03-24T12:09:40.983186+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:09:40.983186+0000","last_undegraded":"2024-03-24T12:09:40.983186+0000","last_fullsized":"2024-03-24T12:09:40.983186+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:06:49.873894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:49.873894+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":701,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.988197+0000","last_change":"2024-03-24T12:08:30.827983+0000","last_active":"2024-03-24T12:09:40.988197+0000","last_peered":"2024-03-24T12:09:40.988197+0000","last_clean":"2024-03-24T12:09:40.988197+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:09:40.988197+0000","last_undegraded":"2024-03-24T12:09:40.988197+0000","last_fullsized":"2024-03-24T12:09:40.988197+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:30.827866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:08:30.827866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:48.184568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":519,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955538+0000","last_change":"2024-03-24T12:08:36.170084+0000","last_active":"2024-03-24T12:09:41.955538+0000","last_peered":"2024-03-24T12:09:41.955538+0000","last_clean":"2024-03-24T12:09:41.955538+0000","last_became_active":"2024-03-24T12:08:36.165715+0000","last_became_peered":"2024-03-24T12:08:36.165715+0000","last_unstale":"2024-03-24T12:09:41.955538+0000","last_undegraded":"2024-03-24T12:09:41.955538+0000","last_fullsized":"2024-03-24T12:09:41.955538+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:11.242845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:04:11.242845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":686,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989431+0000","last_change":"2024-03-24T12:07:30.073083+0000","last_active":"2024-03-24T12:09:40.989431+0000","last_peered":"2024-03-24T12:09:40.989431+0000","last_clean":"2024-03-24T12:09:40.989431+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:09:40.989431+0000","last_undegraded":"2024-03-24T12:09:40.989431+0000","last_fullsized":"2024-03-24T12:09:40.989431+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.072953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.072953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"83'130","reported_seq":582,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953732+0000","last_change":"2024-03-24T12:09:28.747992+0000","last_active":"2024-03-24T12:09:41.953732+0000","last_peered":"2024-03-24T12:09:41.953732+0000","last_clean":"2024-03-24T12:09:41.953732+0000","last_became_active":"2024-03-24T12:08:36.143986+0000","last_became_peered":"2024-03-24T12:08:36.143986+0000","last_unstale":"2024-03-24T12:09:41.953732+0000","last_undegraded":"2024-03-24T12:09:41.953732+0000","last_fullsized":"2024-03-24T12:09:41.953732+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:09:28.747523+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:09:28.747523+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:32.957206+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":711,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981326+0000","last_change":"2024-03-24T12:06:53.944441+0000","last_active":"2024-03-24T12:09:40.981326+0000","last_peered":"2024-03-24T12:09:40.981326+0000","last_clean":"2024-03-24T12:09:40.981326+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:09:40.981326+0000","last_undegraded":"2024-03-24T12:09:40.981326+0000","last_fullsized":"2024-03-24T12:09:40.981326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:06:53.944281+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:06:53.944281+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":570,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953380+0000","last_change":"2024-03-24T12:08:36.171996+0000","last_active":"2024-03-24T12:09:41.953380+0000","last_peered":"2024-03-24T12:09:41.953380+0000","last_clean":"2024-03-24T12:09:41.953380+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:09:41.953380+0000","last_undegraded":"2024-03-24T12:09:41.953380+0000","last_fullsized":"2024-03-24T12:09:41.953380+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.284540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.284540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":796,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953304+0000","last_change":"2024-03-24T12:08:36.167205+0000","last_active":"2024-03-24T12:09:41.953304+0000","last_peered":"2024-03-24T12:09:41.953304+0000","last_clean":"2024-03-24T12:09:41.953304+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:09:41.953304+0000","last_undegraded":"2024-03-24T12:09:41.953304+0000","last_fullsized":"2024-03-24T12:09:41.953304+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:03:51.155727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:51.155727+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":677,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954482+0000","last_change":"2024-03-24T12:07:58.405060+0000","last_active":"2024-03-24T12:09:41.954482+0000","last_peered":"2024-03-24T12:09:41.954482+0000","last_clean":"2024-03-24T12:09:41.954482+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:09:41.954482+0000","last_undegraded":"2024-03-24T12:09:41.954482+0000","last_fullsized":"2024-03-24T12:09:41.954482+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:58.404932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.404932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863716+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"83'130","reported_seq":485,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.952300+0000","last_change":"2024-03-24T12:08:36.620186+0000","last_active":"2024-03-24T12:09:41.952300+0000","last_peered":"2024-03-24T12:09:41.952300+0000","last_clean":"2024-03-24T12:09:41.952300+0000","last_became_active":"2024-03-24T12:08:36.152274+0000","last_became_peered":"2024-03-24T12:08:36.152274+0000","last_unstale":"2024-03-24T12:09:41.952300+0000","last_undegraded":"2024-03-24T12:09:41.952300+0000","last_fullsized":"2024-03-24T12:09:41.952300+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:36.620055+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.620055+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:05.250482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":595,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953729+0000","last_change":"2024-03-24T12:08:36.154109+0000","last_active":"2024-03-24T12:09:41.953729+0000","last_peered":"2024-03-24T12:09:41.953729+0000","last_clean":"2024-03-24T12:09:41.953729+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:09:41.953729+0000","last_undegraded":"2024-03-24T12:09:41.953729+0000","last_fullsized":"2024-03-24T12:09:41.953729+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738254+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":71,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.985880+0000","last_change":"2024-03-24T12:08:15.996146+0000","last_active":"2024-03-24T12:09:40.985880+0000","last_peered":"2024-03-24T12:09:40.985880+0000","last_clean":"2024-03-24T12:09:40.985880+0000","last_became_active":"2024-03-24T12:08:15.995868+0000","last_became_peered":"2024-03-24T12:08:15.995868+0000","last_unstale":"2024-03-24T12:09:40.985880+0000","last_undegraded":"2024-03-24T12:09:40.985880+0000","last_fullsized":"2024-03-24T12:09:40.985880+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:45.449624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.449624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":900,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981466+0000","last_change":"2024-03-24T12:06:10.014367+0000","last_active":"2024-03-24T12:09:40.981466+0000","last_peered":"2024-03-24T12:09:40.981466+0000","last_clean":"2024-03-24T12:09:40.981466+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:09:40.981466+0000","last_undegraded":"2024-03-24T12:09:40.981466+0000","last_fullsized":"2024-03-24T12:09:40.981466+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:06:10.014224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:10.014224+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":675,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.996483+0000","last_change":"2024-03-24T12:08:36.160517+0000","last_active":"2024-03-24T12:09:40.996483+0000","last_peered":"2024-03-24T12:09:40.996483+0000","last_clean":"2024-03-24T12:09:40.996483+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:09:40.996483+0000","last_undegraded":"2024-03-24T12:09:40.996483+0000","last_fullsized":"2024-03-24T12:09:40.996483+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:03.041566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:05:03.041566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":482,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.991261+0000","last_change":"2024-03-24T12:07:38.607052+0000","last_active":"2024-03-24T12:09:40.991261+0000","last_peered":"2024-03-24T12:09:40.991261+0000","last_clean":"2024-03-24T12:09:40.991261+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:09:40.991261+0000","last_undegraded":"2024-03-24T12:09:40.991261+0000","last_fullsized":"2024-03-24T12:09:40.991261+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:38.606914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:38.606914+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":702,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.988213+0000","last_change":"2024-03-24T12:09:37.716173+0000","last_active":"2024-03-24T12:09:40.988213+0000","last_peered":"2024-03-24T12:09:40.988213+0000","last_clean":"2024-03-24T12:09:40.988213+0000","last_became_active":"2024-03-24T12:08:36.152805+0000","last_became_peered":"2024-03-24T12:08:36.152805+0000","last_unstale":"2024-03-24T12:09:40.988213+0000","last_undegraded":"2024-03-24T12:09:40.988213+0000","last_fullsized":"2024-03-24T12:09:40.988213+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:37.716096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:09:37.716096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:43.728287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":678,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981857+0000","last_change":"2024-03-24T12:07:29.080020+0000","last_active":"2024-03-24T12:09:40.981857+0000","last_peered":"2024-03-24T12:09:40.981857+0000","last_clean":"2024-03-24T12:09:40.981857+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:09:40.981857+0000","last_undegraded":"2024-03-24T12:09:40.981857+0000","last_fullsized":"2024-03-24T12:09:40.981857+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:29.079435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:07:29.079435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":686,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955201+0000","last_change":"2024-03-24T12:09:30.837253+0000","last_active":"2024-03-24T12:09:41.955201+0000","last_peered":"2024-03-24T12:09:41.955201+0000","last_clean":"2024-03-24T12:09:41.955201+0000","last_became_active":"2024-03-24T12:08:36.146995+0000","last_became_peered":"2024-03-24T12:08:36.146995+0000","last_unstale":"2024-03-24T12:09:41.955201+0000","last_undegraded":"2024-03-24T12:09:41.955201+0000","last_fullsized":"2024-03-24T12:09:41.955201+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:30.836662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:30.836662+0000","last_clean_scrub_stamp":"2024-03-24T12:09:30.836662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:50.790329+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":705,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.000317+0000","last_change":"2024-03-24T12:06:50.625611+0000","last_active":"2024-03-24T12:09:41.000317+0000","last_peered":"2024-03-24T12:09:41.000317+0000","last_clean":"2024-03-24T12:09:41.000317+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:09:41.000317+0000","last_undegraded":"2024-03-24T12:09:41.000317+0000","last_fullsized":"2024-03-24T12:09:41.000317+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:06:50.625428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:50.625428+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":714,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.956833+0000","last_change":"2024-03-24T12:08:40.215573+0000","last_active":"2024-03-24T12:09:41.956833+0000","last_peered":"2024-03-24T12:09:41.956833+0000","last_clean":"2024-03-24T12:09:41.956833+0000","last_became_active":"2024-03-24T12:08:36.174093+0000","last_became_peered":"2024-03-24T12:08:36.174093+0000","last_unstale":"2024-03-24T12:09:41.956833+0000","last_undegraded":"2024-03-24T12:09:41.956833+0000","last_fullsized":"2024-03-24T12:09:41.956833+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.215463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.215463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:05.250154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":329,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.952745+0000","last_change":"2024-03-24T12:09:38.661845+0000","last_active":"2024-03-24T12:09:41.952745+0000","last_peered":"2024-03-24T12:09:41.952745+0000","last_clean":"2024-03-24T12:09:41.952745+0000","last_became_active":"2024-03-24T12:08:36.166281+0000","last_became_peered":"2024-03-24T12:08:36.166281+0000","last_unstale":"2024-03-24T12:09:41.952745+0000","last_undegraded":"2024-03-24T12:09:41.952745+0000","last_fullsized":"2024-03-24T12:09:41.952745+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.661755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.661755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:51.606689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":669,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.980641+0000","last_change":"2024-03-24T12:08:36.145270+0000","last_active":"2024-03-24T12:09:40.980641+0000","last_peered":"2024-03-24T12:09:40.980641+0000","last_clean":"2024-03-24T12:09:40.980641+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:09:40.980641+0000","last_undegraded":"2024-03-24T12:09:40.980641+0000","last_fullsized":"2024-03-24T12:09:40.980641+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:25.928874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.928874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":698,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.980612+0000","last_change":"2024-03-24T12:07:56.051237+0000","last_active":"2024-03-24T12:09:40.980612+0000","last_peered":"2024-03-24T12:09:40.980612+0000","last_clean":"2024-03-24T12:09:40.980612+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:09:40.980612+0000","last_undegraded":"2024-03-24T12:09:40.980612+0000","last_fullsized":"2024-03-24T12:09:40.980612+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:07:56.051061+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.051061+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":583,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955495+0000","last_change":"2024-03-24T12:08:36.170759+0000","last_active":"2024-03-24T12:09:41.955495+0000","last_peered":"2024-03-24T12:09:41.955495+0000","last_clean":"2024-03-24T12:09:41.955495+0000","last_became_active":"2024-03-24T12:08:36.166947+0000","last_became_peered":"2024-03-24T12:08:36.166947+0000","last_unstale":"2024-03-24T12:09:41.955495+0000","last_undegraded":"2024-03-24T12:09:41.955495+0000","last_fullsized":"2024-03-24T12:09:41.955495+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:18.028433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:03:18.028433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":685,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.988112+0000","last_change":"2024-03-24T12:06:42.538679+0000","last_active":"2024-03-24T12:09:40.988112+0000","last_peered":"2024-03-24T12:09:40.988112+0000","last_clean":"2024-03-24T12:09:40.988112+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:09:40.988112+0000","last_undegraded":"2024-03-24T12:09:40.988112+0000","last_fullsized":"2024-03-24T12:09:40.988112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.538568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.538568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":683,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989781+0000","last_change":"2024-03-24T12:08:33.337568+0000","last_active":"2024-03-24T12:09:40.989781+0000","last_peered":"2024-03-24T12:09:40.989781+0000","last_clean":"2024-03-24T12:09:40.989781+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:09:40.989781+0000","last_undegraded":"2024-03-24T12:09:40.989781+0000","last_fullsized":"2024-03-24T12:09:40.989781+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:33.337437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:08:33.337437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:00.884474+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"83'130","reported_seq":742,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.947260+0000","last_change":"2024-03-24T12:09:20.291224+0000","last_active":"2024-03-24T12:09:41.947260+0000","last_peered":"2024-03-24T12:09:41.947260+0000","last_clean":"2024-03-24T12:09:41.947260+0000","last_became_active":"2024-03-24T12:03:19.819346+0000","last_became_peered":"2024-03-24T12:03:19.819346+0000","last_unstale":"2024-03-24T12:09:41.947260+0000","last_undegraded":"2024-03-24T12:09:41.947260+0000","last_fullsized":"2024-03-24T12:09:41.947260+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:09:20.291101+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:09:20.291101+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:35.522829+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":699,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981073+0000","last_change":"2024-03-24T12:06:13.055590+0000","last_active":"2024-03-24T12:09:40.981073+0000","last_peered":"2024-03-24T12:09:40.981073+0000","last_clean":"2024-03-24T12:09:40.981073+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:09:40.981073+0000","last_undegraded":"2024-03-24T12:09:40.981073+0000","last_fullsized":"2024-03-24T12:09:40.981073+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:06:13.055489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:13.055489+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":700,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.994708+0000","last_change":"2024-03-24T12:09:38.737527+0000","last_active":"2024-03-24T12:09:40.994708+0000","last_peered":"2024-03-24T12:09:40.994708+0000","last_clean":"2024-03-24T12:09:40.994708+0000","last_became_active":"2024-03-24T12:08:36.157290+0000","last_became_peered":"2024-03-24T12:08:36.157290+0000","last_unstale":"2024-03-24T12:09:40.994708+0000","last_undegraded":"2024-03-24T12:09:40.994708+0000","last_fullsized":"2024-03-24T12:09:40.994708+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.737417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:41.821998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":687,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.986099+0000","last_change":"2024-03-24T12:07:55.049337+0000","last_active":"2024-03-24T12:09:40.986099+0000","last_peered":"2024-03-24T12:09:40.986099+0000","last_clean":"2024-03-24T12:09:40.986099+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:09:40.986099+0000","last_undegraded":"2024-03-24T12:09:40.986099+0000","last_fullsized":"2024-03-24T12:09:40.986099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:55.049210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:07:55.049210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":660,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.982039+0000","last_change":"2024-03-24T12:06:18.013886+0000","last_active":"2024-03-24T12:09:40.982039+0000","last_peered":"2024-03-24T12:09:40.982039+0000","last_clean":"2024-03-24T12:09:40.982039+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:09:40.982039+0000","last_undegraded":"2024-03-24T12:09:40.982039+0000","last_fullsized":"2024-03-24T12:09:40.982039+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:18.013768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:06:18.013768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":721,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954853+0000","last_change":"2024-03-24T12:07:51.268007+0000","last_active":"2024-03-24T12:09:41.954853+0000","last_peered":"2024-03-24T12:09:41.954853+0000","last_clean":"2024-03-24T12:09:41.954853+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:09:41.954853+0000","last_undegraded":"2024-03-24T12:09:41.954853+0000","last_fullsized":"2024-03-24T12:09:41.954853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:51.267854+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.267854+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863691+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":535,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.952214+0000","last_change":"2024-03-24T12:08:36.161709+0000","last_active":"2024-03-24T12:09:41.952214+0000","last_peered":"2024-03-24T12:09:41.952214+0000","last_clean":"2024-03-24T12:09:41.952214+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:09:41.952214+0000","last_undegraded":"2024-03-24T12:09:41.952214+0000","last_fullsized":"2024-03-24T12:09:41.952214+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.160982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.160982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":908,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.995073+0000","last_change":"2024-03-24T12:07:51.843434+0000","last_active":"2024-03-24T12:09:40.995073+0000","last_peered":"2024-03-24T12:09:40.995073+0000","last_clean":"2024-03-24T12:09:40.995073+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:09:40.995073+0000","last_undegraded":"2024-03-24T12:09:40.995073+0000","last_fullsized":"2024-03-24T12:09:40.995073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:07:51.843305+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.843305+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":701,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.992397+0000","last_change":"2024-03-24T12:08:42.237732+0000","last_active":"2024-03-24T12:09:40.992397+0000","last_peered":"2024-03-24T12:09:40.992397+0000","last_clean":"2024-03-24T12:09:40.992397+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:09:40.992397+0000","last_undegraded":"2024-03-24T12:09:40.992397+0000","last_fullsized":"2024-03-24T12:09:40.992397+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:08:42.237615+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.237615+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:50.512255+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":730,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.995169+0000","last_change":"2024-03-24T12:08:36.157338+0000","last_active":"2024-03-24T12:09:40.995169+0000","last_peered":"2024-03-24T12:09:40.995169+0000","last_clean":"2024-03-24T12:09:40.995169+0000","last_became_active":"2024-03-24T12:08:36.156563+0000","last_became_peered":"2024-03-24T12:08:36.156563+0000","last_unstale":"2024-03-24T12:09:40.995169+0000","last_undegraded":"2024-03-24T12:09:40.995169+0000","last_fullsized":"2024-03-24T12:09:40.995169+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:05:25.602806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:25.602806+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":673,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.980827+0000","last_change":"2024-03-24T12:06:40.021255+0000","last_active":"2024-03-24T12:09:40.980827+0000","last_peered":"2024-03-24T12:09:40.980827+0000","last_clean":"2024-03-24T12:09:40.980827+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:09:40.980827+0000","last_undegraded":"2024-03-24T12:09:40.980827+0000","last_fullsized":"2024-03-24T12:09:40.980827+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:06:40.021114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:40.021114+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":688,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954956+0000","last_change":"2024-03-24T12:07:36.333159+0000","last_active":"2024-03-24T12:09:41.954956+0000","last_peered":"2024-03-24T12:09:41.954956+0000","last_clean":"2024-03-24T12:09:41.954956+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:09:41.954956+0000","last_undegraded":"2024-03-24T12:09:41.954956+0000","last_fullsized":"2024-03-24T12:09:41.954956+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:36.333044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:07:36.333044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863521+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":283,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953457+0000","last_change":"2024-03-24T12:08:36.167702+0000","last_active":"2024-03-24T12:09:41.953457+0000","last_peered":"2024-03-24T12:09:41.953457+0000","last_clean":"2024-03-24T12:09:41.953457+0000","last_became_active":"2024-03-24T12:08:36.166973+0000","last_became_peered":"2024-03-24T12:08:36.166973+0000","last_unstale":"2024-03-24T12:09:41.953457+0000","last_undegraded":"2024-03-24T12:09:41.953457+0000","last_fullsized":"2024-03-24T12:09:41.953457+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:04:14.188600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.738014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":630,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981553+0000","last_change":"2024-03-24T12:07:42.113052+0000","last_active":"2024-03-24T12:09:40.981553+0000","last_peered":"2024-03-24T12:09:40.981553+0000","last_clean":"2024-03-24T12:09:40.981553+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:09:40.981553+0000","last_undegraded":"2024-03-24T12:09:40.981553+0000","last_fullsized":"2024-03-24T12:09:40.981553+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:07:42.112569+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:07:42.112569+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":692,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.991583+0000","last_change":"2024-03-24T12:06:57.653082+0000","last_active":"2024-03-24T12:09:40.991583+0000","last_peered":"2024-03-24T12:09:40.991583+0000","last_clean":"2024-03-24T12:09:40.991583+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:09:40.991583+0000","last_undegraded":"2024-03-24T12:09:40.991583+0000","last_fullsized":"2024-03-24T12:09:40.991583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:57.653004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:06:57.653004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":439,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997615+0000","last_change":"2024-03-24T12:07:53.354298+0000","last_active":"2024-03-24T12:09:40.997615+0000","last_peered":"2024-03-24T12:09:40.997615+0000","last_clean":"2024-03-24T12:09:40.997615+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:09:40.997615+0000","last_undegraded":"2024-03-24T12:09:40.997615+0000","last_fullsized":"2024-03-24T12:09:40.997615+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:07:53.354213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":679,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997956+0000","last_change":"2024-03-24T12:08:36.167032+0000","last_active":"2024-03-24T12:09:40.997956+0000","last_peered":"2024-03-24T12:09:40.997956+0000","last_clean":"2024-03-24T12:09:40.997956+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:09:40.997956+0000","last_undegraded":"2024-03-24T12:09:40.997956+0000","last_fullsized":"2024-03-24T12:09:40.997956+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:53.809021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:04:53.809021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":730,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981661+0000","last_change":"2024-03-24T12:07:41.142569+0000","last_active":"2024-03-24T12:09:40.981661+0000","last_peered":"2024-03-24T12:09:40.981661+0000","last_clean":"2024-03-24T12:09:40.981661+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:09:40.981661+0000","last_undegraded":"2024-03-24T12:09:40.981661+0000","last_fullsized":"2024-03-24T12:09:40.981661+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:07:41.142421+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.142421+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":685,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981761+0000","last_change":"2024-03-24T12:07:32.002134+0000","last_active":"2024-03-24T12:09:40.981761+0000","last_peered":"2024-03-24T12:09:40.981761+0000","last_clean":"2024-03-24T12:09:40.981761+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:09:40.981761+0000","last_undegraded":"2024-03-24T12:09:40.981761+0000","last_fullsized":"2024-03-24T12:09:40.981761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:07:32.001718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:32.001718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":675,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.997303+0000","last_change":"2024-03-24T12:06:42.722854+0000","last_active":"2024-03-24T12:09:40.997303+0000","last_peered":"2024-03-24T12:09:40.997303+0000","last_clean":"2024-03-24T12:09:40.997303+0000","last_became_active":"2024-03-24T12:05:44.736383+0000","last_became_peered":"2024-03-24T12:05:44.736383+0000","last_unstale":"2024-03-24T12:09:40.997303+0000","last_undegraded":"2024-03-24T12:09:40.997303+0000","last_fullsized":"2024-03-24T12:09:40.997303+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.722733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.722733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":678,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.983128+0000","last_change":"2024-03-24T12:08:39.416538+0000","last_active":"2024-03-24T12:09:40.983128+0000","last_peered":"2024-03-24T12:09:40.983128+0000","last_clean":"2024-03-24T12:09:40.983128+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:09:40.983128+0000","last_undegraded":"2024-03-24T12:09:40.983128+0000","last_fullsized":"2024-03-24T12:09:40.983128+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:39.416426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:39.416426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:54.813785+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":706,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.995682+0000","last_change":"2024-03-24T12:07:40.191735+0000","last_active":"2024-03-24T12:09:40.995682+0000","last_peered":"2024-03-24T12:09:40.995682+0000","last_clean":"2024-03-24T12:09:40.995682+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:09:40.995682+0000","last_undegraded":"2024-03-24T12:09:40.995682+0000","last_fullsized":"2024-03-24T12:09:40.995682+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:40.191513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.191513+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":706,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.992041+0000","last_change":"2024-03-24T12:08:40.240985+0000","last_active":"2024-03-24T12:09:40.992041+0000","last_peered":"2024-03-24T12:09:40.992041+0000","last_clean":"2024-03-24T12:09:40.992041+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:09:40.992041+0000","last_undegraded":"2024-03-24T12:09:40.992041+0000","last_fullsized":"2024-03-24T12:09:40.992041+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.240855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.240855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:41.963342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":596,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.953240+0000","last_change":"2024-03-24T12:08:36.170723+0000","last_active":"2024-03-24T12:09:41.953240+0000","last_peered":"2024-03-24T12:09:41.953240+0000","last_clean":"2024-03-24T12:09:41.953240+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:09:41.953240+0000","last_undegraded":"2024-03-24T12:09:41.953240+0000","last_fullsized":"2024-03-24T12:09:41.953240+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:52.099551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:03:52.099551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":669,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.955859+0000","last_change":"2024-03-24T12:07:46.478165+0000","last_active":"2024-03-24T12:09:41.955859+0000","last_peered":"2024-03-24T12:09:41.955859+0000","last_clean":"2024-03-24T12:09:41.955859+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:09:41.955859+0000","last_undegraded":"2024-03-24T12:09:41.955859+0000","last_fullsized":"2024-03-24T12:09:41.955859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:46.478049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:07:46.478049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863548+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":696,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.000422+0000","last_change":"2024-03-24T12:06:56.609759+0000","last_active":"2024-03-24T12:09:41.000422+0000","last_peered":"2024-03-24T12:09:41.000422+0000","last_clean":"2024-03-24T12:09:41.000422+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:09:41.000422+0000","last_undegraded":"2024-03-24T12:09:41.000422+0000","last_fullsized":"2024-03-24T12:09:41.000422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:06:56.609621+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:06:56.609621+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":710,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989520+0000","last_change":"2024-03-24T12:09:16.215114+0000","last_active":"2024-03-24T12:09:40.989520+0000","last_peered":"2024-03-24T12:09:40.989520+0000","last_clean":"2024-03-24T12:09:40.989520+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:09:40.989520+0000","last_undegraded":"2024-03-24T12:09:40.989520+0000","last_fullsized":"2024-03-24T12:09:40.989520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:16.214991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:09:16.214991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:32.017588+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":702,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.989541+0000","last_change":"2024-03-24T12:09:38.012112+0000","last_active":"2024-03-24T12:09:40.989541+0000","last_peered":"2024-03-24T12:09:40.989541+0000","last_clean":"2024-03-24T12:09:40.989541+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:09:40.989541+0000","last_undegraded":"2024-03-24T12:09:40.989541+0000","last_fullsized":"2024-03-24T12:09:40.989541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.011992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.011992+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.011992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:44.064454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":602,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954950+0000","last_change":"2024-03-24T12:08:36.164485+0000","last_active":"2024-03-24T12:09:41.954950+0000","last_peered":"2024-03-24T12:09:41.954950+0000","last_clean":"2024-03-24T12:09:41.954950+0000","last_became_active":"2024-03-24T12:08:36.164166+0000","last_became_peered":"2024-03-24T12:08:36.164166+0000","last_unstale":"2024-03-24T12:09:41.954950+0000","last_undegraded":"2024-03-24T12:09:41.954950+0000","last_fullsized":"2024-03-24T12:09:41.954950+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:03:56.117875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:03:56.117875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:46.737752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":711,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.994635+0000","last_change":"2024-03-24T12:07:51.280566+0000","last_active":"2024-03-24T12:09:40.994635+0000","last_peered":"2024-03-24T12:09:40.994635+0000","last_clean":"2024-03-24T12:09:40.994635+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:09:40.994635+0000","last_undegraded":"2024-03-24T12:09:40.994635+0000","last_fullsized":"2024-03-24T12:09:40.994635+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:07:51.280424+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.280424+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":690,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981668+0000","last_change":"2024-03-24T12:08:40.183479+0000","last_active":"2024-03-24T12:09:40.981668+0000","last_peered":"2024-03-24T12:09:40.981668+0000","last_clean":"2024-03-24T12:09:40.981668+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:09:40.981668+0000","last_undegraded":"2024-03-24T12:09:40.981668+0000","last_fullsized":"2024-03-24T12:09:40.981668+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.183311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.183311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:55.086388+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":159,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.985878+0000","last_change":"2024-03-24T12:08:36.153334+0000","last_active":"2024-03-24T12:09:40.985878+0000","last_peered":"2024-03-24T12:09:40.985878+0000","last_clean":"2024-03-24T12:09:40.985878+0000","last_became_active":"2024-03-24T12:08:36.152717+0000","last_became_peered":"2024-03-24T12:08:36.152717+0000","last_unstale":"2024-03-24T12:09:40.985878+0000","last_undegraded":"2024-03-24T12:09:40.985878+0000","last_fullsized":"2024-03-24T12:09:40.985878+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.683700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.683700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":674,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.995651+0000","last_change":"2024-03-24T12:08:36.165633+0000","last_active":"2024-03-24T12:09:40.995651+0000","last_peered":"2024-03-24T12:09:40.995651+0000","last_clean":"2024-03-24T12:09:40.995651+0000","last_became_active":"2024-03-24T12:08:36.165334+0000","last_became_peered":"2024-03-24T12:08:36.165334+0000","last_unstale":"2024-03-24T12:09:40.995651+0000","last_undegraded":"2024-03-24T12:09:40.995651+0000","last_fullsized":"2024-03-24T12:09:40.995651+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:12.241824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:04:12.241824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":757,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.981185+0000","last_change":"2024-03-24T12:08:47.101220+0000","last_active":"2024-03-24T12:09:40.981185+0000","last_peered":"2024-03-24T12:09:40.981185+0000","last_clean":"2024-03-24T12:09:40.981185+0000","last_became_active":"2024-03-24T12:08:36.154117+0000","last_became_peered":"2024-03-24T12:08:36.154117+0000","last_unstale":"2024-03-24T12:09:40.981185+0000","last_undegraded":"2024-03-24T12:09:40.981185+0000","last_fullsized":"2024-03-24T12:09:40.981185+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:47.101082+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:08:47.101082+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:07.268703+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":682,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-24T12:09:41.954946+0000","last_change":"2024-03-24T12:07:30.248462+0000","last_active":"2024-03-24T12:09:41.954946+0000","last_peered":"2024-03-24T12:09:41.954946+0000","last_clean":"2024-03-24T12:09:41.954946+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:09:41.954946+0000","last_undegraded":"2024-03-24T12:09:41.954946+0000","last_fullsized":"2024-03-24T12:09:41.954946+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.248337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.248337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:45.863574+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":150,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.980229+0000","last_change":"2024-03-24T12:08:33.256620+0000","last_active":"2024-03-24T12:09:40.980229+0000","last_peered":"2024-03-24T12:09:40.980229+0000","last_clean":"2024-03-24T12:09:40.980229+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:09:40.980229+0000","last_undegraded":"2024-03-24T12:09:40.980229+0000","last_fullsized":"2024-03-24T12:09:40.980229+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:33.255322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:08:33.255322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:51.307702+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":667,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-24T12:09:40.980598+0000","last_change":"2024-03-24T12:07:48.265595+0000","last_active":"2024-03-24T12:09:40.980598+0000","last_peered":"2024-03-24T12:09:40.980598+0000","last_clean":"2024-03-24T12:09:40.980598+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:09:40.980598+0000","last_undegraded":"2024-03-24T12:09:40.980598+0000","last_fullsized":"2024-03-24T12:09:40.980598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:48.265295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:48.265295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2752512024,"num_objects":42001,"num_object_clones":0,"num_object_copies":126003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246829,"num_write_kb":9242497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26637,"num_bytes_recovered":1745616896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8272818176,"data_stored":8272805960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76611,"ondisk_log_size":76611,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":488689,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":80,"num_write":49,"num_write_kb":658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":483028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":243,"num_read_kb":238,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":792,"ondisk_log_size":792,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005269,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346704,"kb_used_data":706052,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92398448,"statfs":{"total":95995035648,"available":94616010752,"internally_reserved":0,"allocated":722997248,"data_stored":722343834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":656013770},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332316,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1844372,"kb_used_data":1235660,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91900780,"statfs":{"total":95995035648,"available":94106398720,"internally_reserved":0,"allocated":1265315840,"data_stored":1264647398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":623311302},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575092,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1210524,"kb_used_data":521292,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92534628,"statfs":{"total":95995035648,"available":94755459072,"internally_reserved":0,"allocated":533803008,"data_stored":533140799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738618,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2453676,"kb_used_data":1668636,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":91291476,"statfs":{"total":95995035648,"available":93482471424,"internally_reserved":0,"allocated":1708683264,"data_stored":1708011760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411568,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1100348,"kb_used_data":531880,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92644804,"statfs":{"total":95995035648,"available":94868279296,"internally_reserved":0,"allocated":544645120,"data_stored":544019649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117226,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1638568,"kb_used_data":895188,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92106584,"statfs":{"total":95995035648,"available":94317142016,"internally_reserved":0,"allocated":916672512,"data_stored":915996359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":761199047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920997,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2370780,"kb_used_data":1721812,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91374372,"statfs":{"total":95995035648,"available":93567356928,"internally_reserved":0,"allocated":1763135488,"data_stored":1762465146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455503,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1209064,"kb_used_data":955872,"kb_used_omap":7,"kb_used_meta":253176,"kb_avail":92536088,"statfs":{"total":95995035648,"available":94756954112,"internally_reserved":0,"allocated":978812928,"data_stored":978138149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":259252336},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1671696384,"data_stored":1671692312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":522715136,"data_stored":522715136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":528547840,"data_stored":528547840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":897122304,"data_stored":897122304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1756827648,"data_stored":1756823576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":961216512,"data_stored":961216512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1226838016,"data_stored":1226833944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":707854336,"data_stored":707854336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:09:48.080 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-03-24T12:09:48.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-24T12:09:48.082 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:48.138 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:48.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:48.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:48.286 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:48.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:48.292 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:48 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:09:48.292 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:48 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:09:48.292 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:48 smithi193 ceph-mon[33593]: from='client.15927 v1:172.21.15.64:0/3203605021' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:09:48.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:48 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:09:48.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:48 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:09:48.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:48 smithi064 ceph-mon[30223]: from='client.15927 v1:172.21.15.64:0/3203605021' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:09:48.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:48 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:09:48.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:48 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:09:48.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:48 smithi064 ceph-mon[42243]: from='client.15927 v1:172.21.15.64:0/3203605021' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:09:48.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:48.447 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:48.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:48.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:48.549 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:48.588 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:48.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:48.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:48.727 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:48.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:48.780 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176786 10.3 10.8db236c3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.171445+0000", 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.606210386000001, 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.97200604300000004, 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.781 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176786 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171445+0000", 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171445+0000", 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171447+0000", 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9889999999999999e-06 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171452+0000", 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9300000000000002e-06 2024-03-24T12:09:48.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171461+0000", 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8319999999999995e-06 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171469+0000", 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4770000000000003e-06 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171545+0000", 2024-03-24T12:09:48.786 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5482000000000007e-05 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171610+0000", 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5467000000000002e-05 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143441+0000", 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.97183107499999999 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.787 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143451+0000", 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7909999999999999e-06 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176787 10.6 10.40263656 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.171484+0000", 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.606170844999994, 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96683129499999998, 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.788 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176787 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171484+0000", 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171484+0000", 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.789 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171485+0000", 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5700000000000002e-07 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171495+0000", 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8139999999999995e-06 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171501+0000", 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8039999999999998e-06 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.790 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171507+0000", 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1229999999999997e-06 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171555+0000", 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7765000000000001e-05 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171562+0000", 2024-03-24T12:09:48.791 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0659999999999998e-06 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138304+0000", 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96674199900000002 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138316+0000", 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2067e-05 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.792 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176798 10.6 10.f2b64d56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.171518+0000", 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.606137329999996, 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96682983300000003, 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176798 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.793 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171518+0000", 2024-03-24T12:09:48.794 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.794 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.794 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.794 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.794 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171518+0000", 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171518+0000", 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6100000000000001e-07 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171521+0000", 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1040000000000001e-06 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171525+0000", 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7940000000000002e-06 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171531+0000", 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1800000000000004e-06 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171593+0000", 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2310000000000005e-05 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.796 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.171598+0000", 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0409999999999997e-06 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138340+0000", 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96674234699999995 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138348+0000", 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4959999999999999e-06 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.797 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176800 10.3 10.84f67093 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.177990+0000", 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.599665705000007, 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96549752, 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176800 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.798 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.177990+0000", 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.177990+0000", 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.177992+0000", 2024-03-24T12:09:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4360000000000001e-06 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.177998+0000", 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3609999999999997e-06 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.178010+0000", 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1782e-05 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.178020+0000", 2024-03-24T12:09:48.800 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0295999999999999e-05 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.178448+0000", 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00042718900000000002 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.178456+0000", 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4630000000000001e-06 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.801 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143479+0000", 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.96502270199999995 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.143487+0000", 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2910000000000007e-06 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.802 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176802 10.9 10.9ba38a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.183562+0000", 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.594093147999999, 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95528263099999999, 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176802 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.183562+0000", 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.183562+0000", 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.183565+0000", 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.382e-06 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.183572+0000", 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4620000000000001e-06 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.183589+0000", 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6886999999999999e-05 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.183598+0000", 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1810000000000007e-06 2024-03-24T12:09:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.183653+0000", 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5256000000000001e-05 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.183662+0000", 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0499999999999997e-06 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138820+0000", 2024-03-24T12:09:48.806 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.95515814499999996 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138845+0000", 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4267999999999999e-05 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176812 10.9 10.acd7f5b9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.192585+0000", 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.585070518999999, 2024-03-24T12:09:48.807 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.94634691299999996, 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176812 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.192585+0000", 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.192585+0000", 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.192587+0000", 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2649999999999999e-06 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.192593+0000", 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.823e-06 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.809 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.192607+0000", 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4267000000000001e-05 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.192615+0000", 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9999999999999996e-06 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.192664+0000", 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9051999999999997e-05 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.810 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.811 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.192672+0000", 2024-03-24T12:09:48.811 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6660000000000008e-06 2024-03-24T12:09:48.811 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.811 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.811 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.811 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138923+0000", 2024-03-24T12:09:48.811 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.94625125700000001 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138932+0000", 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5830000000000007e-06 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.812 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176825 10.6 10.f3bcb236 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.209558+0000", 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.568097507999994, 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92902620199999997, 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176825 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.813 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209558+0000", 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209558+0000", 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209560+0000", 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6520000000000002e-06 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.814 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209570+0000", 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5889999999999992e-06 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209586+0000", 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5922000000000001e-05 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209595+0000", 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.268e-06 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.815 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209687+0000", 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1711000000000003e-05 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209694+0000", 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5170000000000002e-06 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138576+0000", 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.928881124 2024-03-24T12:09:48.816 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138584+0000", 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4190000000000002e-06 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176826 10.9 10.8eb7249 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.209988+0000", 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.567667396000004, 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.928974046, 2024-03-24T12:09:48.817 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176826 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209988+0000", 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.818 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209988+0000", 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209989+0000", 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4600000000000003e-07 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209992+0000", 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1140000000000002e-06 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.819 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.209998+0000", 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0249999999999999e-06 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.210004+0000", 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1149999999999999e-06 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.210041+0000", 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6621000000000001e-05 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.820 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.210045+0000", 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.442e-06 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138954+0000", 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.92890893299999999 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138962+0000", 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9500000000000001e-06 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:176830 10.6 10.569762d6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:35.223739+0000", 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.553916463999997, 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91487453100000005, 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 176830 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.822 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.223739+0000", 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.223739+0000", 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.223740+0000", 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3999999999999999e-06 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.823 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.223745+0000", 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9049999999999996e-06 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.223758+0000", 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2546e-05 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.223765+0000", 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6190000000000002e-06 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.824 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.223821+0000", 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5328000000000003e-05 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:35.223827+0000", 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8329999999999996e-06 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138606+0000", 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91477826699999998 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.825 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:36.138613+0000", 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6329999999999996e-06 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222735 10.6 10:6ffba019:::benchmark_data_smithi064_174908_object78318:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:17.995697+0000", 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.781958234000001, 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.006408671, 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.826 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222735 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.995697+0000", 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.995697+0000", 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.827 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.995699+0000", 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4080000000000001e-06 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.995705+0000", 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3460000000000001e-06 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.995714+0000", 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5200000000000003e-06 2024-03-24T12:09:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.995720+0000", 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.003e-06 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.995759+0000", 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9147000000000003e-05 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.995765+0000", 2024-03-24T12:09:48.829 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.823e-06 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.992835+0000", 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.99706950699999997 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.992960+0000", 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000125155 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.993329+0000", 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00036927599999999998 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.001946+0000", 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086165440000000003 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.001958+0000", 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2584e-05 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.831 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002020+0000", 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1772000000000004e-05 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002069+0000", 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9187999999999997e-05 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002106+0000", 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6397999999999999e-05 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.832 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222736 10.6 10:6390f651:::benchmark_data_smithi064_174908_object78319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:17.996952+0000", 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.780702846000001, 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.006038421, 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222736 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.833 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.996952+0000", 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.996952+0000", 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.996954+0000", 2024-03-24T12:09:48.834 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3510000000000001e-06 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.996959+0000", 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7520000000000004e-06 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.996967+0000", 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7499999999999992e-06 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.835 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.996975+0000", 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9209999999999992e-06 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.997033+0000", 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7834999999999998e-05 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:17.997041+0000", 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7179999999999996e-06 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.996888+0000", 2024-03-24T12:09:48.836 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.99984743899999995 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.997002+0000", 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000114304 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.997143+0000", 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014090700000000001 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002115+0000", 2024-03-24T12:09:48.837 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004971479 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002144+0000", 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9210999999999999e-05 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002897+0000", 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00075302799999999999 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.838 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002947+0000", 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0285999999999999e-05 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002991+0000", 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3439999999999997e-05 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222759 10.6 10:6e1bfa2f:::benchmark_data_smithi064_174908_object78342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.095285+0000", 2024-03-24T12:09:48.839 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.682370755000001, 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.91463287400000004, 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222759 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.095285+0000", 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.840 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.095285+0000", 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.095286+0000", 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.905e-06 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.095291+0000", 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8130000000000003e-06 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.841 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.095313+0000", 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1387999999999999e-05 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.095321+0000", 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9489999999999996e-06 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.100185+0000", 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048643059999999997 2024-03-24T12:09:48.842 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.100199+0000", 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3671e-05 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.999356+0000", 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.89915772199999999 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.999509+0000", 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015311 2024-03-24T12:09:48.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.999644+0000", 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000134726 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.002859+0000", 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032153139999999999 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.005589+0000", 2024-03-24T12:09:48.844 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0027296120000000002 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.009674+0000", 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040854719999999997 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.009865+0000", 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000190531 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.845 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.009917+0000", 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2355000000000003e-05 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222781 10.6 10:630c7097:::benchmark_data_smithi064_174908_object78364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.113885+0000", 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.663770712000002, 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.90443231899999998, 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.846 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222781 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.113885+0000", 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.113885+0000", 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.847 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.113887+0000", 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.119e-06 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.113892+0000", 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5389999999999998e-06 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.113908+0000", 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6052000000000001e-05 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.848 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.113916+0000", 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0299999999999994e-06 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.113972+0000", 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6017000000000003e-05 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.113982+0000", 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5289999999999997e-06 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.849 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.003017+0000", 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88903557600000005 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.003394+0000", 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000376215 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.003517+0000", 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012379200000000001 2024-03-24T12:09:48.850 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.009659+0000", 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0061417909999999997 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.012572+0000", 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0029126769999999998 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.851 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018226+0000", 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0056543950000000004 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018274+0000", 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7976999999999998e-05 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018317+0000", 2024-03-24T12:09:48.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.261e-05 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222796 10.6 10:64efd400:::benchmark_data_smithi064_174908_object78379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.125234+0000", 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.652421269000001, 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.89322438400000004, 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.854 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222796 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.125234+0000", 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.125234+0000", 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.855 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.125236+0000", 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.216e-06 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.125241+0000", 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1649999999999997e-06 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.125256+0000", 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4511999999999999e-05 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.856 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.125264+0000", 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6760000000000007e-06 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.125340+0000", 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6076999999999997e-05 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.125348+0000", 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0989999999999999e-06 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.857 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.005624+0000", 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88027601899999997 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.005737+0000", 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000113614 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.005875+0000", 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013755899999999999 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.858 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.012577+0000", 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0067021060000000002 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018184+0000", 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0056069819999999999 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018387+0000", 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000203067 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018427+0000", 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9654999999999999e-05 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018458+0000", 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1637000000000002e-05 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222804 10.6 10:68997e4f:::benchmark_data_smithi064_174908_object78387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.130768+0000", 2024-03-24T12:09:48.860 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.646887277000001, 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88838960600000005, 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222804 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.130768+0000", 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.130768+0000", 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.130770+0000", 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2350000000000002e-06 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.130778+0000", 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6070000000000002e-06 2024-03-24T12:09:48.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.130804+0000", 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5766999999999999e-05 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.130812+0000", 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6330000000000002e-06 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.130935+0000", 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012263000000000001 2024-03-24T12:09:48.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.130943+0000", 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3680000000000002e-06 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.007632+0000", 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.87668837300000002 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.007736+0000", 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000104065 2024-03-24T12:09:48.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.007886+0000", 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015070599999999999 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.015811+0000", 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0079245670000000004 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018353+0000", 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025422050000000001 2024-03-24T12:09:48.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019072+0000", 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00071892099999999997 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019120+0000", 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8195999999999997e-05 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019158+0000", 2024-03-24T12:09:48.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7333000000000003e-05 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222818 10.6 10:68d70787:::benchmark_data_smithi064_174908_object78401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.146069+0000", 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.631586238000001, 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88739875800000001, 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.867 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222818 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.146069+0000", 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.146069+0000", 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.868 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.146071+0000", 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.55e-06 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.146076+0000", 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7679999999999997e-06 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.146085+0000", 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0059999999999998e-06 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.869 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.146092+0000", 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2539999999999998e-06 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.146188+0000", 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6739999999999999e-05 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.146197+0000", 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6209999999999993e-06 2024-03-24T12:09:48.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.009955+0000", 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.86375770100000004 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.010065+0000", 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000110331 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.010180+0000", 2024-03-24T12:09:48.871 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011498900000000001 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018145+0000", 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0079651979999999997 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019027+0000", 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00088157999999999997 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033374+0000", 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014347718000000001 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033426+0000", 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1255000000000003e-05 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033468+0000", 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2046999999999999e-05 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222825 10.6 10:63fd70fc:::benchmark_data_smithi064_174908_object78408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.151373+0000", 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.626281967000001, 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.88227974300000001, 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222825 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.874 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.151373+0000", 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.151373+0000", 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.151375+0000", 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1969999999999999e-06 2024-03-24T12:09:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.151383+0000", 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4200000000000001e-06 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.151399+0000", 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6047000000000001e-05 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.151407+0000", 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7829999999999999e-06 2024-03-24T12:09:48.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.151463+0000", 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6127e-05 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.151472+0000", 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3840000000000003e-06 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.012605+0000", 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.86113300199999998 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.012709+0000", 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010340099999999999 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.012822+0000", 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000113461 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.018195+0000", 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0053727369999999998 2024-03-24T12:09:48.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019201+0000", 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0010061099999999999 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033574+0000", 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014373115000000001 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033619+0000", 2024-03-24T12:09:48.879 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4972000000000002e-05 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033653+0000", 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3986999999999998e-05 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222833 10.6 10:687ca75a:::benchmark_data_smithi064_174908_object78416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.159761+0000", 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.617894552999999, 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.87405046600000003, 2024-03-24T12:09:48.880 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222833 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.159761+0000", 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.881 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.159761+0000", 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.159763+0000", 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1749999999999999e-06 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.159768+0000", 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1250000000000001e-06 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.882 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.159780+0000", 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1527e-05 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.159796+0000", 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6203999999999999e-05 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.159854+0000", 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7763999999999998e-05 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.159877+0000", 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3889000000000001e-05 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.015841+0000", 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85596404999999998 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.015944+0000", 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000102231 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.884 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.016055+0000", 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011087 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019166+0000", 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031114649999999999 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.025313+0000", 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0061465019999999999 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.885 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033705+0000", 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083926150000000008 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033764+0000", 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8480999999999998e-05 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033811+0000", 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7568000000000003e-05 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.886 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222839 10.6 10:6bc9dbc4:::benchmark_data_smithi064_174908_object78422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.167146+0000", 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.610509689000001, 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.86680270000000004, 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222839 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.887 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.167146+0000", 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.167146+0000", 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.167148+0000", 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5440000000000001e-06 2024-03-24T12:09:48.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.167153+0000", 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7180000000000004e-06 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.167166+0000", 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2697000000000001e-05 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.167177+0000", 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1195999999999999e-05 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.167224+0000", 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7729999999999999e-05 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.167233+0000", 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0669999999999992e-06 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019229+0000", 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.85199672199999998 2024-03-24T12:09:48.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019330+0000", 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000100589 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.019437+0000", 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107261 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.891 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.025355+0000", 2024-03-24T12:09:48.892 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0059177769999999999 2024-03-24T12:09:48.892 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.892 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.892 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.892 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.029843+0000", 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0044881879999999997 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033868+0000", 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040247659999999999 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033915+0000", 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6816e-05 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.893 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033948+0000", 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3628999999999999e-05 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:222847 10.6 10:6dc2ffea:::benchmark_data_smithi064_174908_object78430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:18.186780+0000", 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.590875613000001, 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.84882901099999997, 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:09:48.894 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 222847 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.186780+0000", 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.186780+0000", 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:09:48.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.186781+0000", 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.697e-06 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.186785+0000", 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.963e-06 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.186806+0000", 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0794000000000001e-05 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.186812+0000", 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1330000000000004e-06 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.186851+0000", 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9221000000000003e-05 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:18.186858+0000", 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1410000000000002e-06 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.897 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.023102+0000", 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.83624466399999997 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.023204+0000", 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010187999999999999 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.023325+0000", 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012108 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.029848+0000", 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0065232470000000002 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.033334+0000", 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034851869999999998 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.035521+0000", 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021876679999999998 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.899 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.035571+0000", 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9957000000000003e-05 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:19.035609+0000", 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7379000000000002e-05 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:09:48.900 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:09:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:48.934 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:48.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:49.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:49.070 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:49.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:49 smithi193 ceph-mon[33593]: pgmap v805: 168 pgs: 168 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.53k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-24T12:09:49.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:49.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:49.210 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:49.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:49.261 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:49.266 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:49.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:49.350 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:49.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:49.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:49 smithi064 ceph-mon[30223]: pgmap v805: 168 pgs: 168 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.53k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-24T12:09:49.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:49.407 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:49 smithi064 ceph-mon[42243]: pgmap v805: 168 pgs: 168 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.53k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-24T12:09:49.407 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:49.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:49.497 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:49.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:49.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:49.623 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:49.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:49.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:49.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:49.752 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:49.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:49.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:49.889 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:49.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:49.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:09:49.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:50.030 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:50.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:50.095 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:09:50.132 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:50.161 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:50.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:50.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:50.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:50 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3396778177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:50.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:50 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3396778177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:50.308 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:50.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:50.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:50 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3396778177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:09:50.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:50.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:50.550 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:50.588 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:50.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:50.689 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:50.725 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:50.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:50.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:50.858 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:50.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:50.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:50.995 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:50.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:51.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:51.130 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:51.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:51.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:51.266 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:51.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:51 smithi193 ceph-mon[33593]: pgmap v806: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T12:09:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:51 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:51 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3396778177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:51 smithi193 ceph-mon[33593]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T12:09:51.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:51.405 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:51.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:51.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:51.541 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:51.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:51 smithi064 ceph-mon[30223]: pgmap v806: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T12:09:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:51 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:51 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3396778177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:51 smithi064 ceph-mon[30223]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T12:09:51.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:51 smithi064 ceph-mon[42243]: pgmap v806: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T12:09:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:51 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:09:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:51 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3396778177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:09:51.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:51 smithi064 ceph-mon[42243]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T12:09:51.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:51.684 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:51.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:51.786 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:51.822 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:51.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:51.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:51.962 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:51.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:52.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:52.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:52.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:52.251 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:52.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:52.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:52.358 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:09:52.358 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:09:52.358 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[30223]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T12:09:52.358 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:09:52.358 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:09:52.359 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:09:52.359 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:09:52.359 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[42243]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T12:09:52.359 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:09:52.359 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:52 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:09:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:52.399 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:52.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:52.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:52.535 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:52.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:52 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:09:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:52 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:09:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:52 smithi193 ceph-mon[33593]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T12:09:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:52 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:09:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:52 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:09:52.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:52.670 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:52.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:52.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:52.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:52.807 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:52.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:52.908 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:52.941 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:52.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:53.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:53.079 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:53.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:53.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:53.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:53.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:53.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:53.359 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:53.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:53.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:53.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:53.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:53 smithi193 ceph-mon[33593]: pgmap v809: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:09:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:53 smithi193 ceph-mon[33593]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T12:09:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:53.527 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:53.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:53.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:53 smithi064 ceph-mon[30223]: pgmap v809: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:09:53.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:53 smithi064 ceph-mon[30223]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T12:09:53.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:53 smithi064 ceph-mon[42243]: pgmap v809: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:09:53.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:53 smithi064 ceph-mon[42243]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T12:09:53.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:53.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:53.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:53.804 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:53.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:53.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:09:53.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:53.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:53.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:54.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:54.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:54.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:54.275 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:54.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:54.377 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:54.417 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:54.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:54.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:54.552 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:54.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:54.654 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:54.695 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:54.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:54.796 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:54.833 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:54.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:54.934 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:54.967 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:54.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:09:55.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:55.100 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:55.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:55.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:55.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:55.236 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:55.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:55.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:55 smithi064 ceph-mon[30223]: pgmap v811: 168 pgs: 168 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:09:55.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:55 smithi064 ceph-mon[42243]: pgmap v811: 168 pgs: 168 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:09:55.336 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:55.373 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:55.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:55.474 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:55.509 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:55.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:55 smithi193 ceph-mon[33593]: pgmap v811: 168 pgs: 168 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:09:55.610 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:55.649 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:55.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:55.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:55.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:55.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:55.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:55.910 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:09:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:55.935 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:55.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:55.949 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:09:55.988 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:09:56.033 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:09:56.036 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:56.072 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:56.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:56.078 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:09:56.121 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:09:56.159 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:09:56.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:56.202 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:09:56.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:56 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:09:56.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:56 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:09:56.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:56 smithi064 ceph-mon[30223]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T12:09:56.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:56 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/678599549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:56.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:56 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:09:56.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:56 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:09:56.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:56 smithi064 ceph-mon[42243]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T12:09:56.203 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:56 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/678599549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:09:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:09:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:09:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:09:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:09:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:09:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:09:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:09:56.213 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:56.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:56.314 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:56.349 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:56.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:56.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:56.491 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:56.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:56 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:09:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:56 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:09:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:56 smithi193 ceph-mon[33593]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T12:09:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:56 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/678599549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:09:56.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:56.629 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:56.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:56.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:56.763 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:56.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:56.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:56.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:56.902 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:56.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:57.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:57.038 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:57.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:57.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:57.176 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:57.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:57.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:57.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:57.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:57.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:57.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:57 smithi064 ceph-mon[30223]: pgmap v813: 168 pgs: 168 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:09:57.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:57 smithi064 ceph-mon[42243]: pgmap v813: 168 pgs: 168 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:09:57.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:57.452 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:57.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:57.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:57.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:57 smithi193 ceph-mon[33593]: pgmap v813: 168 pgs: 168 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:09:57.590 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:57.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:57.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:57.723 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:57.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:57.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:09:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:09:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:57.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:57.862 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:57.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:57.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:57.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:58.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:58.028 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:09:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:09:58.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:09:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:58.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:09:58.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:58.272 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:58.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:58.374 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:58.406 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:58.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:58.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:58.539 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:58.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:58.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:58.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:58.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:09:58.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:58.641 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:58.671 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:58.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:58.773 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:58.809 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:58.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:58.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:09:58.953 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:58.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:09:59.055 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:59.088 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:59.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:09:59.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:09:59.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:59.227 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:59.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:09:59.328 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:09:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:59.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:09:59.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:09:59.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:59 smithi064 ceph-mon[30223]: pgmap v814: 168 pgs: 168 active+clean; 1.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-03-24T12:09:59.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:09:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:59.470 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:59 smithi064 ceph-mon[42243]: pgmap v814: 168 pgs: 168 active+clean; 1.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-03-24T12:09:59.470 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:09:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:59.500 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:59.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:09:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:59 smithi193 ceph-mon[33593]: pgmap v814: 168 pgs: 168 active+clean; 1.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-03-24T12:09:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:09:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:59.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:09:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:09:59.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:09:59.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:09:59.778 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:59.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:09:59.880 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:09:59.916 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:09:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:59.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:00.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:00.057 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:00.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:00.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:00.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:00.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:00.334 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:00.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:00.436 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:00.469 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:00.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:00.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:00 smithi064 ceph-mon[30223]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T12:10:00.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:00 smithi064 ceph-mon[30223]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-24T12:10:00.511 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:00 smithi064 ceph-mon[42243]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T12:10:00.511 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:00 smithi064 ceph-mon[42243]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-24T12:10:00.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:00 smithi193 ceph-mon[33593]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T12:10:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:00 smithi193 ceph-mon[33593]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-24T12:10:00.607 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:00.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:00.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:00.749 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:00.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:00.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:00.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:00.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:00.966 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:10:00.966 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:10:00.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:01.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:01.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:01.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:01.172 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:01.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:01.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:01.318 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:01.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:01.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:01.452 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:01.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:01.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:10:01.534 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:01 smithi064 ceph-mon[30223]: pgmap v815: 168 pgs: 168 active+clean; 1.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-24T12:10:01.534 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:01 smithi064 ceph-mon[42243]: pgmap v815: 168 pgs: 168 active+clean; 1.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-24T12:10:01.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:01 smithi193 ceph-mon[33593]: pgmap v815: 168 pgs: 168 active+clean; 1.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-24T12:10:01.593 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:01.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:01.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:01.730 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:01.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:01.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:10:01.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:01.864 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:01.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:01.966 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:02.010 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:02.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:02.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:02.148 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:02.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 1 pg_num to 11 2024-03-24T12:10:02.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:02.296 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:02.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:02.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2034769412' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:10:02.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:02 smithi064 ceph-mon[30223]: from='client.35777 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:10:02.323 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:02 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2034769412' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:10:02.323 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:02 smithi064 ceph-mon[42243]: from='client.35777 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:10:02.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:02.435 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:02.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:02.545 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:02.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:02.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:02 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2034769412' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:10:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:02 smithi193 ceph-mon[33593]: from='client.35777 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:10:02.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:02.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:02.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:02.872 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:02.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:02.974 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:03.009 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:03.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:03.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:03.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:03.244 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:03.282 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:03.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:03 smithi064 ceph-mon[30223]: pgmap v816: 168 pgs: 168 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 544 B/s rd, 0 B/s wr, 928 op/s 2024-03-24T12:10:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:03 smithi064 ceph-mon[30223]: from='client.35777 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T12:10:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:03 smithi064 ceph-mon[30223]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T12:10:03.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:03 smithi064 ceph-mon[42243]: pgmap v816: 168 pgs: 168 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 544 B/s rd, 0 B/s wr, 928 op/s 2024-03-24T12:10:03.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:03 smithi064 ceph-mon[42243]: from='client.35777 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T12:10:03.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:03 smithi064 ceph-mon[42243]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T12:10:03.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:03.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:03.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:03.558 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:03.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:03 smithi193 ceph-mon[33593]: pgmap v816: 168 pgs: 168 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 544 B/s rd, 0 B/s wr, 928 op/s 2024-03-24T12:10:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:03 smithi193 ceph-mon[33593]: from='client.35777 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T12:10:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:03 smithi193 ceph-mon[33593]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T12:10:03.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:03.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:03.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:03.842 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:03.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:03.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:03.875 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:03.882 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:03.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:03.995 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:03.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:04.096 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:04.131 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:04.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:04.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:04.279 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:04.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:04.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:04.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:04.438 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:04 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T12:10:04.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:04 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T12:10:04.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:04.553 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:04.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:04 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T12:10:04.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:04.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:04.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:04.824 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:04.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:04.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:04.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:04.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:05.074 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:05.109 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:05.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:05.211 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:05.252 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:05.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:05.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:05 smithi064 ceph-mon[30223]: pgmap v818: 168 pgs: 168 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:10:05.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:05 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T12:10:05.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:05 smithi064 ceph-mon[30223]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T12:10:05.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:05 smithi064 ceph-mon[42243]: pgmap v818: 168 pgs: 168 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:10:05.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:05 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T12:10:05.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:05 smithi064 ceph-mon[42243]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T12:10:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:05 smithi193 ceph-mon[33593]: pgmap v818: 168 pgs: 168 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T12:10:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:05 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T12:10:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:05 smithi193 ceph-mon[33593]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T12:10:05.354 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:05.392 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:05.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:05.493 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:05.528 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:05.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:05.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:05.662 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:05.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:05.764 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:05.801 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:05.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:05.903 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:05.936 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:05.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:06.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:06.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:06.075 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:06.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:06.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:06.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:06.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:06.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:06.350 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:06.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:06.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:06.489 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:06.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:06.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:06 smithi064 ceph-mon[30223]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T12:10:06.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:06 smithi064 ceph-mon[30223]: pgmap v821: 168 pgs: 1 unknown, 167 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:10:06.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:06 smithi064 ceph-mon[42243]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T12:10:06.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:06 smithi064 ceph-mon[42243]: pgmap v821: 168 pgs: 1 unknown, 167 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:10:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:06 smithi193 ceph-mon[33593]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T12:10:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:06 smithi193 ceph-mon[33593]: pgmap v821: 168 pgs: 1 unknown, 167 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:10:06.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:06.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:06.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:06.776 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:06.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:06.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:06.916 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:06.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:07.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:07.056 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:07.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:07.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:07.198 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:07.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:07.301 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:07.335 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:07.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:07.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:07.476 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:07.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:07.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:07 smithi064 ceph-mon[30223]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T12:10:07.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:07 smithi064 ceph-mon[30223]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T12:10:07.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:07 smithi064 ceph-mon[42243]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T12:10:07.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:07 smithi064 ceph-mon[42243]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T12:10:07.522 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:10:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:10:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:07.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:07 smithi193 ceph-mon[33593]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T12:10:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:07 smithi193 ceph-mon[33593]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T12:10:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:07.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:07.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:07.752 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:07.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:07.853 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:07.889 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:07.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:07.970 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:10:07.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:08.030 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:08.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:08.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:10:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:10:08.134 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:10:08.134 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:10:08.134 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:08.168 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:08.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:08.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3345904275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:10:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[30223]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:10:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[30223]: pgmap v824: 168 pgs: 5 active+undersized+remapped, 1 unknown, 162 active+clean; 746 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.67k op/s; 100 KiB/s, 0 objects/s recovering 2024-03-24T12:10:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[30223]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:10:08.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[30223]: osdmap e387: 8 total, 8 up, 8 in 2024-03-24T12:10:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3345904275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:10:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[42243]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:10:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[42243]: pgmap v824: 168 pgs: 5 active+undersized+remapped, 1 unknown, 162 active+clean; 746 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.67k op/s; 100 KiB/s, 0 objects/s recovering 2024-03-24T12:10:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[42243]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:10:08.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:08 smithi064 ceph-mon[42243]: osdmap e387: 8 total, 8 up, 8 in 2024-03-24T12:10:08.305 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:08.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:08.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:08.407 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:08.442 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:08.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:08.455 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:10:08.455 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:10:08.455 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-24T12:10:08.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:10:08.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:08.578 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:08.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:08 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3345904275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:10:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:08 smithi193 ceph-mon[33593]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:10:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:08 smithi193 ceph-mon[33593]: pgmap v824: 168 pgs: 5 active+undersized+remapped, 1 unknown, 162 active+clean; 746 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.67k op/s; 100 KiB/s, 0 objects/s recovering 2024-03-24T12:10:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:08 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:08 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:08 smithi193 ceph-mon[33593]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:10:08.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:08 smithi193 ceph-mon[33593]: osdmap e387: 8 total, 8 up, 8 in 2024-03-24T12:10:08.679 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:08.708 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:08.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:08.809 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:08.835 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:08.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:08.937 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:08.971 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:08.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:09.073 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:09.113 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:09.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:09.214 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:09.252 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:09.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:09.355 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:09.387 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:09.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:09.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:09.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:09.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:09.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[30223]: 1.12 scrub starts 2024-03-24T12:10:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[30223]: 1.12 scrub ok 2024-03-24T12:10:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T12:10:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T12:10:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[30223]: 1.c scrub starts 2024-03-24T12:10:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[30223]: 1.c scrub ok 2024-03-24T12:10:09.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[30223]: 1.11 scrub starts 2024-03-24T12:10:09.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[30223]: 1.11 scrub ok 2024-03-24T12:10:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[42243]: 1.12 scrub starts 2024-03-24T12:10:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[42243]: 1.12 scrub ok 2024-03-24T12:10:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T12:10:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T12:10:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[42243]: 1.c scrub starts 2024-03-24T12:10:09.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[42243]: 1.c scrub ok 2024-03-24T12:10:09.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[42243]: 1.11 scrub starts 2024-03-24T12:10:09.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:09 smithi064 ceph-mon[42243]: 1.11 scrub ok 2024-03-24T12:10:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:09 smithi193 ceph-mon[33593]: 1.12 scrub starts 2024-03-24T12:10:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:09 smithi193 ceph-mon[33593]: 1.12 scrub ok 2024-03-24T12:10:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:09 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T12:10:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:09 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T12:10:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:09 smithi193 ceph-mon[33593]: 1.c scrub starts 2024-03-24T12:10:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:09 smithi193 ceph-mon[33593]: 1.c scrub ok 2024-03-24T12:10:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:09 smithi193 ceph-mon[33593]: 1.11 scrub starts 2024-03-24T12:10:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:09 smithi193 ceph-mon[33593]: 1.11 scrub ok 2024-03-24T12:10:09.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:09.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:10:09.658 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:09.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:09.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:09.808 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:09.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:09.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:09.949 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:09.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:09.950 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:10.014 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:10:10.051 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:10.056 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:10:10.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:10.086 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:10.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:10.095 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:10:10.137 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:10:10.176 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:10:10.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:10.216 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:10:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:10.234 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:10.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:10.262 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:10:10.306 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:10:10.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:10:10.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:10:10.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:10:10.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:10:10.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:10:10.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:10:10.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:10:10.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:10:10.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:10.382 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:10.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:10.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:10.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:10.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[30223]: 1.13 scrub starts 2024-03-24T12:10:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[30223]: 1.13 scrub ok 2024-03-24T12:10:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[30223]: 1.10 scrub starts 2024-03-24T12:10:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[30223]: 1.10 scrub ok 2024-03-24T12:10:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[30223]: pgmap v826: 168 pgs: 5 active+undersized+remapped, 163 active+clean; 616 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 26 KiB/s wr, 1.57k op/s; 94 KiB/s, 0 objects/s recovering 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3946216820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[42243]: 1.13 scrub starts 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[42243]: 1.13 scrub ok 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[42243]: 1.10 scrub starts 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[42243]: 1.10 scrub ok 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[42243]: pgmap v826: 168 pgs: 5 active+undersized+remapped, 163 active+clean; 616 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 26 KiB/s wr, 1.57k op/s; 94 KiB/s, 0 objects/s recovering 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T12:10:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3946216820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:10 smithi193 ceph-mon[33593]: 1.13 scrub starts 2024-03-24T12:10:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:10 smithi193 ceph-mon[33593]: 1.13 scrub ok 2024-03-24T12:10:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:10 smithi193 ceph-mon[33593]: 1.10 scrub starts 2024-03-24T12:10:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:10 smithi193 ceph-mon[33593]: 1.10 scrub ok 2024-03-24T12:10:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:10 smithi193 ceph-mon[33593]: pgmap v826: 168 pgs: 5 active+undersized+remapped, 163 active+clean; 616 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 26 KiB/s wr, 1.57k op/s; 94 KiB/s, 0 objects/s recovering 2024-03-24T12:10:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:10 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T12:10:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3946216820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:10.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:10.658 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:10.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:10.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:10.798 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:10.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:10.901 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:10.938 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:10.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:11.040 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:11.076 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:11.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:11.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:10:11.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:11.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:11.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:11.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:11.353 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:11.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:11.455 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:11.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:11.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T12:10:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:10:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:10:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 1.8 scrub starts 2024-03-24T12:10:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 1.8 scrub ok 2024-03-24T12:10:11.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 1.e scrub starts 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 1.e scrub ok 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 6.11 scrub starts 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[30223]: 6.11 scrub ok 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 1.8 scrub starts 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 1.8 scrub ok 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 1.e scrub starts 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 1.e scrub ok 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 6.11 scrub starts 2024-03-24T12:10:11.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:11 smithi064 ceph-mon[42243]: 6.11 scrub ok 2024-03-24T12:10:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T12:10:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:10:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:10:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 1.8 scrub starts 2024-03-24T12:10:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 1.8 scrub ok 2024-03-24T12:10:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 1.e scrub starts 2024-03-24T12:10:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 1.e scrub ok 2024-03-24T12:10:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 6.11 scrub starts 2024-03-24T12:10:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:11 smithi193 ceph-mon[33593]: 6.11 scrub ok 2024-03-24T12:10:11.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:11.611 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:11.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:11.713 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:11.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:11.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:11.881 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:11.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:11.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:12.018 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:12.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:12.120 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:12.154 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:12.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:12.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:12.297 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:12.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:12.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:12.436 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:12.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:12.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:12.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: 1.1 scrub starts 2024-03-24T12:10:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: 1.1 scrub ok 2024-03-24T12:10:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:10:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:10:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T12:10:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T12:10:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T12:10:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T12:10:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: pgmap v827: 168 pgs: 4 active+undersized+remapped, 164 active+clean; 616 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 KiB/s wr, 1.12k op/s; 66 KiB/s, 0 objects/s recovering 2024-03-24T12:10:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T12:10:12.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:12.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:12.718 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:12.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: 1.1 scrub starts 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: 1.1 scrub ok 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: pgmap v827: 168 pgs: 4 active+undersized+remapped, 164 active+clean; 616 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 KiB/s wr, 1.12k op/s; 66 KiB/s, 0 objects/s recovering 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T12:10:12.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: 1.1 scrub starts 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: 1.1 scrub ok 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: pgmap v827: 168 pgs: 4 active+undersized+remapped, 164 active+clean; 616 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 KiB/s wr, 1.12k op/s; 66 KiB/s, 0 objects/s recovering 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T12:10:12.794 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:12.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:12.864 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:12.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:12.966 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:13.000 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:13.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:13.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:13.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:13.237 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:13.274 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:13.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:13.370 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:13.375 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:13.408 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:13.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:13.436 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:13.510 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:13.550 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[30223]: 1.a scrub starts 2024-03-24T12:10:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[30223]: 1.a scrub ok 2024-03-24T12:10:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-24T12:10:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[30223]: osdmap e388: 8 total, 8 up, 8 in 2024-03-24T12:10:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T12:10:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T12:10:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T12:10:13.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:13.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[42243]: 1.a scrub starts 2024-03-24T12:10:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[42243]: 1.a scrub ok 2024-03-24T12:10:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-24T12:10:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[42243]: osdmap e388: 8 total, 8 up, 8 in 2024-03-24T12:10:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T12:10:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T12:10:13.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:13 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T12:10:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:13 smithi193 ceph-mon[33593]: 1.a scrub starts 2024-03-24T12:10:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:13 smithi193 ceph-mon[33593]: 1.a scrub ok 2024-03-24T12:10:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:13 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-24T12:10:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:13 smithi193 ceph-mon[33593]: osdmap e388: 8 total, 8 up, 8 in 2024-03-24T12:10:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:13 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T12:10:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:13 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T12:10:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:13 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T12:10:13.653 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:13.687 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:13.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:13.788 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:13.828 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:13.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:13.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:13.964 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:13.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:14.066 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:14.103 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:14.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:14.205 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:14.241 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:14.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:14.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:14.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:14.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:14.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:14.519 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:14.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:14.620 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:14.655 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:14.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:14.757 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:14.793 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:14.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:14.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T12:10:14.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:10:14.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:10:14.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: 2.16 scrub starts 2024-03-24T12:10:14.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: 2.16 scrub ok 2024-03-24T12:10:14.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: osdmap e389: 8 total, 8 up, 8 in 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: pgmap v830: 168 pgs: 168 active+clean; 350 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: 6.13 scrub starts 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[30223]: 6.13 scrub ok 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: 2.16 scrub starts 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: 2.16 scrub ok 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: osdmap e389: 8 total, 8 up, 8 in 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T12:10:14.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: pgmap v830: 168 pgs: 168 active+clean; 350 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-03-24T12:10:14.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: 6.13 scrub starts 2024-03-24T12:10:14.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:14 smithi064 ceph-mon[42243]: 6.13 scrub ok 2024-03-24T12:10:14.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: 2.16 scrub starts 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: 2.16 scrub ok 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: osdmap e389: 8 total, 8 up, 8 in 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: pgmap v830: 168 pgs: 168 active+clean; 350 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: 6.13 scrub starts 2024-03-24T12:10:14.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:14 smithi193 ceph-mon[33593]: 6.13 scrub ok 2024-03-24T12:10:14.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:14.928 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:14.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:15.029 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:15.063 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:15.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:15.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:15.200 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:15.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:15.206 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Removed 144414 objects 2024-03-24T12:10:15.206 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Clean up completed and total clean up time :125.443 2024-03-24T12:10:15.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:15.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:15.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:15.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:10:15.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:15.452 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:15.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:15.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:15.579 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:15.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:15.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:15.704 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:15.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:15.722 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T12:10:15.722 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 6.19 scrub starts 2024-03-24T12:10:15.722 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 6.19 scrub ok 2024-03-24T12:10:15.722 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T12:10:15.722 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T12:10:15.722 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: osdmap e390: 8 total, 8 up, 8 in 2024-03-24T12:10:15.722 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 6.19 scrub starts 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 6.19 scrub ok 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: osdmap e390: 8 total, 8 up, 8 in 2024-03-24T12:10:15.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T12:10:15.724 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T12:10:15.724 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T12:10:15.724 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T12:10:15.724 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T12:10:15.724 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:15 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T12:10:15.803 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:15.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:15.831 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:15.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:15.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T12:10:15.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 6.19 scrub starts 2024-03-24T12:10:15.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 6.19 scrub ok 2024-03-24T12:10:15.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T12:10:15.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T12:10:15.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: osdmap e390: 8 total, 8 up, 8 in 2024-03-24T12:10:15.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T12:10:15.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T12:10:15.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T12:10:15.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T12:10:15.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T12:10:15.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:15 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T12:10:15.932 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:15.957 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:15.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:16.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:16.085 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:16.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:16.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:16.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:16.315 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:16.341 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:16.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:16.442 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:16.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:16.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:16.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:16.594 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:16.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:16.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:16.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:16.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T12:10:16.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T12:10:16.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[42243]: osdmap e391: 8 total, 8 up, 8 in 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[42243]: pgmap v833: 167 pgs: 167 active+clean; 224 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.55k op/s 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[30223]: osdmap e391: 8 total, 8 up, 8 in 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[30223]: pgmap v833: 167 pgs: 167 active+clean; 224 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.55k op/s 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:16 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T12:10:16.822 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:16.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:16 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T12:10:16.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:16 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T12:10:16.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:16 smithi193 ceph-mon[33593]: osdmap e391: 8 total, 8 up, 8 in 2024-03-24T12:10:16.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:16 smithi193 ceph-mon[33593]: pgmap v833: 167 pgs: 167 active+clean; 224 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.55k op/s 2024-03-24T12:10:16.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:16 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T12:10:16.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:16 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T12:10:16.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:16 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T12:10:16.848 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:16.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:16.950 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:16.976 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:16.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:17.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:17.103 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:17.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:17.205 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:17.231 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:17.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:17.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:17.358 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:17.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:17.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:10:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:17.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T12:10:17.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T12:10:17.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T12:10:17.436 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T12:10:17.437 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-24T12:10:17.437 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: osdmap e392: 8 total, 8 up, 8 in 2024-03-24T12:10:17.437 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: osdmap e393: 8 total, 8 up, 8 in 2024-03-24T12:10:17.437 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: 5.1e scrub starts 2024-03-24T12:10:17.437 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[30223]: 5.1e scrub ok 2024-03-24T12:10:17.437 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T12:10:17.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T12:10:17.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T12:10:17.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T12:10:17.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-24T12:10:17.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: osdmap e392: 8 total, 8 up, 8 in 2024-03-24T12:10:17.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: osdmap e393: 8 total, 8 up, 8 in 2024-03-24T12:10:17.439 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: 5.1e scrub starts 2024-03-24T12:10:17.439 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:17 smithi064 ceph-mon[42243]: 5.1e scrub ok 2024-03-24T12:10:17.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:17.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:10:17.493 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:10:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 393, 2024-03-24T12:10:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T12:10:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-03-24T11:46:40.657065+0000", 2024-03-24T12:10:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-03-24T12:10:16.949989+0000", 2024-03-24T12:10:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-03-24T12:08:35.127870+0000", 2024-03-24T12:10:17.493 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-03-24T11:51:43.667453+0000", 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5804032, 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 19, 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:10:17.494 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 10, 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:49:56.369674+0000", 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 20, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 11, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 11, 2024-03-24T12:10:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 11, 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 19, 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "1.14", 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 390, 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 389, 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 389, 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "83'130", 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "83'130" 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "392", 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "392", 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "392", 2024-03-24T12:10:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:17.499 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.500 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.7999999523162842, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.7999999523162842, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.7999999523162842, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.7999999523162842, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-03-24T12:10:17.501 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".rgw.root", 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:13.871974+0000", 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:17.502 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "68", 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:17.503 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:17.504 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:17.505 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.75, 2024-03-24T12:10:17.506 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.75, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "datapool", 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:14.117018+0000", 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-03-24T12:10:17.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 3, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 3, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 3, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 3, 2024-03-24T12:10:17.508 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 3, 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "71", 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:17.509 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 2, 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 71, 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:17.510 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:17.511 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.6700000762939453, 2024-03-24T12:10:17.512 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.6700000762939453, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 4, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:15.453744+0000", 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:17.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:10:17.514 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "70", 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:17.515 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:17.516 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:17.517 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.25, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.25, 2024-03-24T12:10:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.25, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.25, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 5, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:17.611004+0000", 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:17.519 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:17.520 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "72", 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:17.522 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:17.523 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.75, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.75, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-03-24T12:10:17.524 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:17.525 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 6, 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:19.701416+0000", 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:10:17.527 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "74", 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:17.528 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:17.529 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:17.530 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_bias": 4 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2, 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2, 2024-03-24T12:10:17.531 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 10, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "unique_pool_3", 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T12:06:28.568197+0000", 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:17.532 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 16, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 16, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 16, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 16, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 16, 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:17.533 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "323", 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:17.534 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:17.535 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:17.536 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "rados": {} 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 3, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 3, 2024-03-24T12:10:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d", 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:17.538 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 8, 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 385, 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.539 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:10:17.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/2675992571", 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6802/2675992571", 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6804/2675992571", 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6803/2675992571", 2024-03-24T12:10:17.541 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9", 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.2537689208984375, 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:17.542 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 385, 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:17.543 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6805/841667271", 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6806/841667271", 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6808/841667271", 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6807/841667271", 2024-03-24T12:10:17.545 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b", 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.3165130615234375, 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.90252685546875, 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:17.546 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 18, 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 385, 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/435147390", 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6810/435147390", 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6812/435147390", 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6811/435147390", 2024-03-24T12:10:17.549 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479", 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:17.550 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 27, 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 390, 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.552 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6813/3355318038", 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6814/3355318038", 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6816/3355318038", 2024-03-24T12:10:17.553 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6815/3355318038", 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b", 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:17.554 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 33, 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 386, 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6800", 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:17.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6801", 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6803", 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6802", 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6800/3851339176", 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6801/3851339176", 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3851339176", 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3851339176", 2024-03-24T12:10:17.557 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6", 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 41, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 317, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 353, 2024-03-24T12:10:17.558 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 386, 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 318, 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6804", 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6805", 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6807", 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6806", 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6804/1531860860", 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6805/1531860860", 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6807/1531860860", 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6806/1531860860", 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996", 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 51, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 392, 2024-03-24T12:10:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6808", 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6809", 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6811", 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:10:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6810", 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6808/2408237278", 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6809/2408237278", 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6811/2408237278", 2024-03-24T12:10:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6810/2408237278", 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015", 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 61, 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 386, 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6812", 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6813", 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6815", 2024-03-24T12:10:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6814", 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6812/2649989609", 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6813/2649989609", 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6815/2649989609", 2024-03-24T12:10:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6814/2649989609", 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:54.704535+0000", 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:46.893830+0000", 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:50.684279+0000", 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:13.289005+0000", 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:23.023447+0000", 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:40.102762+0000", 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 317 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:51.284967+0000", 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:45.580900+0000", 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.0", 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.13", 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: 0 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [ 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "1.0", 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 0 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.b", 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.4", 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 5 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.9", 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.1d", 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.6", 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.13", 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.18", 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 4, 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.19", 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 6 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": { 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/3538503390": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1277626303": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3407451558": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3753895773": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/842059688": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1610565583": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/2730170969": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3257265142": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1459540658": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/629947599": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/520335082": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3351953651": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/4027721305": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/4028269665": "2024-03-25T11:47:36.035658+0000" 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-03-24T12:10:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-03-24T12:10:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:10:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:10:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-03-24T12:10:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:17.585 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:10:17.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:17.609 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:17.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:17.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:17.734 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:17.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: osdmap e392: 8 total, 8 up, 8 in 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: osdmap e393: 8 total, 8 up, 8 in 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: 5.1e scrub starts 2024-03-24T12:10:17.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-mon[33593]: 5.1e scrub ok 2024-03-24T12:10:17.836 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:17.860 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:17.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:17.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:17.986 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:17.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:18.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:10:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:10:18.089 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:18.111 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:18.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:18.213 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:18.236 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:18.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:18.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:18.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:18.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:18.492 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:18.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:18.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:18.617 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:18.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:18.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:18.743 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:18.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:18.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[30223]: 6.17 scrub starts 2024-03-24T12:10:18.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1263041022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:10:18.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[30223]: pgmap v836: 167 pgs: 2 peering, 165 active+clean; 33 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T12:10:18.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:18.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[30223]: osdmap e394: 8 total, 8 up, 8 in 2024-03-24T12:10:18.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[42243]: 6.17 scrub starts 2024-03-24T12:10:18.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1263041022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:10:18.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[42243]: pgmap v836: 167 pgs: 2 peering, 165 active+clean; 33 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T12:10:18.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:18.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:18 smithi064 ceph-mon[42243]: osdmap e394: 8 total, 8 up, 8 in 2024-03-24T12:10:18.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:18 smithi193 ceph-mon[33593]: 6.17 scrub starts 2024-03-24T12:10:18.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1263041022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:10:18.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:18 smithi193 ceph-mon[33593]: pgmap v836: 167 pgs: 2 peering, 165 active+clean; 33 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T12:10:18.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:18 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:18.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:18 smithi193 ceph-mon[33593]: osdmap e394: 8 total, 8 up, 8 in 2024-03-24T12:10:18.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:18.869 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:18.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:18.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:18.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:19.095 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:19.121 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:19.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:19.153 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-24T12:10:19.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-24T12:10:19.222 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:19.246 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:19.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:19.348 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:19.373 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:19.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:19.475 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:19.499 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:19.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:19.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[30223]: 6.17 scrub ok 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[30223]: 10.7 scrub starts 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[30223]: 10.7 scrub ok 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[30223]: osdmap e395: 8 total, 8 up, 8 in 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[42243]: 6.17 scrub ok 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[42243]: 10.7 scrub starts 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[42243]: 10.7 scrub ok 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:10:19.527 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:19 smithi064 ceph-mon[42243]: osdmap e395: 8 total, 8 up, 8 in 2024-03-24T12:10:19.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:19.625 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:19.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:19.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:19.750 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:19.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:19.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:19 smithi193 ceph-mon[33593]: 6.17 scrub ok 2024-03-24T12:10:19.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:19 smithi193 ceph-mon[33593]: 10.7 scrub starts 2024-03-24T12:10:19.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:19 smithi193 ceph-mon[33593]: 10.7 scrub ok 2024-03-24T12:10:19.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:19 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:10:19.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:19 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:10:19.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:19 smithi193 ceph-mon[33593]: osdmap e395: 8 total, 8 up, 8 in 2024-03-24T12:10:19.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:19.877 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:19.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:19.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:20.005 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:20.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:20.033 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.24379.0:3309 5.3 5:c609908c:::notify.5:head [watch ping cookie 94338803862912 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:35.391087+0000", 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.135619238, 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77374225799999996, 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.24379", 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3309 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391087+0000", 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391087+0000", 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391088+0000", 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6199999999999997e-07 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391091+0000", 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.27e-06 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391099+0000", 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0790000000000001e-06 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391105+0000", 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4189999999999999e-06 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391207+0000", 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010178 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391219+0000", 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1824999999999999e-05 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.163922+0000", 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77270295 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.164740+0000", 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081809799999999996 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.164829+0000", 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9074999999999997e-05 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.24379.0:3319 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94338803862912 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:10:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:35.391522+0000", 2024-03-24T12:10:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.13518421400001, 2024-03-24T12:10:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77603222100000002, 2024-03-24T12:10:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:10:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.24379", 2024-03-24T12:10:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3319 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391522+0000", 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391522+0000", 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391523+0000", 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.257e-06 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391527+0000", 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2200000000000003e-06 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391539+0000", 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2119e-05 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391546+0000", 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.579e-06 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391587+0000", 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0840000000000002e-05 2024-03-24T12:10:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391593+0000", 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4230000000000002e-06 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.165226+0000", 2024-03-24T12:10:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77363315399999999 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.165302+0000", 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5346000000000007e-05 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.167554+0000", 2024-03-24T12:10:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022522829999999999 2024-03-24T12:10:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:176787 10.6 10:6a6c6402:::benchmark_data_smithi064_174908_object32370:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:10:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:36.144560+0000", 2024-03-24T12:10:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.382145959, 2024-03-24T12:10:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0412604430000001, 2024-03-24T12:10:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176787 2024-03-24T12:10:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.144560+0000", 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.144560+0000", 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.144562+0000", 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9190000000000002e-06 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.144567+0000", 2024-03-24T12:10:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1109999999999999e-06 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145924+0000", 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013566159999999999 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145940+0000", 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5937999999999999e-05 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146244+0000", 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030415500000000002 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146255+0000", 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0908e-05 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.170581+0000", 2024-03-24T12:10:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0243266419999999 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.170673+0000", 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1346000000000003e-05 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5", 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.170760+0000", 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780999999999995e-05 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.175687+0000", 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049273870000000001 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.175727+0000", 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0284000000000003e-05 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.185629+0000", 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099018170000000003 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.185696+0000", 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7212000000000006e-05 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.185821+0000", 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124327 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:176798 10.6 10:6ab26d4f:::benchmark_data_smithi064_174908_object32381:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:36.145962+0000", 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.380744213, 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0401340509999999, 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176798 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145962+0000", 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145962+0000", 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145963+0000", 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.285e-06 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145968+0000", 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2329999999999997e-06 2024-03-24T12:10:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145978+0000", 2024-03-24T12:10:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2769999999999994e-06 2024-03-24T12:10:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145985+0000", 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9029999999999998e-06 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146282+0000", 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029692 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146288+0000", 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7939999999999997e-06 2024-03-24T12:10:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.172642+0000", 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.026353413 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.172718+0000", 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6483999999999995e-05 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5", 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.172797+0000", 2024-03-24T12:10:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8979000000000004e-05 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.176900+0000", 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041023550000000002 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.176938+0000", 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8708000000000003e-05 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.185915+0000", 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089769440000000006 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.185976+0000", 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0531e-05 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.186096+0000", 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012022499999999999 2024-03-24T12:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:176825 10.6 10:6c4d3dcf:::benchmark_data_smithi064_174908_object32408:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:36.145994+0000", 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.380711866, 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.040328462, 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176825 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145994+0000", 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145994+0000", 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145995+0000", 2024-03-24T12:10:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2699999999999999e-07 2024-03-24T12:10:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.145998+0000", 2024-03-24T12:10:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2820000000000001e-06 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146004+0000", 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0519999999999998e-06 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146010+0000", 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0310000000000004e-06 2024-03-24T12:10:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146316+0000", 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030599499999999999 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146323+0000", 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.793e-06 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.174220+0000", 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.027897185 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.174292+0000", 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2033999999999995e-05 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5", 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.174375+0000", 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3086999999999999e-05 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.176905+0000", 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025290769999999998 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.178090+0000", 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011851279999999999 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.186158+0000", 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080686830000000001 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.186217+0000", 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8479e-05 2024-03-24T12:10:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.186323+0000", 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010590900000000001 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:176830 10.6 10:6b46e96a:::benchmark_data_smithi064_174908_object32413:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:36.146018+0000", 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.38068802399999, 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0405270609999999, 2024-03-24T12:10:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176830 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146018+0000", 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146018+0000", 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146019+0000", 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4399999999999999e-07 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146023+0000", 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.755e-06 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146063+0000", 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0095000000000002e-05 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146070+0000", 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4309999999999997e-06 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146346+0000", 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027608800000000002 2024-03-24T12:10:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.146353+0000", 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7769999999999997e-06 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.175750+0000", 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0293965730000001 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.175824+0000", 2024-03-24T12:10:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3975999999999998e-05 2024-03-24T12:10:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5", 2024-03-24T12:10:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.175909+0000", 2024-03-24T12:10:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5532999999999996e-05 2024-03-24T12:10:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.177019+0000", 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011098200000000001 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.185573+0000", 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085543719999999993 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.186370+0000", 2024-03-24T12:10:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079670200000000004 2024-03-24T12:10:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.186427+0000", 2024-03-24T12:10:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6586999999999999e-05 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.186545+0000", 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118608 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:224501 10.8 10:14fd65e4:::benchmark_data_smithi064_174908_object80084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:20.474764+0000", 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.051942226999998, 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201381015, 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 224501 2024-03-24T12:10:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.474764+0000", 2024-03-24T12:10:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.474764+0000", 2024-03-24T12:10:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.474766+0000", 2024-03-24T12:10:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.81e-06 2024-03-24T12:10:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.474771+0000", 2024-03-24T12:10:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2580000000000003e-06 2024-03-24T12:10:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.474782+0000", 2024-03-24T12:10:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0618999999999999e-05 2024-03-24T12:10:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.474791+0000", 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8549999999999997e-06 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.474887+0000", 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5098000000000003e-05 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.475005+0000", 2024-03-24T12:10:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118497 2024-03-24T12:10:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:10:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.475113+0000", 2024-03-24T12:10:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107626 2024-03-24T12:10:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.484701+0000", 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095885290000000001 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.486158+0000", 2024-03-24T12:10:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001457035 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.676057+0000", 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18989885100000001 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.676095+0000", 2024-03-24T12:10:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8399999999999998e-05 2024-03-24T12:10:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.676145+0000", 2024-03-24T12:10:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9437000000000001e-05 2024-03-24T12:10:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:224548 10.8 10:1dae92e4:::benchmark_data_smithi064_174908_object80131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-03-24T12:10:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:20.533885+0000", 2024-03-24T12:10:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.992821059000001, 2024-03-24T12:10:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14514685299999999, 2024-03-24T12:10:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 224548 2024-03-24T12:10:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.533885+0000", 2024-03-24T12:10:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.533885+0000", 2024-03-24T12:10:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.533887+0000", 2024-03-24T12:10:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3690000000000001e-06 2024-03-24T12:10:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.533894+0000", 2024-03-24T12:10:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8510000000000001e-06 2024-03-24T12:10:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.533911+0000", 2024-03-24T12:10:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.668e-05 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.533921+0000", 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0316000000000001e-05 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.533986+0000", 2024-03-24T12:10:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4560999999999994e-05 2024-03-24T12:10:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.534110+0000", 2024-03-24T12:10:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012412600000000001 2024-03-24T12:10:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:10:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.534234+0000", 2024-03-24T12:10:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012404000000000001 2024-03-24T12:10:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.536927+0000", 2024-03-24T12:10:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026933439999999999 2024-03-24T12:10:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.539187+0000", 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022592749999999998 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.678952+0000", 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13976523800000001 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.678988+0000", 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6032000000000003e-05 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:20.679032+0000", 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4020999999999997e-05 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231410 10.4 10:25d2fc15:::benchmark_data_smithi064_174908_object86993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.249610+0000", 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.277096000999997, 2024-03-24T12:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.140851168, 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231410 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.249610+0000", 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.249610+0000", 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.249611+0000", 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-07 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.249613+0000", 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.419e-06 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.249618+0000", 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8729999999999998e-06 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.249623+0000", 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.78e-06 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.249666+0000", 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3263999999999998e-05 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.250220+0000", 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382200000000004 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.250351+0000", 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013086099999999999 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.254860+0000", 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045093800000000003 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.256541+0000", 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016811700000000001 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.390232+0000", 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.133690903 2024-03-24T12:10:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.390408+0000", 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017572399999999999 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.390461+0000", 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3470000000000001e-05 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231422 10.4 10:2dd8f7eb:::benchmark_data_smithi064_174908_object87005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.260239+0000", 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.266467118000001, 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13665433399999999, 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231422 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260239+0000", 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260239+0000", 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260241+0000", 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.939e-06 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260246+0000", 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.831e-06 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260263+0000", 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6898e-05 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260270+0000", 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7349999999999996e-06 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260324+0000", 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3976e-05 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260396+0000", 2024-03-24T12:10:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1329999999999996e-05 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.260498+0000", 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101891 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.266811+0000", 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006313908 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.266886+0000", 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4109000000000005e-05 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.396771+0000", 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.129885536 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.396830+0000", 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9049999999999999e-05 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.396893+0000", 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3131000000000003e-05 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231426 10.4 10:22438482:::benchmark_data_smithi064_174908_object87009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.261852+0000", 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.264854348, 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13519579100000001, 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231426 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.261852+0000", 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.261852+0000", 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.261853+0000", 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.119e-06 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.261857+0000", 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3030000000000001e-06 2024-03-24T12:10:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.261879+0000", 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2127e-05 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.261888+0000", 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6829999999999997e-06 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.262847+0000", 2024-03-24T12:10:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095929399999999997 2024-03-24T12:10:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.262942+0000", 2024-03-24T12:10:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5068999999999994e-05 2024-03-24T12:10:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:10:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.263099+0000", 2024-03-24T12:10:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015671700000000001 2024-03-24T12:10:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267715+0000", 2024-03-24T12:10:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046155090000000003 2024-03-24T12:10:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.269479+0000", 2024-03-24T12:10:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001764757 2024-03-24T12:10:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.396940+0000", 2024-03-24T12:10:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12746097000000001 2024-03-24T12:10:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.397005+0000", 2024-03-24T12:10:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4391000000000001e-05 2024-03-24T12:10:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.397048+0000", 2024-03-24T12:10:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2852000000000002e-05 2024-03-24T12:10:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:255676 10.8 10:1a45541d:::benchmark_data_smithi064_174908_object111259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-24T12:10:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:49.461659+0000", 2024-03-24T12:10:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.065047149000002, 2024-03-24T12:10:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17837575899999999, 2024-03-24T12:10:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 255676 2024-03-24T12:10:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461659+0000", 2024-03-24T12:10:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461659+0000", 2024-03-24T12:10:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461660+0000", 2024-03-24T12:10:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.37e-06 2024-03-24T12:10:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461664+0000", 2024-03-24T12:10:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.121e-06 2024-03-24T12:10:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461673+0000", 2024-03-24T12:10:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9900000000000003e-06 2024-03-24T12:10:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461679+0000", 2024-03-24T12:10:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.643e-06 2024-03-24T12:10:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461735+0000", 2024-03-24T12:10:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6137999999999997e-05 2024-03-24T12:10:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461827+0000", 2024-03-24T12:10:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1399999999999999e-05 2024-03-24T12:10:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:10:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461904+0000", 2024-03-24T12:10:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7486999999999999e-05 2024-03-24T12:10:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.465791+0000", 2024-03-24T12:10:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038869379999999999 2024-03-24T12:10:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.469533+0000", 2024-03-24T12:10:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037423479999999999 2024-03-24T12:10:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639902+0000", 2024-03-24T12:10:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17036889999999999 2024-03-24T12:10:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639963+0000", 2024-03-24T12:10:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0705999999999998e-05 2024-03-24T12:10:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.640035+0000", 2024-03-24T12:10:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1717999999999993e-05 2024-03-24T12:10:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:255681 10.8 10:1d81bdc4:::benchmark_data_smithi064_174908_object111264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-24T12:10:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:49.462664+0000", 2024-03-24T12:10:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.064042273999998, 2024-03-24T12:10:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19068927499999999, 2024-03-24T12:10:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 255681 2024-03-24T12:10:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.462664+0000", 2024-03-24T12:10:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.462664+0000", 2024-03-24T12:10:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.462665+0000", 2024-03-24T12:10:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.331e-06 2024-03-24T12:10:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.462670+0000", 2024-03-24T12:10:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7609999999999998e-06 2024-03-24T12:10:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.462682+0000", 2024-03-24T12:10:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2255999999999999e-05 2024-03-24T12:10:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.462690+0000", 2024-03-24T12:10:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6979999999999998e-06 2024-03-24T12:10:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.464050+0000", 2024-03-24T12:10:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001360399 2024-03-24T12:10:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.464125+0000", 2024-03-24T12:10:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4580999999999996e-05 2024-03-24T12:10:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.464191+0000", 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5687999999999999e-05 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.465949+0000", 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001758538 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.469574+0000", 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036245140000000001 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653255+0000", 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18368174500000001 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653309+0000", 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3295999999999999e-05 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653353+0000", 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4468000000000002e-05 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:255703 10.8 10:1c59ceda:::benchmark_data_smithi064_174908_object111286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:49.482293+0000", 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.044413562999999, 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17120736, 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 255703 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482293+0000", 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482293+0000", 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482294+0000", 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482298+0000", 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0799999999999999e-06 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482312+0000", 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4082e-05 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482321+0000", 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6309999999999992e-06 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482436+0000", 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115054 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482529+0000", 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3170000000000004e-05 2024-03-24T12:10:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482665+0000", 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135399 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.488824+0000", 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061590990000000003 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.488877+0000", 2024-03-24T12:10:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3520000000000003e-05 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653417+0000", 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16453954400000001 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653463+0000", 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6236000000000003e-05 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653500+0000", 2024-03-24T12:10:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6983000000000001e-05 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:255705 10.8 10:145b299f:::benchmark_data_smithi064_174908_object111288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:49.482352+0000", 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.044354309999999, 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17126571500000001, 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 255705 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482352+0000", 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482352+0000", 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482353+0000", 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0160000000000001e-06 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482357+0000", 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0090000000000001e-06 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482365+0000", 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0649999999999999e-06 2024-03-24T12:10:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.482372+0000", 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.007e-06 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.486813+0000", 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044415289999999996 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.486966+0000", 2024-03-24T12:10:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015207799999999999 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.487095+0000", 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012958800000000001 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.488978+0000", 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018828600000000001 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.490956+0000", 2024-03-24T12:10:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019780969999999998 2024-03-24T12:10:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653541+0000", 2024-03-24T12:10:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16258478700000001 2024-03-24T12:10:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653583+0000", 2024-03-24T12:10:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2653e-05 2024-03-24T12:10:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653618+0000", 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4026000000000003e-05 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:255744 10.8 10:1ae85e2d:::benchmark_data_smithi064_174908_object111327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:49.516843+0000", 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.009863063000001, 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13689211500000001, 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 255744 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.516843+0000", 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.516843+0000", 2024-03-24T12:10:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.516845+0000", 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.623e-06 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.516850+0000", 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2510000000000002e-06 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.516871+0000", 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0784999999999998e-05 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.516879+0000", 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0430000000000008e-06 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.516949+0000", 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0178999999999993e-05 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.517078+0000", 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012942699999999999 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.517202+0000", 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012402800000000001 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.519394+0000", 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021917019999999998 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.520324+0000", 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000930141 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653658+0000", 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13333408199999999 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653701+0000", 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2947000000000001e-05 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.653735+0000", 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3906999999999999e-05 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:286201 10.5 e388/320 10:aada9ead:::benchmark_data_smithi064_174908_object141784:head v 388'18360, mlcod=388'18357)", 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:13.190760+0000", 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.3359460439999999, 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040042907000000003, 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.190760+0000", 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.190760+0000", 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.190762+0000", 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4750000000000001e-06 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.190768+0000", 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.917e-06 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.190779+0000", 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0129000000000001e-05 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.190786+0000", 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3989999999999998e-06 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.191473+0000", 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068691800000000001 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.191492+0000", 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8762000000000001e-05 2024-03-24T12:10:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.230772+0000", 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039280278000000002 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.230803+0000", 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1028999999999997e-05 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:287644 10.5 e389/320 10:a82822aa:::benchmark_data_smithi064_174908_object143227:head v 389'18458, mlcod=389'18452)", 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:14.307279+0000", 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2194275640000001, 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054682629000000003, 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.307279+0000", 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.307279+0000", 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.307280+0000", 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.015e-06 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.307286+0000", 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1369999999999999e-06 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.307298+0000", 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2636000000000001e-05 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.307305+0000", 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5409999999999997e-06 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.317895+0000", 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010590232999999999 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.317916+0000", 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0673999999999999e-05 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.361917+0000", 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044000824000000001 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.361961+0000", 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4569e-05 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:287648 10.5 e389/320 10:a81a90e5:::benchmark_data_smithi064_174908_object143231:head v 389'18459, mlcod=389'18452)", 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:14.311496+0000", 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.215209615, 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050580185, 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.311496+0000", 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.311496+0000", 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.311498+0000", 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.215e-06 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.311505+0000", 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9210000000000003e-06 2024-03-24T12:10:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.311518+0000", 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3633e-05 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.311526+0000", 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3900000000000004e-06 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.327154+0000", 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015627931000000001 2024-03-24T12:10:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.327172+0000", 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8544000000000001e-05 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.362050+0000", 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034877699999999998 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.362077+0000", 2024-03-24T12:10:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6851000000000001e-05 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:287652 10.5 e389/320 10:af45c59d:::benchmark_data_smithi064_174908_object143235:head v 389'18460, mlcod=389'18452)", 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:14.318541+0000", 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.208164751, 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043561812999999998, 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.318541+0000", 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.318541+0000", 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.318542+0000", 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0780000000000001e-06 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.318548+0000", 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9850000000000003e-06 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.318557+0000", 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7280000000000001e-06 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.318564+0000", 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3770000000000001e-06 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.335961+0000", 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017397695000000001 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.335980+0000", 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8542999999999998e-05 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.362079+0000", 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026099527000000001 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.362103+0000", 2024-03-24T12:10:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3879999999999998e-05 2024-03-24T12:10:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:20.188 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:10:20.188 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:20.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:20.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:20.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:20.341 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:20.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:20.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:20.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:20.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:20.595 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:20.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:20.697 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:20.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:20.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:20.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T12:10:20.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T12:10:20.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 4.1 scrub starts 2024-03-24T12:10:20.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 4.1 scrub ok 2024-03-24T12:10:20.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: pgmap v839: 166 pgs: 1 peering, 165 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 895 op/s 2024-03-24T12:10:20.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T12:10:20.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T12:10:20.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T12:10:20.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 6.a scrub starts 2024-03-24T12:10:20.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 6.a scrub ok 2024-03-24T12:10:20.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T12:10:20.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T12:10:20.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T12:10:20.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T12:10:20.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 4.1 scrub starts 2024-03-24T12:10:20.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 4.1 scrub ok 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: pgmap v839: 166 pgs: 1 peering, 165 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 895 op/s 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 6.a scrub starts 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 6.a scrub ok 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:20 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T12:10:20.827 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:20.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T12:10:20.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T12:10:20.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 4.1 scrub starts 2024-03-24T12:10:20.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 4.1 scrub ok 2024-03-24T12:10:20.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: pgmap v839: 166 pgs: 1 peering, 165 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 895 op/s 2024-03-24T12:10:20.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T12:10:20.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T12:10:20.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T12:10:20.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 6.a scrub starts 2024-03-24T12:10:20.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 6.a scrub ok 2024-03-24T12:10:20.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T12:10:20.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:20 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T12:10:20.855 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:20.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:20.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:20.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:20.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:21.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:21.115 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:21.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:21.216 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:21.244 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:21.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:21.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:21.374 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:21.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:21.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:21.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:21 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-24T12:10:21.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:21 smithi064 ceph-mon[30223]: osdmap e396: 8 total, 8 up, 8 in 2024-03-24T12:10:21.484 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:21 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-24T12:10:21.484 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:21 smithi064 ceph-mon[42243]: osdmap e396: 8 total, 8 up, 8 in 2024-03-24T12:10:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:21.502 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:21.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:21.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:21.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:21.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:21.725 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:10:21.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:21.745 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:10:21.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:21.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:21.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:10:21.784 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:10:21.803 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:10:21.824 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:10:21.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:21 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-24T12:10:21.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:21 smithi193 ceph-mon[33593]: osdmap e396: 8 total, 8 up, 8 in 2024-03-24T12:10:21.844 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:10:21.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:21.864 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:10:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:10:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:10:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:10:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:10:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:10:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:10:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:10:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:10:21.884 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:21.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:21.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:22.011 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:22.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:22.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:22.136 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:22.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:22.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:22.266 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:22.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:22.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:22.398 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:22.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:22.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:22.500 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:22.527 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:22.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:22 smithi064 ceph-mon[30223]: osdmap e397: 8 total, 8 up, 8 in 2024-03-24T12:10:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:22 smithi064 ceph-mon[30223]: pgmap v842: 166 pgs: 1 peering, 165 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 130 op/s 2024-03-24T12:10:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:22 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1817466831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:22.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:22 smithi064 ceph-mon[42243]: osdmap e397: 8 total, 8 up, 8 in 2024-03-24T12:10:22.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:22 smithi064 ceph-mon[42243]: pgmap v842: 166 pgs: 1 peering, 165 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 130 op/s 2024-03-24T12:10:22.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:22 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1817466831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:22.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:22.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:22.689 INFO:tasks.thrashosds.thrasher:j is {'epoch': 393, 'fsid': '06f23456-e9d4-11ee-95cd-87774f69a715', 'created': '2024-03-24T11:46:40.657065+0000', 'modified': '2024-03-24T12:10:16.949989+0000', 'last_up_change': '2024-03-24T12:08:35.127870+0000', 'last_in_change': '2024-03-24T11:51:43.667453+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:49:56.369674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 20, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 19, 'last_pg_merge_meta': {'source_pgid': '1.14', 'ready_epoch': 390, 'last_epoch_started': 389, 'last_epoch_clean': 389, 'source_version': "83'130", 'target_version': "83'130"}, 'last_change': '392', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '392', 'last_force_op_resend_preluminous': '392', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.799999952316284, 'score_stable': 2.799999952316284, 'optimal_score': 1, 'raw_score_acting': 2.799999952316284, 'raw_score_stable': 2.799999952316284, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T11:52:13.871974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-24T11:52:14.117018+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T11:52:15.453744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T11:52:17.611004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T11:52:19.701416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-24T12:06:28.568197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '323', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'bc14a64f-9e6c-41f1-901e-c49fabffb53d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 2675992571}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6802', 'nonce': 2675992571}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6804', 'nonce': 2675992571}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 2675992571}]}, 'public_addr': '172.21.15.64:6801/2675992571', 'cluster_addr': '172.21.15.64:6802/2675992571', 'heartbeat_back_addr': '172.21.15.64:6804/2675992571', 'heartbeat_front_addr': '172.21.15.64:6803/2675992571', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '74587332-f0b2-4905-9888-7bdb604cc0e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2537689208984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 841667271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6806', 'nonce': 841667271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6808', 'nonce': 841667271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 841667271}]}, 'public_addr': '172.21.15.64:6805/841667271', 'cluster_addr': '172.21.15.64:6806/841667271', 'heartbeat_back_addr': '172.21.15.64:6808/841667271', 'heartbeat_front_addr': '172.21.15.64:6807/841667271', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a0c0ed85-9f33-4198-a1f2-447eb17c364b', 'up': 1, 'in': 1, 'weight': 0.3165130615234375, 'primary_affinity': 0.90252685546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 435147390}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6810', 'nonce': 435147390}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6812', 'nonce': 435147390}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 435147390}]}, 'public_addr': '172.21.15.64:6809/435147390', 'cluster_addr': '172.21.15.64:6810/435147390', 'heartbeat_back_addr': '172.21.15.64:6812/435147390', 'heartbeat_front_addr': '172.21.15.64:6811/435147390', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1396f66c-3e0c-4a01-b0a7-6f763989a479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 390, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3355318038}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6814', 'nonce': 3355318038}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6816', 'nonce': 3355318038}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3355318038}]}, 'public_addr': '172.21.15.64:6813/3355318038', 'cluster_addr': '172.21.15.64:6814/3355318038', 'heartbeat_back_addr': '172.21.15.64:6816/3355318038', 'heartbeat_front_addr': '172.21.15.64:6815/3355318038', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50d65d8b-3e90-46a4-b19d-50c3015af40b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 386, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3851339176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3851339176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3851339176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3851339176}]}, 'public_addr': '172.21.15.193:6800/3851339176', 'cluster_addr': '172.21.15.193:6801/3851339176', 'heartbeat_back_addr': '172.21.15.193:6803/3851339176', 'heartbeat_front_addr': '172.21.15.193:6802/3851339176', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '50c5d50e-7e68-487c-9f68-1abb8c927ff6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 317, 'up_from': 353, 'up_thru': 386, 'down_at': 318, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 1531860860}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 1531860860}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 1531860860}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 1531860860}]}, 'public_addr': '172.21.15.193:6804/1531860860', 'cluster_addr': '172.21.15.193:6805/1531860860', 'heartbeat_back_addr': '172.21.15.193:6807/1531860860', 'heartbeat_front_addr': '172.21.15.193:6806/1531860860', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7e74fe39-5707-4c90-ad38-1083cd7a6996', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 392, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 2408237278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 2408237278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 2408237278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 2408237278}]}, 'public_addr': '172.21.15.193:6808/2408237278', 'cluster_addr': '172.21.15.193:6809/2408237278', 'heartbeat_back_addr': '172.21.15.193:6811/2408237278', 'heartbeat_front_addr': '172.21.15.193:6810/2408237278', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e568df64-cc9a-43f8-a432-8628d1da1015', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 386, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 2649989609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 2649989609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 2649989609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 2649989609}]}, 'public_addr': '172.21.15.193:6812/2649989609', 'cluster_addr': '172.21.15.193:6813/2649989609', 'heartbeat_back_addr': '172.21.15.193:6815/2649989609', 'heartbeat_front_addr': '172.21.15.193:6814/2649989609', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:54.704535+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:46.893830+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:50.684279+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:13.289005+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:23.023447+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:40.102762+0000', 'dead_epoch': 317}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:51.284967+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:45.580900+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [0, 1, 7]}, {'pgid': '2.13', 'osds': [6, 3, 0]}], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 7, 'to': 0}]}, {'pgid': '2.b', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.6', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.64:6800/3538503390': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/1277626303': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3407451558': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3753895773': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/842059688': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/1610565583': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/2730170969': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3257265142': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/1459540658': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/629947599': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:6800/520335082': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3351953651': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:6800/4027721305': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/4028269665': '2024-03-25T11:47:36.035658+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:10:22.689 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.13 2024-03-24T12:10:22.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd rm-pg-upmap 2.13 2024-03-24T12:10:22.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:22.783 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:22.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:22.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:22 smithi193 ceph-mon[33593]: osdmap e397: 8 total, 8 up, 8 in 2024-03-24T12:10:22.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:22 smithi193 ceph-mon[33593]: pgmap v842: 166 pgs: 1 peering, 165 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 130 op/s 2024-03-24T12:10:22.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:22 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1817466831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:22.886 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:22.966 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:22.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:23.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:23.096 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:23.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:23.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:23.225 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:23.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:23.328 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:23.356 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:23.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:23.458 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:23.484 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:23.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:23.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:10:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:10:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[30223]: osdmap e398: 8 total, 8 up, 8 in 2024-03-24T12:10:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[30223]: 4.11 scrub starts 2024-03-24T12:10:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[30223]: 4.11 scrub ok 2024-03-24T12:10:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:10:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:10:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[42243]: osdmap e398: 8 total, 8 up, 8 in 2024-03-24T12:10:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[42243]: 4.11 scrub starts 2024-03-24T12:10:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:23 smithi064 ceph-mon[42243]: 4.11 scrub ok 2024-03-24T12:10:23.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:23.608 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:23.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:23.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:23.736 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:23 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:10:23.736 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:23 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:10:23.736 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:23 smithi193 ceph-mon[33593]: osdmap e398: 8 total, 8 up, 8 in 2024-03-24T12:10:23.736 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:23 smithi193 ceph-mon[33593]: 4.11 scrub starts 2024-03-24T12:10:23.736 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:23 smithi193 ceph-mon[33593]: 4.11 scrub ok 2024-03-24T12:10:23.738 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:23.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:23.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:23.864 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:23.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:23.965 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:23.993 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:23.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:24.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:24.118 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:24.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:24.220 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:24.245 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:24.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:24.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:10:24.346 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:24.352 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:24.360 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:24.370 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:24.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:24.410 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:10:24.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:24.498 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:24.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:24.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:24.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: osdmap e399: 8 total, 8 up, 8 in 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/736720433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: from='client.35807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: pgmap v845: 165 pgs: 1 clean+premerge+peered, 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: 2.19 scrub starts 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: 2.19 scrub ok 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: 5.1b scrub starts 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: 5.1b scrub ok 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:10:24.621 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: osdmap e399: 8 total, 8 up, 8 in 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/736720433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: from='client.35807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: pgmap v845: 165 pgs: 1 clean+premerge+peered, 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: 2.19 scrub starts 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: 2.19 scrub ok 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: 5.1b scrub starts 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: 5.1b scrub ok 2024-03-24T12:10:24.622 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:24 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:10:24.626 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:24.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:24.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:24.752 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:24.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:24.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:10:24.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:10:24.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: osdmap e399: 8 total, 8 up, 8 in 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/736720433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: from='client.35807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: pgmap v845: 165 pgs: 1 clean+premerge+peered, 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: 2.19 scrub starts 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: 2.19 scrub ok 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: 5.1b scrub starts 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: 5.1b scrub ok 2024-03-24T12:10:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:24 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:10:24.854 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:24.878 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:24.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:24.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:25.003 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:25.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:25.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:25.129 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:25.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:25.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:25.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:25.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:25.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:25.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:25.481 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:25.506 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:25.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:10:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:10:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: from='client.35807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:10:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: osdmap e400: 8 total, 8 up, 8 in 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: 2.18 scrub starts 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: 2.18 scrub ok 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:10:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: from='client.35807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: osdmap e400: 8 total, 8 up, 8 in 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: 2.18 scrub starts 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: 2.18 scrub ok 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T12:10:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:25 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T12:10:25.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:25.632 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:25.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:25.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:25.757 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:25.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:25.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: from='client.35807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: osdmap e400: 8 total, 8 up, 8 in 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: 2.18 scrub starts 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: 2.18 scrub ok 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:25.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T12:10:25.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:25 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T12:10:25.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:25.884 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:25.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:25.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:26.010 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:26.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:26.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:10:26.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:26.134 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:26.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:26.236 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:26.260 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:26.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:26.362 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:26.385 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:26.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:26.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:26.511 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:26.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 1.5 scrub starts 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 1.5 scrub ok 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: osdmap e401: 8 total, 8 up, 8 in 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: pgmap v848: 165 pgs: 1 clean+premerge+peered, 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T12:10:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 5.16 scrub starts 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 5.16 scrub ok 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 1.5 scrub starts 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 1.5 scrub ok 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: osdmap e401: 8 total, 8 up, 8 in 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: pgmap v848: 165 pgs: 1 clean+premerge+peered, 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 5.16 scrub starts 2024-03-24T12:10:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 5.16 scrub ok 2024-03-24T12:10:26.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T12:10:26.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:26 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T12:10:26.613 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:26.637 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:26.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:26.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:26.765 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:26.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:26.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T12:10:26.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T12:10:26.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 1.5 scrub starts 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 1.5 scrub ok 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: osdmap e401: 8 total, 8 up, 8 in 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: pgmap v848: 165 pgs: 1 clean+premerge+peered, 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 5.16 scrub starts 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 5.16 scrub ok 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T12:10:26.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:26 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T12:10:26.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:26.890 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:26.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:26.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:27.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:27.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:27.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:27.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:27.244 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:27.270 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:27.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:27.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:27.397 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:27.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:27.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:27.521 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:27.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:27.589 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[42243]: 1.0 deep-scrub starts 2024-03-24T12:10:27.590 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[42243]: 1.0 deep-scrub ok 2024-03-24T12:10:27.590 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[42243]: osdmap e402: 8 total, 8 up, 8 in 2024-03-24T12:10:27.590 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[42243]: osdmap e403: 8 total, 8 up, 8 in 2024-03-24T12:10:27.590 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:27.590 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:27.590 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:10:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:27.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[30223]: 1.0 deep-scrub starts 2024-03-24T12:10:27.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[30223]: 1.0 deep-scrub ok 2024-03-24T12:10:27.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[30223]: osdmap e402: 8 total, 8 up, 8 in 2024-03-24T12:10:27.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[30223]: osdmap e403: 8 total, 8 up, 8 in 2024-03-24T12:10:27.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:27.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:27.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:27.648 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:27.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:27.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:10:27.749 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:27.794 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:27.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:27 smithi193 ceph-mon[33593]: 1.0 deep-scrub starts 2024-03-24T12:10:27.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:27 smithi193 ceph-mon[33593]: 1.0 deep-scrub ok 2024-03-24T12:10:27.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:27 smithi193 ceph-mon[33593]: osdmap e402: 8 total, 8 up, 8 in 2024-03-24T12:10:27.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:27 smithi193 ceph-mon[33593]: osdmap e403: 8 total, 8 up, 8 in 2024-03-24T12:10:27.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:27.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:27.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:27.897 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:27.925 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:27.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:28.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:28.054 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:28.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:28.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:10:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:10:28.156 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:28.185 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:28.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:28.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:28.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:28.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:28.443 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:28.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:28.544 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:28.572 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:28.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:28.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:28.699 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:28.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[30223]: pgmap v851: 164 pgs: 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]: dispatch 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.0"}]: dispatch 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T12:10:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[42243]: pgmap v851: 164 pgs: 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]: dispatch 2024-03-24T12:10:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.0"}]: dispatch 2024-03-24T12:10:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:28.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:28.830 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:28.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:28.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:28 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T12:10:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:28 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T12:10:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:28 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T12:10:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:28 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T12:10:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:28 smithi193 ceph-mon[33593]: pgmap v851: 164 pgs: 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]: dispatch 2024-03-24T12:10:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.0"}]: dispatch 2024-03-24T12:10:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:28.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:28.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:28.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:29.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:29.081 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:29.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:29.183 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:29.212 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:29.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:29.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:29.338 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:29.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:29.436 INFO:teuthology.orchestra.run.smithi064.stderr:clear 2.13 pg_upmap mapping 2024-03-24T12:10:29.440 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:29.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:29.469 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:29.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:29.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:29.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:29.595 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:29.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:29.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:29.721 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:29.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:29.786 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]': finished 2024-03-24T12:10:29.786 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.0"}]': finished 2024-03-24T12:10:29.786 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[42243]: osdmap e404: 8 total, 8 up, 8 in 2024-03-24T12:10:29.786 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[42243]: 4.14 scrub starts 2024-03-24T12:10:29.786 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[42243]: 4.14 scrub ok 2024-03-24T12:10:29.787 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1133476657' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]: dispatch 2024-03-24T12:10:29.787 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[42243]: from='client.35813 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]: dispatch 2024-03-24T12:10:29.787 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]': finished 2024-03-24T12:10:29.787 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.0"}]': finished 2024-03-24T12:10:29.787 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[30223]: osdmap e404: 8 total, 8 up, 8 in 2024-03-24T12:10:29.787 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[30223]: 4.14 scrub starts 2024-03-24T12:10:29.787 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[30223]: 4.14 scrub ok 2024-03-24T12:10:29.788 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1133476657' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]: dispatch 2024-03-24T12:10:29.788 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:29 smithi064 ceph-mon[30223]: from='client.35813 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]: dispatch 2024-03-24T12:10:29.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:29 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]': finished 2024-03-24T12:10:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:29 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.0"}]': finished 2024-03-24T12:10:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:29 smithi193 ceph-mon[33593]: osdmap e404: 8 total, 8 up, 8 in 2024-03-24T12:10:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:29 smithi193 ceph-mon[33593]: 4.14 scrub starts 2024-03-24T12:10:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:29 smithi193 ceph-mon[33593]: 4.14 scrub ok 2024-03-24T12:10:29.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:29 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1133476657' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]: dispatch 2024-03-24T12:10:29.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:29 smithi193 ceph-mon[33593]: from='client.35813 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]: dispatch 2024-03-24T12:10:29.848 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:29.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:29.950 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:29.975 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:29.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:30.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:30.100 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:30.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:30.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:30.231 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:30.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:30.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:30.358 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:30.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:30.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:30.486 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:30.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:30.526 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:10:30.526 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:10:30.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:30.611 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:30.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:30.713 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:30.738 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:30.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:10:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:10:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: 10.a scrub starts 2024-03-24T12:10:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: 10.a scrub ok 2024-03-24T12:10:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: from='client.35813 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]': finished 2024-03-24T12:10:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: osdmap e405: 8 total, 8 up, 8 in 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: pgmap v854: 164 pgs: 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: 4.0 deep-scrub starts 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: 4.0 deep-scrub ok 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: 10.a scrub starts 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: 10.a scrub ok 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: from='client.35813 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]': finished 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: osdmap e405: 8 total, 8 up, 8 in 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: pgmap v854: 164 pgs: 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:30.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T12:10:30.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: 4.0 deep-scrub starts 2024-03-24T12:10:30.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: 4.0 deep-scrub ok 2024-03-24T12:10:30.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T12:10:30.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:30 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T12:10:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:10:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:10:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: 10.a scrub starts 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: 10.a scrub ok 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: from='client.35813 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.13"}]': finished 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: osdmap e405: 8 total, 8 up, 8 in 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: pgmap v854: 164 pgs: 164 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: 4.0 deep-scrub starts 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: 4.0 deep-scrub ok 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T12:10:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:30 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T12:10:30.839 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:30.868 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:30.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:30.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:30.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:31.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:31.121 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:31.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:31.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:31.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:31.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:31.348 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:31.376 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:31.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:31.447 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_3' removed 2024-03-24T12:10:31.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:31.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:31.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:10:31.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:31.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:31.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:31.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:31.779 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:31.781 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:31.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:31.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 10.2 scrub starts 2024-03-24T12:10:31.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 10.2 scrub ok 2024-03-24T12:10:31.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 10.5 scrub starts 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 10.5 scrub ok 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 10.e scrub starts 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 10.e scrub ok 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: osdmap e406: 8 total, 8 up, 8 in 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2720629529' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 5.17 scrub starts 2024-03-24T12:10:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[42243]: 5.17 scrub ok 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 10.2 scrub starts 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 10.2 scrub ok 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 10.5 scrub starts 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 10.5 scrub ok 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 10.e scrub starts 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 10.e scrub ok 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: osdmap e406: 8 total, 8 up, 8 in 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2720629529' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 5.17 scrub starts 2024-03-24T12:10:31.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:31 smithi064 ceph-mon[30223]: 5.17 scrub ok 2024-03-24T12:10:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 10.2 scrub starts 2024-03-24T12:10:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 10.2 scrub ok 2024-03-24T12:10:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T12:10:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T12:10:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 10.5 scrub starts 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 10.5 scrub ok 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 10.e scrub starts 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 10.e scrub ok 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: osdmap e406: 8 total, 8 up, 8 in 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2720629529' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 5.17 scrub starts 2024-03-24T12:10:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:31 smithi193 ceph-mon[33593]: 5.17 scrub ok 2024-03-24T12:10:31.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:31.906 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:31.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:32.008 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:32.032 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:32.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:32.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:32.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:32.158 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:32.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:32.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:32.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:32.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:32.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:32.409 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:32.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:32.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:32.482 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T12:10:32.482 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T12:10:32.482 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[30223]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:10:32.482 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[30223]: osdmap e407: 8 total, 8 up, 8 in 2024-03-24T12:10:32.482 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[30223]: pgmap v857: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-03-24T12:10:32.482 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[30223]: 2.a scrub starts 2024-03-24T12:10:32.482 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[30223]: 2.a scrub ok 2024-03-24T12:10:32.483 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T12:10:32.483 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T12:10:32.483 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[42243]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:10:32.483 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[42243]: osdmap e407: 8 total, 8 up, 8 in 2024-03-24T12:10:32.483 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[42243]: pgmap v857: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-03-24T12:10:32.483 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[42243]: 2.a scrub starts 2024-03-24T12:10:32.483 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:32 smithi064 ceph-mon[42243]: 2.a scrub ok 2024-03-24T12:10:32.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:32.536 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:32.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:32.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:32.661 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:32.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:32.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:32.788 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:32.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:32 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T12:10:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:32 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T12:10:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:32 smithi193 ceph-mon[33593]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:10:32.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:32 smithi193 ceph-mon[33593]: osdmap e407: 8 total, 8 up, 8 in 2024-03-24T12:10:32.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:32 smithi193 ceph-mon[33593]: pgmap v857: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-03-24T12:10:32.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:32 smithi193 ceph-mon[33593]: 2.a scrub starts 2024-03-24T12:10:32.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:32 smithi193 ceph-mon[33593]: 2.a scrub ok 2024-03-24T12:10:32.889 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:32.914 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:32.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:32.932 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T12:10:32.933 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T12:10:32.933 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-24T12:10:32.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-24T12:10:33.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:33.038 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:33.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:33.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:33.164 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:33.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:33.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:33.289 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:33.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:33.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:33.404 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:33.416 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:33.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:33.442 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:33.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:33.541 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:33.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:33.643 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:33.667 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:33.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:33.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:33 smithi064 ceph-mon[30223]: osdmap e408: 8 total, 8 up, 8 in 2024-03-24T12:10:33.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:33 smithi064 ceph-mon[42243]: osdmap e408: 8 total, 8 up, 8 in 2024-03-24T12:10:33.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:33.793 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:33.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:33 smithi193 ceph-mon[33593]: osdmap e408: 8 total, 8 up, 8 in 2024-03-24T12:10:33.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:33.917 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:33.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:34.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:34.045 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:34.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:34.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:34.169 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:34.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:34.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:34.294 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:34.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:34.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:34.421 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:34.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:34.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:34.549 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:34.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:34.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:34.678 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:34.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:34.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:34.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:10:34.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:10:34.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: osdmap e409: 8 total, 8 up, 8 in 2024-03-24T12:10:34.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: pgmap v860: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:10:34.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: 4.4 scrub starts 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: 4.4 scrub ok 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: osdmap e409: 8 total, 8 up, 8 in 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: pgmap v860: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:10:34.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T12:10:34.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: 4.4 scrub starts 2024-03-24T12:10:34.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: 4.4 scrub ok 2024-03-24T12:10:34.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T12:10:34.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:34 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T12:10:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:34.808 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:34.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:10:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:10:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: osdmap e409: 8 total, 8 up, 8 in 2024-03-24T12:10:34.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: pgmap v860: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:10:34.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T12:10:34.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: 4.4 scrub starts 2024-03-24T12:10:34.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: 4.4 scrub ok 2024-03-24T12:10:34.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T12:10:34.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:34 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T12:10:34.909 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:34.936 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:34.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:35.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:35.062 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:35.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:35.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:35.191 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:35.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:35.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:35.317 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:35.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:35.419 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:35.448 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:35.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:35.549 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:35.577 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:35.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:35.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:35.702 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:35.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:35.795 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[30223]: osdmap e410: 8 total, 8 up, 8 in 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[30223]: 4.1e scrub starts 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[30223]: 4.1e scrub ok 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-24T12:10:35.796 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[42243]: osdmap e410: 8 total, 8 up, 8 in 2024-03-24T12:10:35.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:10:35.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:10:35.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[42243]: 4.1e scrub starts 2024-03-24T12:10:35.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:35 smithi064 ceph-mon[42243]: 4.1e scrub ok 2024-03-24T12:10:35.804 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:35.834 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:35.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:35 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:10:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:35 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:10:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:35 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-24T12:10:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:35 smithi193 ceph-mon[33593]: osdmap e410: 8 total, 8 up, 8 in 2024-03-24T12:10:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:35 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:10:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:35 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:10:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:35 smithi193 ceph-mon[33593]: 4.1e scrub starts 2024-03-24T12:10:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:35 smithi193 ceph-mon[33593]: 4.1e scrub ok 2024-03-24T12:10:35.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:35.963 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:35.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:36.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:36.089 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:36.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:36.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:36.215 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:36.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:36.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:36.340 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:36.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:36.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:36.467 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:36.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:36.483 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:36.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:36.593 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:36.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:36.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:36.722 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:36.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:36.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:36 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:10:36.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:36 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:10:36.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:36 smithi064 ceph-mon[30223]: osdmap e411: 8 total, 8 up, 8 in 2024-03-24T12:10:36.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:36 smithi064 ceph-mon[30223]: pgmap v863: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:36.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:36 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:10:36.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:36 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:10:36.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:36 smithi064 ceph-mon[42243]: osdmap e411: 8 total, 8 up, 8 in 2024-03-24T12:10:36.758 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:36 smithi064 ceph-mon[42243]: pgmap v863: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:36.824 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:36 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:10:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:36 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:10:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:36 smithi193 ceph-mon[33593]: osdmap e411: 8 total, 8 up, 8 in 2024-03-24T12:10:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:36 smithi193 ceph-mon[33593]: pgmap v863: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:36.853 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:36.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:36.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:36.982 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:36.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:36.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:37.084 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:37.111 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:37.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:37.213 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:37.240 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:37.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:37.248 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:10:37.248 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:10:37.248 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-24T12:10:37.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:10:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:10:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:10:37.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:37.369 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:37.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:37.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:37.496 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:37.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:37.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:37.624 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:37.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:37.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:37.749 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:37.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:37.778 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:37 smithi064 ceph-mon[30223]: osdmap e412: 8 total, 8 up, 8 in 2024-03-24T12:10:37.778 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:37 smithi064 ceph-mon[30223]: osdmap e413: 8 total, 8 up, 8 in 2024-03-24T12:10:37.778 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:10:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:10:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:37.779 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:37 smithi064 ceph-mon[42243]: osdmap e412: 8 total, 8 up, 8 in 2024-03-24T12:10:37.779 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:37 smithi064 ceph-mon[42243]: osdmap e413: 8 total, 8 up, 8 in 2024-03-24T12:10:37.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:37 smithi193 ceph-mon[33593]: osdmap e412: 8 total, 8 up, 8 in 2024-03-24T12:10:37.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:37 smithi193 ceph-mon[33593]: osdmap e413: 8 total, 8 up, 8 in 2024-03-24T12:10:37.851 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:37.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:37.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:37.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:38.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:38.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:10:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:10:38.107 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:38.134 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:38.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:38.236 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:38.264 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:38.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:38.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:38.366 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:38.394 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:38.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:38.485 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:10:38.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:38.523 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:38.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:38.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:38.642 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T12:10:38.642 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T12:10:38.642 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[42243]: pgmap v866: 146 pgs: 1 peering, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:38.642 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T12:10:38.642 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:38.643 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3080337171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:10:38.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T12:10:38.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T12:10:38.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[30223]: pgmap v866: 146 pgs: 1 peering, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:38.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T12:10:38.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:38.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3080337171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:10:38.651 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:38.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:38.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:38.777 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:38.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:38 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T12:10:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:38 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T12:10:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:38 smithi193 ceph-mon[33593]: pgmap v866: 146 pgs: 1 peering, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:38 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T12:10:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:38.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3080337171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:10:38.878 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:38.905 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:38.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:39.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:39.031 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:39.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:39.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:39.157 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:39.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:39.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:39.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:39.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:39.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:39.413 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:39.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:39.515 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:39.542 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:39.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:39.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:39.668 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:39.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: 1.9 scrub starts 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: 1.9 scrub ok 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3080337171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:10:39.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: osdmap e414: 8 total, 8 up, 8 in 2024-03-24T12:10:39.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3050945910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:39.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T12:10:39.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T12:10:39.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: 1.9 scrub starts 2024-03-24T12:10:39.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: 1.9 scrub ok 2024-03-24T12:10:39.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T12:10:39.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T12:10:39.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:39.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-24T12:10:39.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3080337171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:10:39.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: osdmap e414: 8 total, 8 up, 8 in 2024-03-24T12:10:39.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:39 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3050945910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:39.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:39.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:10:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:39.797 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:39.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:39.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: 1.9 scrub starts 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: 1.9 scrub ok 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3080337171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: osdmap e414: 8 total, 8 up, 8 in 2024-03-24T12:10:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:39 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3050945910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:39.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:39.922 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:39.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:39.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:40.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:40.048 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:40.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:40.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:40.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:40.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:40.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:40.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:40.303 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:40.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:40.404 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:40.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:40.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:40.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:40.566 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:40.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:40.649 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:10:40.649 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:10:40.650 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:10:40.651 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:10:40.651 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:10:40.651 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:10:40.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:40.695 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:40.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:40.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:40 smithi064 ceph-mon[30223]: osdmap e415: 8 total, 8 up, 8 in 2024-03-24T12:10:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:40 smithi064 ceph-mon[30223]: pgmap v869: 146 pgs: 1 peering, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:40.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:40 smithi064 ceph-mon[42243]: osdmap e415: 8 total, 8 up, 8 in 2024-03-24T12:10:40.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:40 smithi064 ceph-mon[42243]: pgmap v869: 146 pgs: 1 peering, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:40.823 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:40.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:40.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:40 smithi193 ceph-mon[33593]: osdmap e415: 8 total, 8 up, 8 in 2024-03-24T12:10:40.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:40 smithi193 ceph-mon[33593]: pgmap v869: 146 pgs: 1 peering, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:40.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:40.952 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:40.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:41.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:41.077 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:41.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:41.179 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:41.208 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:41.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:41.310 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:41.338 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:41.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:41.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:41.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:41.552 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_4' created 2024-03-24T12:10:41.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:41.595 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:41.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:41.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:41.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:41.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:41.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:41 smithi193 ceph-mon[33593]: osdmap e416: 8 total, 8 up, 8 in 2024-03-24T12:10:41.851 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:41.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:41.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:41.884 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:41 smithi064 ceph-mon[30223]: osdmap e416: 8 total, 8 up, 8 in 2024-03-24T12:10:41.885 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:41 smithi064 ceph-mon[42243]: osdmap e416: 8 total, 8 up, 8 in 2024-03-24T12:10:41.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:41.982 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:41.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:42.084 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:42.112 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:42.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:42.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:10:42.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:42.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-24T12:10:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:42.253 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:42.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:42.340 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:42.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:42.384 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:42.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:42.486 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:42.515 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:42.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:42.617 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:42.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:42.745 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:42.771 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:42.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:42.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1421340670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:10:42.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: from='client.35837 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:10:42.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: pgmap v871: 145 pgs: 1 peering, 144 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: from='client.35837 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: osdmap e417: 8 total, 8 up, 8 in 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[42243]: osdmap e418: 8 total, 8 up, 8 in 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1421340670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:10:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: from='client.35837 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: pgmap v871: 145 pgs: 1 peering, 144 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: from='client.35837 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: osdmap e417: 8 total, 8 up, 8 in 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-24T12:10:42.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:42 smithi064 ceph-mon[30223]: osdmap e418: 8 total, 8 up, 8 in 2024-03-24T12:10:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1421340670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: from='client.35837 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: pgmap v871: 145 pgs: 1 peering, 144 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: from='client.35837 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: osdmap e417: 8 total, 8 up, 8 in 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:10:42.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:42.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-24T12:10:42.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:42 smithi193 ceph-mon[33593]: osdmap e418: 8 total, 8 up, 8 in 2024-03-24T12:10:42.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:42.898 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:42.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:42.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:43.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:43.128 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:43.153 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:43.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:43.255 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:43.279 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:43.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:43.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:43.449 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:43.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:43.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:43.577 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:43.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:43.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:10:43.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:43.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:43.805 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:43.834 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:43.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:43.932 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:43.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:43.963 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:43.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:44.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:44.088 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:44.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:44.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:44.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:44.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:44.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:44.342 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:44.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:44.443 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:44.472 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:44.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:44.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:44.597 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:44.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:44.699 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:44.727 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:44.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:44.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:44 smithi064 ceph-mon[30223]: pgmap v874: 161 pgs: 8 creating+peering, 8 unknown, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:44.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:44 smithi064 ceph-mon[30223]: osdmap e419: 8 total, 8 up, 8 in 2024-03-24T12:10:44.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:44 smithi064 ceph-mon[30223]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:10:44.770 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:44 smithi064 ceph-mon[42243]: pgmap v874: 161 pgs: 8 creating+peering, 8 unknown, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:44.770 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:44 smithi064 ceph-mon[42243]: osdmap e419: 8 total, 8 up, 8 in 2024-03-24T12:10:44.770 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:44 smithi064 ceph-mon[42243]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:10:44.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:44 smithi193 ceph-mon[33593]: pgmap v874: 161 pgs: 8 creating+peering, 8 unknown, 145 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:44 smithi193 ceph-mon[33593]: osdmap e419: 8 total, 8 up, 8 in 2024-03-24T12:10:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:44 smithi193 ceph-mon[33593]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:10:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:44.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:44.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:44.983 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:44.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:45.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:45.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:45.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:45.236 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:45.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:45.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:45.365 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:45.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:45.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:45.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:45.494 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:45.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:45.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:45.621 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:45.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:45.696 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:45.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:45.751 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:45.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:45.818 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:45 smithi064 ceph-mon[42243]: osdmap e420: 8 total, 8 up, 8 in 2024-03-24T12:10:45.819 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:45 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T12:10:45.819 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:45 smithi064 ceph-mon[30223]: osdmap e420: 8 total, 8 up, 8 in 2024-03-24T12:10:45.819 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:45 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T12:10:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:45 smithi193 ceph-mon[33593]: osdmap e420: 8 total, 8 up, 8 in 2024-03-24T12:10:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:45 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T12:10:45.853 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:45.880 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:45.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 421, 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-03-24T11:46:40.657065+0000", 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-03-24T12:10:45.550880+0000", 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-03-24T12:08:35.127870+0000", 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-03-24T11:51:43.667453+0000", 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5801984, 2024-03-24T12:10:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 19, 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 11, 2024-03-24T12:10:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:49:56.369674+0000", 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 13, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 11, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 11, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 11, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 12, 2024-03-24T12:10:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "1.d", 2024-03-24T12:10:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 420, 2024-03-24T12:10:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 419, 2024-03-24T12:10:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 419, 2024-03-24T12:10:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "83'130", 2024-03-24T12:10:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "83'130" 2024-03-24T12:10:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "421", 2024-03-24T12:10:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "421", 2024-03-24T12:10:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "421", 2024-03-24T12:10:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:45.895 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:45.895 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:45.895 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:45.895 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:45.895 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:10:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-03-24T12:10:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-03-24T12:10:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-03-24T12:10:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.4600000381469727, 2024-03-24T12:10:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.4600000381469727, 2024-03-24T12:10:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.4600000381469727, 2024-03-24T12:10:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.4600000381469727, 2024-03-24T12:10:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-03-24T12:10:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".rgw.root", 2024-03-24T12:10:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:13.871974+0000", 2024-03-24T12:10:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "68", 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.5, 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.5, 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.5, 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.5, 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "datapool", 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:14.117018+0000", 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 3, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 3, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 3, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 3, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 3, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "71", 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 2, 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 71, 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-03-24T12:10:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.6700000762939453, 2024-03-24T12:10:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.6700000762939453, 2024-03-24T12:10:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-03-24T12:10:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T12:10:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T12:10:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 4, 2024-03-24T12:10:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:10:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:15.453744+0000", 2024-03-24T12:10:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:45.928 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:45.928 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:45.928 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:45.928 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:45.928 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:45.928 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:10:45.928 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:10:45.929 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:10:45.929 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:10:45.929 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:10:45.929 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:45.929 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:45.929 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:45.929 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:45.930 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:45.930 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:45.930 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:45.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.930 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "70", 2024-03-24T12:10:45.930 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:45.931 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:45.931 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:45.931 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:45.931 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:45.931 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:45.931 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:45.932 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:45.932 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:45.932 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:45.932 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:45.932 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:45.932 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:45.932 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:45.933 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:45.933 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:45.933 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:45.933 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:45.933 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:45.933 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:45.933 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:45.934 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:45.934 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:45.934 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:45.934 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:45.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:45.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.935 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:45.935 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:45.935 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:45.935 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:45.935 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:45.935 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:45.935 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:45.936 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:45.936 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:45.936 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:45.936 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:45.936 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:45.936 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:45.936 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:10:45.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.937 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:45.937 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:45.937 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.25, 2024-03-24T12:10:45.937 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.25, 2024-03-24T12:10:45.937 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:45.937 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.25, 2024-03-24T12:10:45.938 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.25, 2024-03-24T12:10:45.938 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:45.938 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:45.938 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:45.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.939 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 5, 2024-03-24T12:10:45.939 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:10:45.939 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:17.611004+0000", 2024-03-24T12:10:45.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:45.940 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:45.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:45.942 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:45.942 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:10:45.942 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:10:45.942 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:10:45.942 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:10:45.942 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "72", 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:45.943 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:45.944 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:45.945 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:45.946 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:45.947 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:45.947 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:45.947 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:45.947 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:45.947 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:45.947 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.75, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.75, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:45.948 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 6, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:19.701416+0000", 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:45.949 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:45.950 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:45.950 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:45.950 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:45.950 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:10:45.950 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:10:45.950 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.951 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "74", 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:45.952 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:45.953 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:45.953 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:45.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:45.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:45.953 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:45.953 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:45.953 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:45.954 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:45.954 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:45.954 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:45.954 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:45.954 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:45.954 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:45.954 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:45.955 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_bias": 4 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:45.956 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:45.957 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2, 2024-03-24T12:10:45.957 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2, 2024-03-24T12:10:45.957 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:45.957 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T12:10:45.957 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T12:10:45.957 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:45.957 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:45.957 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:45.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.958 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 11, 2024-03-24T12:10:45.958 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "unique_pool_4", 2024-03-24T12:10:45.958 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T12:10:41.539113+0000", 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:10:45.959 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:10:45.960 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:10:45.960 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:10:45.960 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:10:45.960 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 16, 2024-03-24T12:10:45.960 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 16, 2024-03-24T12:10:45.960 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 16, 2024-03-24T12:10:45.960 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 16, 2024-03-24T12:10:45.960 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 16, 2024-03-24T12:10:45.961 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:10:45.961 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:10:45.961 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:10:45.961 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:10:45.962 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:10:45.962 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:10:45.962 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:10:45.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.962 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "419", 2024-03-24T12:10:45.962 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:10:45.962 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:10:45.963 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:10:45.963 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:10:45.963 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:10:45.963 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:10:45.963 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:10:45.963 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:10:45.963 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:10:45.963 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:10:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:10:45.965 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:10:45.965 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:10:45.965 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:10:45.965 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:10:45.965 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:10:45.965 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:10:45.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:10:45.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.966 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:10:45.966 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:10:45.966 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:10:45.966 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:10:45.966 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:10:45.966 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:10:45.966 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": {}, 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:10:45.967 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3, 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3, 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 3, 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 3, 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.969 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:45.969 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:10:45.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.969 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:10:45.969 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d", 2024-03-24T12:10:45.969 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:45.969 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:45.970 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:45.970 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:45.970 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:45.970 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:45.970 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 8, 2024-03-24T12:10:45.970 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:10:45.970 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:45.970 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:45.972 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:10:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/2675992571", 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6802/2675992571", 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6804/2675992571", 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6803/2675992571", 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.974 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9", 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.2537689208984375, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 385, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:10:45.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6805/841667271", 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6806/841667271", 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6808/841667271", 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6807/841667271", 2024-03-24T12:10:45.978 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b", 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.3165130615234375, 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.90252685546875, 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:45.979 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 18, 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-03-24T12:10:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.982 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/435147390", 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6810/435147390", 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6812/435147390", 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6811/435147390", 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479", 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:45.983 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 27, 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.984 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.986 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6813/3355318038", 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6814/3355318038", 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6816/3355318038", 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6815/3355318038", 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b", 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:45.987 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 33, 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 420, 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6800", 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6801", 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6803", 2024-03-24T12:10:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6802", 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6800/3851339176", 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6801/3851339176", 2024-03-24T12:10:45.990 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3851339176", 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3851339176", 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6", 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 41, 2024-03-24T12:10:45.991 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 317, 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 353, 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 318, 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6804", 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:45.992 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6805", 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6807", 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:10:45.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6806", 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6804/1531860860", 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6805/1531860860", 2024-03-24T12:10:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6807/1531860860", 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6806/1531860860", 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996", 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:45.995 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 51, 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6808", 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6809", 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6811", 2024-03-24T12:10:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6810", 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6808/2408237278", 2024-03-24T12:10:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6809/2408237278", 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6811/2408237278", 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6810/2408237278", 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015", 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:10:45.999 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 61, 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6812", 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6813", 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6815", 2024-03-24T12:10:46.001 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6814", 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6812/2649989609", 2024-03-24T12:10:46.002 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6813/2649989609", 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6815/2649989609", 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6814/2649989609", 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:54.704535+0000", 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:46.893830+0000", 2024-03-24T12:10:46.004 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:50.684279+0000", 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:46.005 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:10:34.546015+0000", 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:46.006 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:10:41.332483+0000", 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:40.102762+0000", 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 317 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:10:46.007 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:51.284967+0000", 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:10:46.008 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:09:45.580900+0000", 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.0", 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.009 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [ 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.b", 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.4", 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 5 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.9", 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.1d", 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.6", 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:10:46.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.13", 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.18", 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:46.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 4, 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.19", 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 6 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.014 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": { 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/3538503390": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1277626303": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3407451558": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3753895773": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/842059688": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1610565583": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/2730170969": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3257265142": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1459540658": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/629947599": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:10:46.015 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/520335082": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3351953651": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/4027721305": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/4028269665": "2024-03-25T11:47:36.035658+0000" 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:46.016 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:46.017 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:10:46.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:46.046 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:46.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:46.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:46.176 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:46.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:46.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:46.303 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:46.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:46.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:46.430 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:46.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:46.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:46.559 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:46.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:46.662 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:46.690 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:46.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:46.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:46.817 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:46.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:46.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:46 smithi193 ceph-mon[33593]: pgmap v877: 160 pgs: 8 creating+peering, 6 unknown, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:46.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:46 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-24T12:10:46.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:46 smithi193 ceph-mon[33593]: osdmap e421: 8 total, 8 up, 8 in 2024-03-24T12:10:46.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:46 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3682613183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:10:46.901 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:46 smithi064 ceph-mon[30223]: pgmap v877: 160 pgs: 8 creating+peering, 6 unknown, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:46.901 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:46 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-24T12:10:46.901 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:46 smithi064 ceph-mon[30223]: osdmap e421: 8 total, 8 up, 8 in 2024-03-24T12:10:46.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:46 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3682613183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:10:46.902 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:46 smithi064 ceph-mon[42243]: pgmap v877: 160 pgs: 8 creating+peering, 6 unknown, 146 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:46.902 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:46 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-24T12:10:46.902 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:46 smithi064 ceph-mon[42243]: osdmap e421: 8 total, 8 up, 8 in 2024-03-24T12:10:46.902 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:46 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3682613183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:10:46.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:46.947 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:46.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:47.049 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:47.076 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:47.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:47.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:47.203 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:47.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:47.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:47.334 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:47.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:47.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:47.459 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:47.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:47.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:10:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:47.561 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:47.591 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:47.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:47.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:47.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:47.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:47 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:10:47.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:47 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:10:47.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:47 smithi193 ceph-mon[33593]: osdmap e422: 8 total, 8 up, 8 in 2024-03-24T12:10:47.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:47 smithi193 ceph-mon[33593]: osdmap e423: 8 total, 8 up, 8 in 2024-03-24T12:10:47.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:47 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3707213325' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:10:47.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:47 smithi193 ceph-mon[33593]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:10:47.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:47 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T12:10:47.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:47.845 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:47.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:47.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:47.970 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-24T12:10:47.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[30223]: osdmap e422: 8 total, 8 up, 8 in 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[30223]: osdmap e423: 8 total, 8 up, 8 in 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3707213325' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[30223]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:10:47.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[42243]: osdmap e422: 8 total, 8 up, 8 in 2024-03-24T12:10:47.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[42243]: osdmap e423: 8 total, 8 up, 8 in 2024-03-24T12:10:47.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3707213325' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:10:47.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[42243]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:10:47.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:47 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T12:10:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:47.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:48.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:48.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:10:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:10:48.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:48.105 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:48.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:48.172 INFO:tasks.thrashosds.thrasher:j is {'epoch': 421, 'fsid': '06f23456-e9d4-11ee-95cd-87774f69a715', 'created': '2024-03-24T11:46:40.657065+0000', 'modified': '2024-03-24T12:10:45.550880+0000', 'last_up_change': '2024-03-24T12:08:35.127870+0000', 'last_in_change': '2024-03-24T11:51:43.667453+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:49:56.369674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 12, 'last_pg_merge_meta': {'source_pgid': '1.d', 'ready_epoch': 420, 'last_epoch_started': 419, 'last_epoch_clean': 419, 'source_version': "83'130", 'target_version': "83'130"}, 'last_change': '421', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '421', 'last_force_op_resend_preluminous': '421', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4600000381469727, 'score_stable': 2.4600000381469727, 'optimal_score': 1, 'raw_score_acting': 2.4600000381469727, 'raw_score_stable': 2.4600000381469727, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T11:52:13.871974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-24T11:52:14.117018+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T11:52:15.453744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T11:52:17.611004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T11:52:19.701416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-24T12:10:41.539113+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '419', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'bc14a64f-9e6c-41f1-901e-c49fabffb53d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 2675992571}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6802', 'nonce': 2675992571}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6804', 'nonce': 2675992571}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 2675992571}]}, 'public_addr': '172.21.15.64:6801/2675992571', 'cluster_addr': '172.21.15.64:6802/2675992571', 'heartbeat_back_addr': '172.21.15.64:6804/2675992571', 'heartbeat_front_addr': '172.21.15.64:6803/2675992571', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '74587332-f0b2-4905-9888-7bdb604cc0e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2537689208984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 841667271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6806', 'nonce': 841667271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6808', 'nonce': 841667271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 841667271}]}, 'public_addr': '172.21.15.64:6805/841667271', 'cluster_addr': '172.21.15.64:6806/841667271', 'heartbeat_back_addr': '172.21.15.64:6808/841667271', 'heartbeat_front_addr': '172.21.15.64:6807/841667271', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a0c0ed85-9f33-4198-a1f2-447eb17c364b', 'up': 1, 'in': 1, 'weight': 0.3165130615234375, 'primary_affinity': 0.90252685546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 435147390}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6810', 'nonce': 435147390}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6812', 'nonce': 435147390}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 435147390}]}, 'public_addr': '172.21.15.64:6809/435147390', 'cluster_addr': '172.21.15.64:6810/435147390', 'heartbeat_back_addr': '172.21.15.64:6812/435147390', 'heartbeat_front_addr': '172.21.15.64:6811/435147390', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1396f66c-3e0c-4a01-b0a7-6f763989a479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3355318038}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6814', 'nonce': 3355318038}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6816', 'nonce': 3355318038}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3355318038}]}, 'public_addr': '172.21.15.64:6813/3355318038', 'cluster_addr': '172.21.15.64:6814/3355318038', 'heartbeat_back_addr': '172.21.15.64:6816/3355318038', 'heartbeat_front_addr': '172.21.15.64:6815/3355318038', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50d65d8b-3e90-46a4-b19d-50c3015af40b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3851339176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3851339176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3851339176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3851339176}]}, 'public_addr': '172.21.15.193:6800/3851339176', 'cluster_addr': '172.21.15.193:6801/3851339176', 'heartbeat_back_addr': '172.21.15.193:6803/3851339176', 'heartbeat_front_addr': '172.21.15.193:6802/3851339176', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '50c5d50e-7e68-487c-9f68-1abb8c927ff6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 317, 'up_from': 353, 'up_thru': 417, 'down_at': 318, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 1531860860}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 1531860860}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 1531860860}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 1531860860}]}, 'public_addr': '172.21.15.193:6804/1531860860', 'cluster_addr': '172.21.15.193:6805/1531860860', 'heartbeat_back_addr': '172.21.15.193:6807/1531860860', 'heartbeat_front_addr': '172.21.15.193:6806/1531860860', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7e74fe39-5707-4c90-ad38-1083cd7a6996', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 2408237278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 2408237278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 2408237278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 2408237278}]}, 'public_addr': '172.21.15.193:6808/2408237278', 'cluster_addr': '172.21.15.193:6809/2408237278', 'heartbeat_back_addr': '172.21.15.193:6811/2408237278', 'heartbeat_front_addr': '172.21.15.193:6810/2408237278', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e568df64-cc9a-43f8-a432-8628d1da1015', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 2649989609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 2649989609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 2649989609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 2649989609}]}, 'public_addr': '172.21.15.193:6812/2649989609', 'cluster_addr': '172.21.15.193:6813/2649989609', 'heartbeat_back_addr': '172.21.15.193:6815/2649989609', 'heartbeat_front_addr': '172.21.15.193:6814/2649989609', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:54.704535+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:46.893830+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:50.684279+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:10:34.546015+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:10:41.332483+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:40.102762+0000', 'dead_epoch': 317}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:51.284967+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:09:45.580900+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [0, 1, 7]}], 'pg_upmap_items': [{'pgid': '2.b', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.6', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.64:6800/3538503390': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/1277626303': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3407451558': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3753895773': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/842059688': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/1610565583': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/2730170969': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3257265142': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/1459540658': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/629947599': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:6800/520335082': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3351953651': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:6800/4027721305': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/4028269665': '2024-03-25T11:47:36.035658+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:10:48.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:10:48.206 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:48.234 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:48.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:48.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:48.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:48.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:48.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:48.491 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:48.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:48.593 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:48.621 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:48.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:48.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:48.750 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:48.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:48.851 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:48.879 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:48.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:48.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:48.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:49.007 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:49.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:49.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:49.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[30223]: pgmap v881: 159 pgs: 1 peering, 158 active+clean; 262 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[30223]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[30223]: osdmap e424: 8 total, 8 up, 8 in 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[42243]: pgmap v881: 159 pgs: 1 peering, 158 active+clean; 262 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[42243]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T12:10:49.115 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:48 smithi064 ceph-mon[42243]: osdmap e424: 8 total, 8 up, 8 in 2024-03-24T12:10:49.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:49.136 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:49.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:49.237 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:49.266 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:49.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:48 smithi193 ceph-mon[33593]: pgmap v881: 159 pgs: 1 peering, 158 active+clean; 262 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:10:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:48 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:48 smithi193 ceph-mon[33593]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:10:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:48 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T12:10:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:48 smithi193 ceph-mon[33593]: osdmap e424: 8 total, 8 up, 8 in 2024-03-24T12:10:49.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:49.393 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:49.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:49.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:49.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:49.518 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:49.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:49.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:49.648 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:49.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:49.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:49.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:49.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:49.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:49.899 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:49.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:50.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:50.024 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:50.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:50.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:49 smithi064 ceph-mon[30223]: osdmap e425: 8 total, 8 up, 8 in 2024-03-24T12:10:50.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:49 smithi064 ceph-mon[42243]: osdmap e425: 8 total, 8 up, 8 in 2024-03-24T12:10:50.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:50.149 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:50.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:50.153 DEBUG:teuthology.orchestra.run.smithi064:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-24T12:10:50.192 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T12:10:50.193 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T12:10:50.251 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:50.280 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:50.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:50.316 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:hints = 1 2024-03-24T12:10:50.316 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T12:10:50.316 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object prefix: benchmark_data_smithi064_206665 2024-03-24T12:10:50.317 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:10:50.317 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 0 0 0 0 0 0 - 0 2024-03-24T12:10:50.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:49 smithi193 ceph-mon[33593]: osdmap e425: 8 total, 8 up, 8 in 2024-03-24T12:10:50.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:50.421 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:50.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:50.523 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:50.558 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:50.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:50.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:50.699 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:50.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:50.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:50.838 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:50.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:50.940 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:50.970 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:50.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:51.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:50 smithi064 ceph-mon[30223]: pgmap v884: 159 pgs: 1 peering, 158 active+clean; 262 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:10:51.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:50 smithi064 ceph-mon[30223]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:10:51.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:50 smithi064 ceph-mon[30223]: osdmap e426: 8 total, 8 up, 8 in 2024-03-24T12:10:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:50 smithi064 ceph-mon[42243]: pgmap v884: 159 pgs: 1 peering, 158 active+clean; 262 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:10:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:50 smithi064 ceph-mon[42243]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:10:51.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:50 smithi064 ceph-mon[42243]: osdmap e426: 8 total, 8 up, 8 in 2024-03-24T12:10:51.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:51.107 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:10:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:10:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:10:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:10:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35033.0:144410 8.17 8:eb4d00be:::benchmark_data_smithi064_120708_object22384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-03-24T12:10:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 598.437158963, 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17982219199999999, 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35033", 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3758901177", 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 144410 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666464+0000", 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666465+0000", 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.615e-06 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666470+0000", 2024-03-24T12:10:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2420000000000002e-06 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666477+0000", 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6329999999999996e-06 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.666484+0000", 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2450000000000003e-06 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669073+0000", 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002589755 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669197+0000", 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012349999999999999 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.669550+0000", 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035300000000000002 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.710796+0000", 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041246370999999997 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711483+0000", 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068668600000000005 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711540+0000", 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7515000000000001e-05 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.711848+0000", 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030785799999999999 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.845781+0000", 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13393276200000001 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.845858+0000", 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7063000000000004e-05 2024-03-24T12:10:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:00:52.846286+0000", 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042794699999999997 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:141226 10.2 10:411c0e54:::benchmark_data_smithi064_174908_object141225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:06.900287+0000", 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.203335791, 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15928548100000001, 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141226 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900287+0000", 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900287+0000", 2024-03-24T12:10:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900289+0000", 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.339e-06 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900322+0000", 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2171000000000001e-05 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900337+0000", 2024-03-24T12:10:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5877000000000001e-05 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900347+0000", 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1050000000000001e-06 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.947992+0000", 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047645310000000003 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.948150+0000", 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015810299999999999 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.948361+0000", 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021125899999999999 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.954702+0000", 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063403089999999997 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.955498+0000", 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079653500000000002 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059413+0000", 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10391539499999999 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059485+0000", 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1997999999999997e-05 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059573+0000", 2024-03-24T12:10:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.708e-05 2024-03-24T12:10:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:141235 10.2 10:42255cb8:::benchmark_data_smithi064_174908_object141234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:06.914042+0000", 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.189581003, 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14571151800000001, 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141235 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.914042+0000", 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.914042+0000", 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.914045+0000", 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9890000000000001e-06 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.914090+0000", 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4836000000000002e-05 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.914116+0000", 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6523e-05 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.914131+0000", 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4986e-05 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.951072+0000", 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036940392000000002 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.951234+0000", 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016222599999999999 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.951424+0000", 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019052199999999999 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.955505+0000", 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040810680000000002 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.958937+0000", 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034315399999999999 2024-03-24T12:10:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059625+0000", 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100688258 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059695+0000", 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0124999999999997e-05 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059753+0000", 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8053000000000001e-05 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:142303 10.2 10:46c397f2:::benchmark_data_smithi064_174908_object142302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:07.820736+0000", 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.28288642199999, 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15797519400000001, 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 142303 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820736+0000", 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820736+0000", 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820739+0000", 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4169999999999999e-06 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820765+0000", 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6361000000000001e-05 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820781+0000", 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.558e-05 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820791+0000", 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7599999999999997e-06 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.853298+0000", 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032507340000000003 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.853631+0000", 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033304500000000002 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.854099+0000", 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000468269 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.858231+0000", 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041313319999999997 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.861699+0000", 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034681320000000001 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.978359+0000", 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116660629 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.978531+0000", 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017170099999999999 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.978712+0000", 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018062800000000001 2024-03-24T12:10:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:143342 10.2 10:46640381:::benchmark_data_smithi064_174908_object143341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:08.758084+0000", 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.345538944, 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15685858599999999, 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143342 2024-03-24T12:10:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758084+0000", 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758084+0000", 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758086+0000", 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.181e-06 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758113+0000", 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6704e-05 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758126+0000", 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3035e-05 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758136+0000", 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.805263+0000", 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047127304000000002 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.805429+0000", 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016577200000000001 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.805602+0000", 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017333100000000001 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.813177+0000", 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075751949999999998 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.813218+0000", 2024-03-24T12:10:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0893000000000003e-05 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.914242+0000", 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101023847 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.914412+0000", 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016971 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.914942+0000", 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053054199999999995 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:147067 10.2 10:4040c259:::benchmark_data_smithi064_174908_object2650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:12.081571+0000", 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.02205202100001, 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.160370979, 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147067 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081571+0000", 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081571+0000", 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081573+0000", 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8199999999999999e-06 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081577+0000", 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5589999999999999e-06 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081586+0000", 2024-03-24T12:10:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0969999999999995e-06 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081594+0000", 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.644e-06 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082124+0000", 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053023299999999996 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082237+0000", 2024-03-24T12:10:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112945 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082367+0000", 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130149 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.090336+0000", 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079686170000000008 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.091486+0000", 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011497689999999999 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.241833+0000", 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.150347017 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.241893+0000", 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0747e-05 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.241942+0000", 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8381999999999999e-05 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:147070 10.2 10:45ca98de:::benchmark_data_smithi064_174908_object2653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:12.082601+0000", 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.02102151700001, 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.159632149, 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147070 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082601+0000", 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082601+0000", 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082603+0000", 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.31e-06 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082607+0000", 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5870000000000003e-06 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082618+0000", 2024-03-24T12:10:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:51.149 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:51.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0579e-05 2024-03-24T12:10:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082625+0000", 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5129999999999999e-06 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.084948+0000", 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023222579999999998 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.085044+0000", 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6760000000000002e-05 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.085119+0000", 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4929e-05 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.090506+0000", 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053868950000000001 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.091542+0000", 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001036326 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242074+0000", 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.150531058 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242136+0000", 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2284000000000002e-05 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242233+0000", 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7650000000000002e-05 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:147072 10.2 10:4cd5e450:::benchmark_data_smithi064_174908_object2655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:12.082637+0000", 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.02098624000001, 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15983945199999999, 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147072 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082637+0000", 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082637+0000", 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082637+0000", 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4799999999999998e-07 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082640+0000", 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3039999999999999e-06 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082645+0000", 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3299999999999998e-06 2024-03-24T12:10:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082650+0000", 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1580000000000004e-06 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.087176+0000", 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004526408 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.087286+0000", 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010958700000000001 2024-03-24T12:10:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.087401+0000", 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115451 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.090512+0000", 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031100939999999999 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.091629+0000", 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001117844 2024-03-24T12:10:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242362+0000", 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.150732586 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242414+0000", 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2225999999999998e-05 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242476+0000", 2024-03-24T12:10:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1816000000000006e-05 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:147094 10.2 10:4b1eeff5:::benchmark_data_smithi064_174908_object2677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:12.099027+0000", 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.00459613199999, 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.143741747, 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147094 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.099027+0000", 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.099027+0000", 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.099029+0000", 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1500000000000002e-06 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.099035+0000", 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5130000000000002e-06 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.099049+0000", 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4032000000000001e-05 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.099059+0000", 2024-03-24T12:10:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9389999999999992e-06 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.103068+0000", 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040083000000000002 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.103183+0000", 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011496400000000001 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.103297+0000", 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011466100000000001 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.107934+0000", 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046369109999999996 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.111152+0000", 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032179470000000001 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242659+0000", 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13150652199999999 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242712+0000", 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3594000000000002e-05 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242768+0000", 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6214000000000001e-05 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:229477 10.2 10:40483315:::benchmark_data_smithi064_174908_object85060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:25.314287+0000", 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.789335665999999, 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073178139000000003, 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 229477 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314287+0000", 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314287+0000", 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314291+0000", 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3639999999999999e-06 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314299+0000", 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0649999999999999e-06 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314315+0000", 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6753999999999999e-05 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314326+0000", 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0523000000000001e-05 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314390+0000", 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4227999999999996e-05 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314509+0000", 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118669 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.314619+0000", 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110197 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.318421+0000", 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038015950000000001 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.319210+0000", 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078907599999999999 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.387355+0000", 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.068145265999999996 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.387414+0000", 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9332e-05 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:25.387465+0000", 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1069999999999997e-05 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231186 10.2 10:49235045:::benchmark_data_smithi064_174908_object86769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:10:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:26.942298+0000", 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.161324358000002, 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.070609384999999997, 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231186 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942298+0000", 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942298+0000", 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942302+0000", 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8569999999999997e-06 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942314+0000", 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1732999999999999e-05 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942335+0000", 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1288999999999999e-05 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942350+0000", 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4853e-05 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942421+0000", 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0767999999999997e-05 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942525+0000", 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010380800000000001 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.942609+0000", 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3826999999999995e-05 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.945007+0000", 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023986929999999999 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:26.945914+0000", 2024-03-24T12:10:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090676400000000001 2024-03-24T12:10:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.012774+0000", 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066860365000000005 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.012848+0000", 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3773e-05 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.012908+0000", 2024-03-24T12:10:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9654999999999997e-05 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231570 10.2 10:4004fd04:::benchmark_data_smithi064_174908_object87153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.478354+0000", 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.625269044999996, 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12149012300000001, 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231570 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478354+0000", 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478354+0000", 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478356+0000", 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0509999999999999e-06 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478361+0000", 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5849999999999999e-06 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478374+0000", 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2232000000000001e-05 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478382+0000", 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2269999999999993e-06 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478442+0000", 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0291999999999998e-05 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478558+0000", 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115735 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.478666+0000", 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010815900000000001 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.481245+0000", 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025791130000000001 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.481538+0000", 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029296699999999999 2024-03-24T12:10:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.599744+0000", 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118206039 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.599799+0000", 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4426999999999999e-05 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.599844+0000", 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5296000000000001e-05 2024-03-24T12:10:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:232008 10.2 10:4ec7c223:::benchmark_data_smithi064_174908_object87591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.934314+0000", 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.169309119000005, 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058375387000000001, 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 232008 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934314+0000", 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934314+0000", 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934317+0000", 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8600000000000001e-06 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934325+0000", 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6449999999999994e-06 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934353+0000", 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8195999999999999e-05 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934364+0000", 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0841e-05 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934439+0000", 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4270000000000006e-05 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934592+0000", 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015304899999999999 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.934721+0000", 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012916699999999999 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.937552+0000", 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028316410000000002 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.937937+0000", 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000384409 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.992527+0000", 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054590171999999999 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.992639+0000", 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112438 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.992689+0000", 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9699e-05 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:283971 10.2 10:4a19583e:::benchmark_data_smithi064_174908_object139554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:11.429374+0000", 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.674249058999997, 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.185421002, 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 283971 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429374+0000", 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429374+0000", 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429377+0000", 2024-03-24T12:10:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1980000000000001e-06 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429384+0000", 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3549999999999999e-06 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429400+0000", 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6112000000000001e-05 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429410+0000", 2024-03-24T12:10:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3850000000000008e-06 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429484+0000", 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3849000000000004e-05 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429629+0000", 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014504000000000001 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429750+0000", 2024-03-24T12:10:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121698 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614645+0000", 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18489487700000001 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614703+0000", 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8159999999999999e-05 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614715+0000", 2024-03-24T12:10:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1252e-05 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614752+0000", 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7679000000000003e-05 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614795+0000", 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2397000000000001e-05 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:283972 10.2 10:47bf4e9c:::benchmark_data_smithi064_174908_object139555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:11.429789+0000", 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.673833588000001, 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18520739, 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 283972 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429789+0000", 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429789+0000", 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429791+0000", 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.691e-06 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429796+0000", 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2490000000000001e-06 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429808+0000", 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1728e-05 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429816+0000", 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3690000000000007e-06 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.432627+0000", 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028106009999999998 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611971+0000", 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17934376599999999 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.612103+0000", 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132075 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614841+0000", 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027382499999999998 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614907+0000", 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5557999999999998e-05 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614920+0000", 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2979000000000001e-05 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614960+0000", 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0302000000000001e-05 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614997+0000", 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6822000000000001e-05 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.15714.0:283971 10.2 e387/320 ondisk, result = 0)", 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:11.433904+0000", 2024-03-24T12:10:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.669718979000002, 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18075080800000001, 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433904+0000", 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433904+0000", 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433905+0000", 2024-03-24T12:10:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.584e-06 2024-03-24T12:10:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433911+0000", 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4700000000000001e-06 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611801+0000", 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17789015399999999 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611818+0000", 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7056000000000001e-05 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614630+0000", 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028122820000000001 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614638+0000", 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9719999999999992e-06 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614655+0000", 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6290000000000002e-05 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:283977 10.0 e387/320 10:0061538f:::benchmark_data_smithi064_174908_object139560:head v 387'17936, mlcod=387'17934)", 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:11.433964+0000", 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.669658384999998, 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18580016399999999, 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433964+0000", 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433964+0000", 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433966+0000", 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.051e-06 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433972+0000", 2024-03-24T12:10:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5479999999999998e-06 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611807+0000", 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17783502200000001 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611818+0000", 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1031e-05 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611888+0000", 2024-03-24T12:10:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9573999999999995e-05 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611911+0000", 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3061000000000002e-05 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.619732+0000", 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078214849999999995 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.619765+0000", 2024-03-24T12:10:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2391999999999998e-05 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:287187 10.2 10:4eeba74a:::benchmark_data_smithi064_174908_object142770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:13.965536+0000", 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.138087145, 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023849263999999998, 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 287187 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.965536+0000", 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.965536+0000", 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.965537+0000", 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.48e-06 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.965542+0000", 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2800000000000003e-06 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.965558+0000", 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5143999999999999e-05 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.965566+0000", 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8599999999999999e-06 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.969456+0000", 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038897620000000002 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.969708+0000", 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025206499999999999 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.969970+0000", 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000261661 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.973392+0000", 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034225169999999999 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.973586+0000", 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019360300000000001 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.989275+0000", 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015689063 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.989336+0000", 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0528000000000001e-05 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:13.989385+0000", 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9301000000000001e-05 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:287716 10.2 10:4fef385a:::benchmark_data_smithi064_174908_object143299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:14.370966+0000", 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.732656910000003, 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023731733000000001, 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 287716 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.370966+0000", 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.370966+0000", 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.370967+0000", 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-03-24T12:10:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.370969+0000", 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6350000000000002e-06 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.370975+0000", 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5770000000000001e-06 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.370980+0000", 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2379999999999997e-06 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.387009+0000", 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016028954000000002 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.387159+0000", 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150328 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.387372+0000", 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000212957 2024-03-24T12:10:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.393183+0000", 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058105680000000003 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.393206+0000", 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3017999999999999e-05 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.394605+0000", 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013994389999999999 2024-03-24T12:10:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.394657+0000", 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1575e-05 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.394698+0000", 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0834999999999998e-05 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:288503 10.2 10:459c3ae2:::benchmark_data_smithi064_174908_object144086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:14.945115+0000", 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.158508015000002, 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025704841999999999, 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 288503 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:10:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.945115+0000", 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.945115+0000", 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.945117+0000", 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.491e-06 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:10:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.945126+0000", 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3639999999999999e-06 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.945141+0000", 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5009e-05 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.945150+0000", 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6919999999999997e-06 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:10:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.955765+0000", 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010614342000000001 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.955904+0000", 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000139758 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.956086+0000", 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018145800000000001 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.959639+0000", 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035529229999999999 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.965319+0000", 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005679697 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.970650+0000", 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053310170000000004 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.970764+0000", 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114806 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:14.970820+0000", 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5284999999999997e-05 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:10:51.226 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:10:51.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:51.285 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:51.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:51.317 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 1 16 3306 3290 205.613 205.625 0.00582325 0.00484908 2024-03-24T12:10:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:50 smithi193 ceph-mon[33593]: pgmap v884: 159 pgs: 1 peering, 158 active+clean; 262 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:10:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:50 smithi193 ceph-mon[33593]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:10:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:50 smithi193 ceph-mon[33593]: osdmap e426: 8 total, 8 up, 8 in 2024-03-24T12:10:51.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:51.430 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:51.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:51.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:51.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:51.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:51.714 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:51.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:51.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:51.854 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:51.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:51.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:51.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:52.011 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:10:52.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:52.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:52.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:52.271 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:52.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:52.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:52 smithi064 ceph-mon[30223]: osdmap e427: 8 total, 8 up, 8 in 2024-03-24T12:10:52.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3381509016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:52.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:52 smithi064 ceph-mon[30223]: from='client.25849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:52.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:52 smithi064 ceph-mon[42243]: osdmap e427: 8 total, 8 up, 8 in 2024-03-24T12:10:52.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:52 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3381509016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:52.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:52 smithi064 ceph-mon[42243]: from='client.25849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:52.317 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 2 16 6338 6322 197.548 189.5 0.00516306 0.00505256 2024-03-24T12:10:52.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:52 smithi193 ceph-mon[33593]: osdmap e427: 8 total, 8 up, 8 in 2024-03-24T12:10:52.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:52 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3381509016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:52.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:52 smithi193 ceph-mon[33593]: from='client.25849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:10:52.374 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:52.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:52.412 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:52.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:52.513 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:52.550 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:52.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:52.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:52.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:52.792 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:52.825 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:52.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:52.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:52.965 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:52.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:53.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:53.115 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:53.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:53.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:10:53.216 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:53.249 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:53.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:53.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:53 smithi064 ceph-mon[30223]: pgmap v887: 158 pgs: 158 active+clean; 75 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 19 MiB/s wr, 299 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-24T12:10:53.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:53 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:53.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:53 smithi064 ceph-mon[30223]: from='client.25849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:10:53.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:53 smithi064 ceph-mon[30223]: osdmap e428: 8 total, 8 up, 8 in 2024-03-24T12:10:53.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:53 smithi064 ceph-mon[42243]: pgmap v887: 158 pgs: 158 active+clean; 75 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 19 MiB/s wr, 299 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-24T12:10:53.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:53 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:53.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:53 smithi064 ceph-mon[42243]: from='client.25849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:10:53.264 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:53 smithi064 ceph-mon[42243]: osdmap e428: 8 total, 8 up, 8 in 2024-03-24T12:10:53.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:53 smithi193 ceph-mon[33593]: pgmap v887: 158 pgs: 158 active+clean; 75 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 19 MiB/s wr, 299 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-24T12:10:53.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:53 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:10:53.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:53 smithi193 ceph-mon[33593]: from='client.25849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:10:53.264 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:53 smithi193 ceph-mon[33593]: osdmap e428: 8 total, 8 up, 8 in 2024-03-24T12:10:53.317 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 3 16 9512 9496 197.816 198.375 0.003212 0.00504587 2024-03-24T12:10:53.352 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:53.388 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:53.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:53.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:53.530 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:53.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:53.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:53.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:53.681 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:53.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:53.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:53.786 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:53.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:53.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:53.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:53.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:53.959 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:53.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:54.061 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:54.097 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:54.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:54.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:54.238 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:54.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:54.317 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 4 16 12346 12330 192.639 177.125 0.00509537 0.00518364 2024-03-24T12:10:54.340 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:54.376 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:54.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:54.478 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:54.517 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:54.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:54.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:54.659 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:54.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:54.760 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:54.798 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:54.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:54.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:54.942 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:54.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:55.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:55.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:55.185 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:55.222 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:55.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:55.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:55 smithi064 ceph-mon[42243]: pgmap v889: 158 pgs: 158 active+clean; 222 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 780 op/s; 70 KiB/s, 0 objects/s recovering 2024-03-24T12:10:55.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:55 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:10:55.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:55 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:10:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:55 smithi064 ceph-mon[30223]: pgmap v889: 158 pgs: 158 active+clean; 222 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 780 op/s; 70 KiB/s, 0 objects/s recovering 2024-03-24T12:10:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:55 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:10:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:55 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:10:55.317 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 5 16 15110 15094 188.658 172.75 0.00532696 0.00529446 2024-03-24T12:10:55.326 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:55 smithi193 ceph-mon[33593]: pgmap v889: 158 pgs: 158 active+clean; 222 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 780 op/s; 70 KiB/s, 0 objects/s recovering 2024-03-24T12:10:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:55 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:10:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:55 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:10:55.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:55.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:55.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:55.501 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:55.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:55.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:55.640 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:55.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:55.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:55.783 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:55.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:55.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:10:55.924 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:55.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:10:56.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:56.061 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:56.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:56.101 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:56 smithi064 ceph-mon[30223]: osdmap e429: 8 total, 8 up, 8 in 2024-03-24T12:10:56.102 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:56 smithi064 ceph-mon[42243]: osdmap e429: 8 total, 8 up, 8 in 2024-03-24T12:10:56.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:56.207 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:56.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:56.276 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:56 smithi193 ceph-mon[33593]: osdmap e429: 8 total, 8 up, 8 in 2024-03-24T12:10:56.309 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:56.317 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 6 16 17752 17736 184.726 165.125 0.00354572 0.00540631 2024-03-24T12:10:56.342 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:56.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:56.444 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:56.479 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:56.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:56.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:56.618 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:56.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:56.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:56.760 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:56.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:56.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:56.901 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:56.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:57.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:57.051 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:57.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:57.153 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:57.190 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:57.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:57.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:57.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:57 smithi064 ceph-mon[30223]: pgmap v891: 158 pgs: 158 active+clean; 312 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 924 B/s rd, 56 MiB/s wr, 898 op/s; 58 KiB/s, 0 objects/s recovering 2024-03-24T12:10:57.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:57 smithi064 ceph-mon[42243]: pgmap v891: 158 pgs: 158 active+clean; 312 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 924 B/s rd, 56 MiB/s wr, 898 op/s; 58 KiB/s, 0 objects/s recovering 2024-03-24T12:10:57.318 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 7 16 20198 20182 180.174 152.875 0.0061152 0.00554394 2024-03-24T12:10:57.326 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:57.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:10:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:57 smithi193 ceph-mon[33593]: pgmap v891: 158 pgs: 158 active+clean; 312 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 924 B/s rd, 56 MiB/s wr, 898 op/s; 58 KiB/s, 0 objects/s recovering 2024-03-24T12:10:57.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:10:57.429 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:57.442 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:10:57.462 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:57.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:57.485 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:10:57.526 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:10:57.565 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:10:57.571 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:10:57.571 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:10:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:10:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:57.598 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:57.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:10:57.613 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:10:57.655 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:10:57.697 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:10:57.699 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:57.733 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:57.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:57.741 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:10:57.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:10:57.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:10:57.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:10:57.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:10:57.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:10:57.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:10:57.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:10:57.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:10:57.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:57.874 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:57.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:57.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:58.010 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:58.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:58.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:58.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:58 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/55316089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:58.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:58 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:10:58.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:58 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:10:58.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:10:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:10:58.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:58.144 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:58.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:58.246 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:58.281 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:58.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:58.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:58.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:58 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/55316089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:58.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:58 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:10:58.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:58 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:10:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:58 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/55316089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:10:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:58 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:10:58.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:58 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:10:58.318 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 8 16 22381 22365 174.706 136.438 0.00672873 0.00571733 2024-03-24T12:10:58.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:58.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:58.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:58.566 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:58.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:58.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:10:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:58.691 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:58.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:10:58.697 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:10:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:10:58.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:58.831 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:58.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:58.933 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:58.967 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:58.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:59.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:10:59.107 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:59.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:10:59.209 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:10:59.245 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:59.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:10:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:59 smithi064 ceph-mon[30223]: pgmap v892: 158 pgs: 158 active+clean; 1.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 168 MiB/s wr, 2.69k op/s; 49 KiB/s, 0 objects/s recovering 2024-03-24T12:10:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:10:59 smithi064 ceph-mon[30223]: osdmap e430: 8 total, 8 up, 8 in 2024-03-24T12:10:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:59 smithi064 ceph-mon[42243]: pgmap v892: 158 pgs: 158 active+clean; 1.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 168 MiB/s wr, 2.69k op/s; 49 KiB/s, 0 objects/s recovering 2024-03-24T12:10:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:10:59 smithi064 ceph-mon[42243]: osdmap e430: 8 total, 8 up, 8 in 2024-03-24T12:10:59.318 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 9 16 24617 24601 170.821 139.75 0.0119804 0.00584856 2024-03-24T12:10:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:59 smithi193 ceph-mon[33593]: pgmap v892: 158 pgs: 158 active+clean; 1.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 168 MiB/s wr, 2.69k op/s; 49 KiB/s, 0 objects/s recovering 2024-03-24T12:10:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:10:59 smithi193 ceph-mon[33593]: osdmap e430: 8 total, 8 up, 8 in 2024-03-24T12:10:59.347 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:10:59.381 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:10:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:59.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:10:59.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:10:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:59.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:10:59.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:59.670 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:59.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:59.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:59.809 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:59.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:10:59.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:10:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:10:59.951 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:10:59.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:00.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:00.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:00.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:00.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:00.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:00.318 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 10 16 26679 26663 166.625 128.875 0.00274888 0.00599551 2024-03-24T12:11:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:00 smithi193 ceph-mon[33593]: from='client.25861 v1:172.21.15.64:0/735258418' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:11:00.355 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:00.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:00.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:00.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:00.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:00 smithi064 ceph-mon[30223]: from='client.25861 v1:172.21.15.64:0/735258418' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:11:00.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:00 smithi064 ceph-mon[42243]: from='client.25861 v1:172.21.15.64:0/735258418' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:11:00.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:00.637 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:00.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:00.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:00.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:00.781 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:00.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:00.882 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:00.917 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:00.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:01.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:01.052 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:01.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:01.154 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:01.188 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:01.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:01.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:01.318 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 11 16 28805 28789 163.554 132.875 0.00623734 0.00610808 2024-03-24T12:11:01.330 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:01.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:01 smithi193 ceph-mon[33593]: pgmap v894: 158 pgs: 158 active+clean; 1.2 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 680 B/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-24T12:11:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:01 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:11:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:01 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:11:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:01 smithi193 ceph-mon[33593]: osdmap e431: 8 total, 8 up, 8 in 2024-03-24T12:11:01.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:01 smithi064 ceph-mon[30223]: pgmap v894: 158 pgs: 158 active+clean; 1.2 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 680 B/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-24T12:11:01.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:01 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:11:01.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:01 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:11:01.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:01 smithi064 ceph-mon[30223]: osdmap e431: 8 total, 8 up, 8 in 2024-03-24T12:11:01.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:01 smithi064 ceph-mon[42243]: pgmap v894: 158 pgs: 158 active+clean; 1.2 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 680 B/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-24T12:11:01.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:01 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:11:01.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:01 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:11:01.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:01 smithi064 ceph-mon[42243]: osdmap e431: 8 total, 8 up, 8 in 2024-03-24T12:11:01.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:01.468 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:01.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:01.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:01.607 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:01.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:01.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:01.744 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:01.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:01.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:01.870 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:01.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:01.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:01.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:02.009 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:02.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:02.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:02.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:02.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:02.289 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:02.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:02.318 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 12 16 30642 30626 159.492 114.812 0.00544566 0.00626346 2024-03-24T12:11:02.391 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:02.426 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:02.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:02.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:02.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:02.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:02.668 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:02.702 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:02.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:02.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:02.836 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:02.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:02.938 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:02.973 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:02.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:03.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:03.108 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:03.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:03.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:03.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:03.318 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 13 16 32533 32517 156.314 118.188 0.0130628 0.00639243 2024-03-24T12:11:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:03 smithi193 ceph-mon[33593]: pgmap v896: 158 pgs: 158 active+clean; 1.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 786 B/s rd, 175 MiB/s wr, 2.81k op/s 2024-03-24T12:11:03.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:03.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:03.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:03.526 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:03.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:03.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:03 smithi064 ceph-mon[30223]: pgmap v896: 158 pgs: 158 active+clean; 1.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 786 B/s rd, 175 MiB/s wr, 2.81k op/s 2024-03-24T12:11:03.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:03 smithi064 ceph-mon[42243]: pgmap v896: 158 pgs: 158 active+clean; 1.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 786 B/s rd, 175 MiB/s wr, 2.81k op/s 2024-03-24T12:11:03.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:03.669 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:03.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:03.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:03.808 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:03.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:03.910 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:03.945 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:03.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:03.982 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":893,"stamp":"2024-03-24T12:10:58.047131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1150753118,"num_objects":17748,"num_object_clones":0,"num_object_copies":53244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17748,"num_whiteouts":0,"num_read":1457,"num_read_kb":1209,"num_write":18272,"num_write_kb":1123245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":329608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18824,"ondisk_log_size":18824,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8305728,"kb_used_data":3094628,"kb_used_omap":18,"kb_used_meta":5210989,"kb_avail":741655488,"statfs":{"total":767960285184,"available":759455219712,"internally_reserved":0,"allocated":3168899072,"data_stored":3162251426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":5336052955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,10,11,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1071644672,"num_objects":16352,"num_object_clones":0,"num_object_copies":49056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16352,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16352,"num_write_kb":1046528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.528218"},"pg_stats":[{"pgid":"11.7","version":"428'810","reported_seq":842,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:54.279686+0000","last_change":"2024-03-24T12:10:42.552347+0000","last_active":"2024-03-24T12:10:54.279686+0000","last_peered":"2024-03-24T12:10:54.279686+0000","last_clean":"2024-03-24T12:10:54.279686+0000","last_became_active":"2024-03-24T12:10:42.552023+0000","last_became_peered":"2024-03-24T12:10:42.552023+0000","last_unstale":"2024-03-24T12:10:54.279686+0000","last_undegraded":"2024-03-24T12:10:54.279686+0000","last_fullsized":"2024-03-24T12:10:54.279686+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:07.500025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53084160,"num_objects":810,"num_object_clones":0,"num_object_copies":2430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":810,"num_write_kb":51840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":1048,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.021129+0000","last_change":"2024-03-24T12:10:29.922054+0000","last_active":"2024-03-24T12:10:55.021129+0000","last_peered":"2024-03-24T12:10:55.021129+0000","last_clean":"2024-03-24T12:10:55.021129+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:10:55.021129+0000","last_undegraded":"2024-03-24T12:10:55.021129+0000","last_fullsized":"2024-03-24T12:10:55.021129+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:10:29.921946+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.921946+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:43.508199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":717,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014023+0000","last_change":"2024-03-24T12:08:36.154379+0000","last_active":"2024-03-24T12:10:52.014023+0000","last_peered":"2024-03-24T12:10:52.014023+0000","last_clean":"2024-03-24T12:10:52.014023+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:10:52.014023+0000","last_undegraded":"2024-03-24T12:10:52.014023+0000","last_fullsized":"2024-03-24T12:10:52.014023+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:05:17.253630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.253630+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":279,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016406+0000","last_change":"2024-03-24T12:08:32.282623+0000","last_active":"2024-03-24T12:10:55.016406+0000","last_peered":"2024-03-24T12:10:55.016406+0000","last_clean":"2024-03-24T12:10:55.016406+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:10:55.016406+0000","last_undegraded":"2024-03-24T12:10:55.016406+0000","last_fullsized":"2024-03-24T12:10:55.016406+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:32.282513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:32.282513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":709,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014752+0000","last_change":"2024-03-24T12:10:19.713100+0000","last_active":"2024-03-24T12:10:52.014752+0000","last_peered":"2024-03-24T12:10:52.014752+0000","last_clean":"2024-03-24T12:10:52.014752+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:10:52.014752+0000","last_undegraded":"2024-03-24T12:10:52.014752+0000","last_fullsized":"2024-03-24T12:10:52.014752+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.713016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.713016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:28.578775+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"429'1203","reported_seq":1237,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.700500+0000","last_change":"2024-03-24T12:10:42.558953+0000","last_active":"2024-03-24T12:10:56.700500+0000","last_peered":"2024-03-24T12:10:56.700500+0000","last_clean":"2024-03-24T12:10:56.700500+0000","last_became_active":"2024-03-24T12:10:42.558408+0000","last_became_peered":"2024-03-24T12:10:42.558408+0000","last_unstale":"2024-03-24T12:10:56.700500+0000","last_undegraded":"2024-03-24T12:10:56.700500+0000","last_fullsized":"2024-03-24T12:10:56.700500+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1203,"log_dups_size":0,"ondisk_log_size":1203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:41.716019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78839808,"num_objects":1203,"num_object_clones":0,"num_object_copies":3609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1203,"num_write_kb":76992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":816,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.010780+0000","last_change":"2024-03-24T12:07:52.313886+0000","last_active":"2024-03-24T12:10:52.010780+0000","last_peered":"2024-03-24T12:10:52.010780+0000","last_clean":"2024-03-24T12:10:52.010780+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:10:52.010780+0000","last_undegraded":"2024-03-24T12:10:52.010780+0000","last_fullsized":"2024-03-24T12:10:52.010780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:52.313429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:52.313429+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.180910+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":830,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.021258+0000","last_change":"2024-03-24T12:10:21.667543+0000","last_active":"2024-03-24T12:10:55.021258+0000","last_peered":"2024-03-24T12:10:55.021258+0000","last_clean":"2024-03-24T12:10:55.021258+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:10:55.021258+0000","last_undegraded":"2024-03-24T12:10:55.021258+0000","last_fullsized":"2024-03-24T12:10:55.021258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:21.667427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:10:21.667427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:33.797730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":277,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015709+0000","last_change":"2024-03-24T12:07:49.261571+0000","last_active":"2024-03-24T12:10:55.015709+0000","last_peered":"2024-03-24T12:10:55.015709+0000","last_clean":"2024-03-24T12:10:55.015709+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:10:55.015709+0000","last_undegraded":"2024-03-24T12:10:55.015709+0000","last_fullsized":"2024-03-24T12:10:55.015709+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:07:49.261483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":789,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.011923+0000","last_change":"2024-03-24T12:10:26.001526+0000","last_active":"2024-03-24T12:10:52.011923+0000","last_peered":"2024-03-24T12:10:52.011923+0000","last_clean":"2024-03-24T12:10:52.011923+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:10:52.011923+0000","last_undegraded":"2024-03-24T12:10:52.011923+0000","last_fullsized":"2024-03-24T12:10:52.011923+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:26.001400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:26.001400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:46.486749+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"429'1167","reported_seq":1201,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.694868+0000","last_change":"2024-03-24T12:10:42.558826+0000","last_active":"2024-03-24T12:10:56.694868+0000","last_peered":"2024-03-24T12:10:56.694868+0000","last_clean":"2024-03-24T12:10:56.694868+0000","last_became_active":"2024-03-24T12:10:42.558497+0000","last_became_peered":"2024-03-24T12:10:42.558497+0000","last_unstale":"2024-03-24T12:10:56.694868+0000","last_undegraded":"2024-03-24T12:10:56.694868+0000","last_fullsized":"2024-03-24T12:10:56.694868+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1167,"log_dups_size":0,"ondisk_log_size":1167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:09.648782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76480512,"num_objects":1167,"num_object_clones":0,"num_object_copies":3501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1167,"num_write_kb":74688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"296'44","reported_seq":864,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.018725+0000","last_change":"2024-03-24T12:07:37.639999+0000","last_active":"2024-03-24T12:10:55.018725+0000","last_peered":"2024-03-24T12:10:55.018725+0000","last_clean":"2024-03-24T12:10:55.018725+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:10:55.018725+0000","last_undegraded":"2024-03-24T12:10:55.018725+0000","last_fullsized":"2024-03-24T12:10:55.018725+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:07:37.639868+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:07:37.639868+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962714+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":171,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014830+0000","last_change":"2024-03-24T12:08:36.164911+0000","last_active":"2024-03-24T12:10:52.014830+0000","last_peered":"2024-03-24T12:10:52.014830+0000","last_clean":"2024-03-24T12:10:52.014830+0000","last_became_active":"2024-03-24T12:08:36.164337+0000","last_became_peered":"2024-03-24T12:08:36.164337+0000","last_unstale":"2024-03-24T12:10:52.014830+0000","last_undegraded":"2024-03-24T12:10:52.014830+0000","last_fullsized":"2024-03-24T12:10:52.014830+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:00.489543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:05:00.489543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":284,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.010367+0000","last_change":"2024-03-24T12:09:19.669080+0000","last_active":"2024-03-24T12:10:52.010367+0000","last_peered":"2024-03-24T12:10:52.010367+0000","last_clean":"2024-03-24T12:10:52.010367+0000","last_became_active":"2024-03-24T12:08:36.155105+0000","last_became_peered":"2024-03-24T12:08:36.155105+0000","last_unstale":"2024-03-24T12:10:52.010367+0000","last_undegraded":"2024-03-24T12:10:52.010367+0000","last_fullsized":"2024-03-24T12:10:52.010367+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:19.668946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:09:19.668946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.181002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":808,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.018844+0000","last_change":"2024-03-24T12:10:11.066929+0000","last_active":"2024-03-24T12:10:55.018844+0000","last_peered":"2024-03-24T12:10:55.018844+0000","last_clean":"2024-03-24T12:10:55.018844+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:10:55.018844+0000","last_undegraded":"2024-03-24T12:10:55.018844+0000","last_fullsized":"2024-03-24T12:10:55.018844+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:11.066783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:10:11.066783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:11.811726+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"429'1110","reported_seq":1144,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.307565+0000","last_change":"2024-03-24T12:10:42.555659+0000","last_active":"2024-03-24T12:10:56.307565+0000","last_peered":"2024-03-24T12:10:56.307565+0000","last_clean":"2024-03-24T12:10:56.307565+0000","last_became_active":"2024-03-24T12:10:42.555322+0000","last_became_peered":"2024-03-24T12:10:42.555322+0000","last_unstale":"2024-03-24T12:10:56.307565+0000","last_undegraded":"2024-03-24T12:10:56.307565+0000","last_fullsized":"2024-03-24T12:10:56.307565+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1110,"log_dups_size":0,"ondisk_log_size":1110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:54.319589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1110,"num_write_kb":71040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":873,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016158+0000","last_change":"2024-03-24T12:07:41.143174+0000","last_active":"2024-03-24T12:10:55.016158+0000","last_peered":"2024-03-24T12:10:55.016158+0000","last_clean":"2024-03-24T12:10:55.016158+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:10:55.016158+0000","last_undegraded":"2024-03-24T12:10:55.016158+0000","last_fullsized":"2024-03-24T12:10:55.016158+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:07:41.143037+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.143037+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":585,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015480+0000","last_change":"2024-03-24T12:09:04.230431+0000","last_active":"2024-03-24T12:10:55.015480+0000","last_peered":"2024-03-24T12:10:55.015480+0000","last_clean":"2024-03-24T12:10:55.015480+0000","last_became_active":"2024-03-24T12:09:04.230111+0000","last_became_peered":"2024-03-24T12:09:04.230111+0000","last_unstale":"2024-03-24T12:10:55.015480+0000","last_undegraded":"2024-03-24T12:10:55.015480+0000","last_fullsized":"2024-03-24T12:10:55.015480+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:08:42.842178+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.842178+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":541,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015854+0000","last_change":"2024-03-24T12:07:48.250602+0000","last_active":"2024-03-24T12:10:55.015854+0000","last_peered":"2024-03-24T12:10:55.015854+0000","last_clean":"2024-03-24T12:10:55.015854+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:10:55.015854+0000","last_undegraded":"2024-03-24T12:10:55.015854+0000","last_fullsized":"2024-03-24T12:10:55.015854+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_clean_scrub_stamp":"2024-03-24T12:07:48.250404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600551+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":278,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016623+0000","last_change":"2024-03-24T12:08:34.445597+0000","last_active":"2024-03-24T12:10:55.016623+0000","last_peered":"2024-03-24T12:10:55.016623+0000","last_clean":"2024-03-24T12:10:55.016623+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:10:55.016623+0000","last_undegraded":"2024-03-24T12:10:55.016623+0000","last_fullsized":"2024-03-24T12:10:55.016623+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:34.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.445477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":739,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014156+0000","last_change":"2024-03-24T12:10:09.762381+0000","last_active":"2024-03-24T12:10:52.014156+0000","last_peered":"2024-03-24T12:10:52.014156+0000","last_clean":"2024-03-24T12:10:52.014156+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:10:52.014156+0000","last_undegraded":"2024-03-24T12:10:52.014156+0000","last_fullsized":"2024-03-24T12:10:52.014156+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:09.762247+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.762247+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:09.850020+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"429'1213","reported_seq":1247,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.719164+0000","last_change":"2024-03-24T12:10:42.557591+0000","last_active":"2024-03-24T12:10:56.719164+0000","last_peered":"2024-03-24T12:10:56.719164+0000","last_clean":"2024-03-24T12:10:56.719164+0000","last_became_active":"2024-03-24T12:10:42.557222+0000","last_became_peered":"2024-03-24T12:10:42.557222+0000","last_unstale":"2024-03-24T12:10:56.719164+0000","last_undegraded":"2024-03-24T12:10:56.719164+0000","last_fullsized":"2024-03-24T12:10:56.719164+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:54.567849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79495168,"num_objects":1213,"num_object_clones":0,"num_object_copies":3639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1213,"num_write_kb":77632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":827,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015841+0000","last_change":"2024-03-24T12:07:56.328634+0000","last_active":"2024-03-24T12:10:55.015841+0000","last_peered":"2024-03-24T12:10:55.015841+0000","last_clean":"2024-03-24T12:10:55.015841+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:10:55.015841+0000","last_undegraded":"2024-03-24T12:10:55.015841+0000","last_fullsized":"2024-03-24T12:10:55.015841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:56.328393+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.328393+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":549,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015114+0000","last_change":"2024-03-24T12:10:31.694887+0000","last_active":"2024-03-24T12:10:55.015114+0000","last_peered":"2024-03-24T12:10:55.015114+0000","last_clean":"2024-03-24T12:10:55.015114+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:10:55.015114+0000","last_undegraded":"2024-03-24T12:10:55.015114+0000","last_fullsized":"2024-03-24T12:10:55.015114+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:31.694756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:10:31.694756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:56.762052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":1049,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.012615+0000","last_change":"2024-03-24T12:10:13.987176+0000","last_active":"2024-03-24T12:10:55.012615+0000","last_peered":"2024-03-24T12:10:55.012615+0000","last_clean":"2024-03-24T12:10:55.012615+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:10:55.012615+0000","last_undegraded":"2024-03-24T12:10:55.012615+0000","last_fullsized":"2024-03-24T12:10:55.012615+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:13.987052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.987052+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:40.193982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":804,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.017188+0000","last_change":"2024-03-24T12:10:09.602488+0000","last_active":"2024-03-24T12:10:55.017188+0000","last_peered":"2024-03-24T12:10:55.017188+0000","last_clean":"2024-03-24T12:10:55.017188+0000","last_became_active":"2024-03-24T12:08:36.161088+0000","last_became_peered":"2024-03-24T12:08:36.161088+0000","last_unstale":"2024-03-24T12:10:55.017188+0000","last_undegraded":"2024-03-24T12:10:55.017188+0000","last_fullsized":"2024-03-24T12:10:55.017188+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:10:09.602371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.602371+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:15.711609+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":1129,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.020571+0000","last_change":"2024-03-24T12:10:38.059045+0000","last_active":"2024-03-24T12:10:55.020571+0000","last_peered":"2024-03-24T12:10:55.020571+0000","last_clean":"2024-03-24T12:10:55.020571+0000","last_became_active":"2024-03-24T12:08:36.161668+0000","last_became_peered":"2024-03-24T12:08:36.161668+0000","last_unstale":"2024-03-24T12:10:55.020571+0000","last_undegraded":"2024-03-24T12:10:55.020571+0000","last_fullsized":"2024-03-24T12:10:55.020571+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:38.058938+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:10:38.058938+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:39.038451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"429'1099","reported_seq":1133,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.287048+0000","last_change":"2024-03-24T12:10:42.559881+0000","last_active":"2024-03-24T12:10:56.287048+0000","last_peered":"2024-03-24T12:10:56.287048+0000","last_clean":"2024-03-24T12:10:56.287048+0000","last_became_active":"2024-03-24T12:10:42.559566+0000","last_became_peered":"2024-03-24T12:10:42.559566+0000","last_unstale":"2024-03-24T12:10:56.287048+0000","last_undegraded":"2024-03-24T12:10:56.287048+0000","last_fullsized":"2024-03-24T12:10:56.287048+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:08.554116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72024064,"num_objects":1099,"num_object_clones":0,"num_object_copies":3297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":70336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":868,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.021653+0000","last_change":"2024-03-24T12:10:32.769377+0000","last_active":"2024-03-24T12:10:55.021653+0000","last_peered":"2024-03-24T12:10:55.021653+0000","last_clean":"2024-03-24T12:10:55.021653+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:10:55.021653+0000","last_undegraded":"2024-03-24T12:10:55.021653+0000","last_fullsized":"2024-03-24T12:10:55.021653+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:10:32.769281+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:10:32.769281+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:58.251612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":822,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.018382+0000","last_change":"2024-03-24T12:10:28.697217+0000","last_active":"2024-03-24T12:10:55.018382+0000","last_peered":"2024-03-24T12:10:55.018382+0000","last_clean":"2024-03-24T12:10:55.018382+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:10:55.018382+0000","last_undegraded":"2024-03-24T12:10:55.018382+0000","last_fullsized":"2024-03-24T12:10:55.018382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:28.697115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:10:28.697115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:42.182877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":1035,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.017039+0000","last_change":"2024-03-24T12:10:14.509080+0000","last_active":"2024-03-24T12:10:55.017039+0000","last_peered":"2024-03-24T12:10:55.017039+0000","last_clean":"2024-03-24T12:10:55.017039+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:10:55.017039+0000","last_undegraded":"2024-03-24T12:10:55.017039+0000","last_fullsized":"2024-03-24T12:10:55.017039+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:14.508959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.508959+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:16.508133+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":709,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014993+0000","last_change":"2024-03-24T12:08:36.167496+0000","last_active":"2024-03-24T12:10:52.014993+0000","last_peered":"2024-03-24T12:10:52.014993+0000","last_clean":"2024-03-24T12:10:52.014993+0000","last_became_active":"2024-03-24T12:08:36.166897+0000","last_became_peered":"2024-03-24T12:08:36.166897+0000","last_unstale":"2024-03-24T12:10:52.014993+0000","last_undegraded":"2024-03-24T12:10:52.014993+0000","last_fullsized":"2024-03-24T12:10:52.014993+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:02.219541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:04:02.219541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":739,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015035+0000","last_change":"2024-03-24T12:10:11.797368+0000","last_active":"2024-03-24T12:10:52.015035+0000","last_peered":"2024-03-24T12:10:52.015035+0000","last_clean":"2024-03-24T12:10:52.015035+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:10:52.015035+0000","last_undegraded":"2024-03-24T12:10:52.015035+0000","last_fullsized":"2024-03-24T12:10:52.015035+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:11.797287+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:11.797287+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:26.815442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"429'1162","reported_seq":1196,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.712393+0000","last_change":"2024-03-24T12:10:42.557448+0000","last_active":"2024-03-24T12:10:56.712393+0000","last_peered":"2024-03-24T12:10:56.712393+0000","last_clean":"2024-03-24T12:10:56.712393+0000","last_became_active":"2024-03-24T12:10:42.557136+0000","last_became_peered":"2024-03-24T12:10:42.557136+0000","last_unstale":"2024-03-24T12:10:56.712393+0000","last_undegraded":"2024-03-24T12:10:56.712393+0000","last_fullsized":"2024-03-24T12:10:56.712393+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1162,"log_dups_size":0,"ondisk_log_size":1162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:10.219799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76152832,"num_objects":1162,"num_object_clones":0,"num_object_copies":3486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1162,"num_write_kb":74368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":820,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016164+0000","last_change":"2024-03-24T12:06:46.690666+0000","last_active":"2024-03-24T12:10:55.016164+0000","last_peered":"2024-03-24T12:10:55.016164+0000","last_clean":"2024-03-24T12:10:55.016164+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:10:55.016164+0000","last_undegraded":"2024-03-24T12:10:55.016164+0000","last_fullsized":"2024-03-24T12:10:55.016164+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:06:46.690548+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:06:46.690548+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600390+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":814,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.020420+0000","last_change":"2024-03-24T12:09:26.059856+0000","last_active":"2024-03-24T12:10:55.020420+0000","last_peered":"2024-03-24T12:10:55.020420+0000","last_clean":"2024-03-24T12:10:55.020420+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:10:55.020420+0000","last_undegraded":"2024-03-24T12:10:55.020420+0000","last_fullsized":"2024-03-24T12:10:55.020420+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:26.059376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:09:26.059376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":705,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015621+0000","last_change":"2024-03-24T12:08:36.163502+0000","last_active":"2024-03-24T12:10:52.015621+0000","last_peered":"2024-03-24T12:10:52.015621+0000","last_clean":"2024-03-24T12:10:52.015621+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:10:52.015621+0000","last_undegraded":"2024-03-24T12:10:52.015621+0000","last_fullsized":"2024-03-24T12:10:52.015621+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:15.334211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:15.334211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":796,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.013818+0000","last_change":"2024-03-24T12:10:14.749596+0000","last_active":"2024-03-24T12:10:55.013818+0000","last_peered":"2024-03-24T12:10:55.013818+0000","last_clean":"2024-03-24T12:10:55.013818+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:10:55.013818+0000","last_undegraded":"2024-03-24T12:10:55.013818+0000","last_fullsized":"2024-03-24T12:10:55.013818+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:14.749468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.749468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:20.468702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"429'1183","reported_seq":1217,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.646252+0000","last_change":"2024-03-24T12:10:42.552380+0000","last_active":"2024-03-24T12:10:56.646252+0000","last_peered":"2024-03-24T12:10:56.646252+0000","last_clean":"2024-03-24T12:10:56.646252+0000","last_became_active":"2024-03-24T12:10:42.552054+0000","last_became_peered":"2024-03-24T12:10:42.552054+0000","last_unstale":"2024-03-24T12:10:56.646252+0000","last_undegraded":"2024-03-24T12:10:56.646252+0000","last_fullsized":"2024-03-24T12:10:56.646252+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1183,"log_dups_size":0,"ondisk_log_size":1183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:08.753590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77529088,"num_objects":1183,"num_object_clones":0,"num_object_copies":3549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":1017,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014431+0000","last_change":"2024-03-24T12:07:34.996457+0000","last_active":"2024-03-24T12:10:55.014431+0000","last_peered":"2024-03-24T12:10:55.014431+0000","last_clean":"2024-03-24T12:10:55.014431+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:10:55.014431+0000","last_undegraded":"2024-03-24T12:10:55.014431+0000","last_fullsized":"2024-03-24T12:10:55.014431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:07:34.996314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:07:34.996314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":850,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015458+0000","last_change":"2024-03-24T12:10:19.407534+0000","last_active":"2024-03-24T12:10:55.015458+0000","last_peered":"2024-03-24T12:10:55.015458+0000","last_clean":"2024-03-24T12:10:55.015458+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:10:55.015458+0000","last_undegraded":"2024-03-24T12:10:55.015458+0000","last_fullsized":"2024-03-24T12:10:55.015458+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:10:19.407433+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.407433+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:32.472969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":804,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019809+0000","last_change":"2024-03-24T12:07:30.557017+0000","last_active":"2024-03-24T12:10:55.019809+0000","last_peered":"2024-03-24T12:10:55.019809+0000","last_clean":"2024-03-24T12:10:55.019809+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:10:55.019809+0000","last_undegraded":"2024-03-24T12:10:55.019809+0000","last_fullsized":"2024-03-24T12:10:55.019809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:30.556903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:07:30.556903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962686+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":816,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.022017+0000","last_change":"2024-03-24T12:10:18.089858+0000","last_active":"2024-03-24T12:10:55.022017+0000","last_peered":"2024-03-24T12:10:55.022017+0000","last_clean":"2024-03-24T12:10:55.022017+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:10:55.022017+0000","last_undegraded":"2024-03-24T12:10:55.022017+0000","last_fullsized":"2024-03-24T12:10:55.022017+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:18.089744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:18.089744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:43.549688+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":875,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014111+0000","last_change":"2024-03-24T12:10:47.972288+0000","last_active":"2024-03-24T12:10:55.014111+0000","last_peered":"2024-03-24T12:10:55.014111+0000","last_clean":"2024-03-24T12:10:55.014111+0000","last_became_active":"2024-03-24T12:10:47.971906+0000","last_became_peered":"2024-03-24T12:10:47.971906+0000","last_unstale":"2024-03-24T12:10:55.014111+0000","last_undegraded":"2024-03-24T12:10:55.014111+0000","last_fullsized":"2024-03-24T12:10:55.014111+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:36.201565+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.201565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.748002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"429'1169","reported_seq":1203,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.695370+0000","last_change":"2024-03-24T12:10:42.562451+0000","last_active":"2024-03-24T12:10:56.695370+0000","last_peered":"2024-03-24T12:10:56.695370+0000","last_clean":"2024-03-24T12:10:56.695370+0000","last_became_active":"2024-03-24T12:10:42.562110+0000","last_became_peered":"2024-03-24T12:10:42.562110+0000","last_unstale":"2024-03-24T12:10:56.695370+0000","last_undegraded":"2024-03-24T12:10:56.695370+0000","last_fullsized":"2024-03-24T12:10:56.695370+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1169,"log_dups_size":0,"ondisk_log_size":1169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:08.384408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76611584,"num_objects":1169,"num_object_clones":0,"num_object_copies":3507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1169,"num_write_kb":74816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":780,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015571+0000","last_change":"2024-03-24T12:06:44.708778+0000","last_active":"2024-03-24T12:10:55.015571+0000","last_peered":"2024-03-24T12:10:55.015571+0000","last_clean":"2024-03-24T12:10:55.015571+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:10:55.015571+0000","last_undegraded":"2024-03-24T12:10:55.015571+0000","last_fullsized":"2024-03-24T12:10:55.015571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:44.708677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:06:44.708677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":854,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014280+0000","last_change":"2024-03-24T12:10:29.733172+0000","last_active":"2024-03-24T12:10:55.014280+0000","last_peered":"2024-03-24T12:10:55.014280+0000","last_clean":"2024-03-24T12:10:55.014280+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:10:55.014280+0000","last_undegraded":"2024-03-24T12:10:55.014280+0000","last_fullsized":"2024-03-24T12:10:55.014280+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_deep_scrub":"241'38","last_deep_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.733073+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:34.889776+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":557,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019292+0000","last_change":"2024-03-24T12:07:40.603920+0000","last_active":"2024-03-24T12:10:55.019292+0000","last_peered":"2024-03-24T12:10:55.019292+0000","last_clean":"2024-03-24T12:10:55.019292+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:10:55.019292+0000","last_undegraded":"2024-03-24T12:10:55.019292+0000","last_fullsized":"2024-03-24T12:10:55.019292+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:07:40.603768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.603768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":806,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015756+0000","last_change":"2024-03-24T12:10:38.401132+0000","last_active":"2024-03-24T12:10:55.015756+0000","last_peered":"2024-03-24T12:10:55.015756+0000","last_clean":"2024-03-24T12:10:55.015756+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:10:55.015756+0000","last_undegraded":"2024-03-24T12:10:55.015756+0000","last_fullsized":"2024-03-24T12:10:55.015756+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:38.401028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:10:38.401028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:07.118106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":881,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015700+0000","last_change":"2024-03-24T12:10:45.574511+0000","last_active":"2024-03-24T12:10:55.015700+0000","last_peered":"2024-03-24T12:10:55.015700+0000","last_clean":"2024-03-24T12:10:55.015700+0000","last_became_active":"2024-03-24T12:10:45.573960+0000","last_became_peered":"2024-03-24T12:10:45.573960+0000","last_unstale":"2024-03-24T12:10:55.015700+0000","last_undegraded":"2024-03-24T12:10:55.015700+0000","last_fullsized":"2024-03-24T12:10:55.015700+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:25.359883+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.359883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:48.227960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"429'1222","reported_seq":1256,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:57.210652+0000","last_change":"2024-03-24T12:10:42.552647+0000","last_active":"2024-03-24T12:10:57.210652+0000","last_peered":"2024-03-24T12:10:57.210652+0000","last_clean":"2024-03-24T12:10:57.210652+0000","last_became_active":"2024-03-24T12:10:42.552346+0000","last_became_peered":"2024-03-24T12:10:42.552346+0000","last_unstale":"2024-03-24T12:10:57.210652+0000","last_undegraded":"2024-03-24T12:10:57.210652+0000","last_fullsized":"2024-03-24T12:10:57.210652+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1222,"log_dups_size":0,"ondisk_log_size":1222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:41.710777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80084992,"num_objects":1222,"num_object_clones":0,"num_object_copies":3666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1222,"num_write_kb":78208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":805,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019493+0000","last_change":"2024-03-24T12:10:14.821427+0000","last_active":"2024-03-24T12:10:55.019493+0000","last_peered":"2024-03-24T12:10:55.019493+0000","last_clean":"2024-03-24T12:10:55.019493+0000","last_became_active":"2024-03-24T12:08:36.161658+0000","last_became_peered":"2024-03-24T12:08:36.161658+0000","last_unstale":"2024-03-24T12:10:55.019493+0000","last_undegraded":"2024-03-24T12:10:55.019493+0000","last_fullsized":"2024-03-24T12:10:55.019493+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:14.821157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.821157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:42.131667+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"369'37","reported_seq":851,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.010523+0000","last_change":"2024-03-24T12:09:25.695502+0000","last_active":"2024-03-24T12:10:52.010523+0000","last_peered":"2024-03-24T12:10:52.010523+0000","last_clean":"2024-03-24T12:10:52.010523+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:10:52.010523+0000","last_undegraded":"2024-03-24T12:10:52.010523+0000","last_fullsized":"2024-03-24T12:10:52.010523+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'37","last_scrub_stamp":"2024-03-24T12:09:25.695317+0000","last_deep_scrub":"369'37","last_deep_scrub_stamp":"2024-03-24T12:09:25.695317+0000","last_clean_scrub_stamp":"2024-03-24T12:09:25.695317+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.181133+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":822,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.022296+0000","last_change":"2024-03-24T12:09:30.041137+0000","last_active":"2024-03-24T12:10:55.022296+0000","last_peered":"2024-03-24T12:10:55.022296+0000","last_clean":"2024-03-24T12:10:55.022296+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:10:55.022296+0000","last_undegraded":"2024-03-24T12:10:55.022296+0000","last_fullsized":"2024-03-24T12:10:55.022296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:30.041008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:09:30.041008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":555,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.018841+0000","last_change":"2024-03-24T12:10:26.791811+0000","last_active":"2024-03-24T12:10:55.018841+0000","last_peered":"2024-03-24T12:10:55.018841+0000","last_clean":"2024-03-24T12:10:55.018841+0000","last_became_active":"2024-03-24T12:05:44.738926+0000","last_became_peered":"2024-03-24T12:05:44.738926+0000","last_unstale":"2024-03-24T12:10:55.018841+0000","last_undegraded":"2024-03-24T12:10:55.018841+0000","last_fullsized":"2024-03-24T12:10:55.018841+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:26.791681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:26.791681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:33.228795+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":887,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019497+0000","last_change":"2024-03-24T12:10:41.554720+0000","last_active":"2024-03-24T12:10:55.019497+0000","last_peered":"2024-03-24T12:10:55.019497+0000","last_clean":"2024-03-24T12:10:55.019497+0000","last_became_active":"2024-03-24T12:10:41.554328+0000","last_became_peered":"2024-03-24T12:10:41.554328+0000","last_unstale":"2024-03-24T12:10:55.019497+0000","last_undegraded":"2024-03-24T12:10:55.019497+0000","last_fullsized":"2024-03-24T12:10:55.019497+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:35.901858+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:35.901858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:57.976484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"429'1090","reported_seq":1124,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.312153+0000","last_change":"2024-03-24T12:10:42.558775+0000","last_active":"2024-03-24T12:10:56.312153+0000","last_peered":"2024-03-24T12:10:56.312153+0000","last_clean":"2024-03-24T12:10:56.312153+0000","last_became_active":"2024-03-24T12:10:42.558462+0000","last_became_peered":"2024-03-24T12:10:42.558462+0000","last_unstale":"2024-03-24T12:10:56.312153+0000","last_undegraded":"2024-03-24T12:10:56.312153+0000","last_fullsized":"2024-03-24T12:10:56.312153+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1090,"log_dups_size":0,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:00.091227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71434240,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":69760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":671,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015753+0000","last_change":"2024-03-24T12:08:36.165575+0000","last_active":"2024-03-24T12:10:52.015753+0000","last_peered":"2024-03-24T12:10:52.015753+0000","last_clean":"2024-03-24T12:10:52.015753+0000","last_became_active":"2024-03-24T12:08:36.165265+0000","last_became_peered":"2024-03-24T12:08:36.165265+0000","last_unstale":"2024-03-24T12:10:52.015753+0000","last_undegraded":"2024-03-24T12:10:52.015753+0000","last_fullsized":"2024-03-24T12:10:52.015753+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:28.232417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:28.232417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":853,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.011463+0000","last_change":"2024-03-24T12:09:35.807760+0000","last_active":"2024-03-24T12:10:52.011463+0000","last_peered":"2024-03-24T12:10:52.011463+0000","last_clean":"2024-03-24T12:10:52.011463+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:10:52.011463+0000","last_undegraded":"2024-03-24T12:10:52.011463+0000","last_fullsized":"2024-03-24T12:10:52.011463+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:09:35.807608+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:09:35.807608+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.181037+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":639,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.009903+0000","last_change":"2024-03-24T12:10:16.036972+0000","last_active":"2024-03-24T12:10:52.009903+0000","last_peered":"2024-03-24T12:10:52.009903+0000","last_clean":"2024-03-24T12:10:52.009903+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:10:52.009903+0000","last_undegraded":"2024-03-24T12:10:52.009903+0000","last_fullsized":"2024-03-24T12:10:52.009903+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:16.036850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:16.036850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:43.369423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":810,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.011573+0000","last_change":"2024-03-24T12:10:29.995345+0000","last_active":"2024-03-24T12:10:52.011573+0000","last_peered":"2024-03-24T12:10:52.011573+0000","last_clean":"2024-03-24T12:10:52.011573+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:10:52.011573+0000","last_undegraded":"2024-03-24T12:10:52.011573+0000","last_fullsized":"2024-03-24T12:10:52.011573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:29.995266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.995266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:41.677910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":811,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.018777+0000","last_change":"2024-03-24T12:10:30.767073+0000","last_active":"2024-03-24T12:10:55.018777+0000","last_peered":"2024-03-24T12:10:55.018777+0000","last_clean":"2024-03-24T12:10:55.018777+0000","last_became_active":"2024-03-24T12:08:36.173357+0000","last_became_peered":"2024-03-24T12:08:36.173357+0000","last_unstale":"2024-03-24T12:10:55.018777+0000","last_undegraded":"2024-03-24T12:10:55.018777+0000","last_fullsized":"2024-03-24T12:10:55.018777+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:30.766987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:30.766987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:38.690974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":897,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019609+0000","last_change":"2024-03-24T12:10:50.999143+0000","last_active":"2024-03-24T12:10:55.019609+0000","last_peered":"2024-03-24T12:10:55.019609+0000","last_clean":"2024-03-24T12:10:55.019609+0000","last_became_active":"2024-03-24T12:10:50.998877+0000","last_became_peered":"2024-03-24T12:10:50.998877+0000","last_unstale":"2024-03-24T12:10:55.019609+0000","last_undegraded":"2024-03-24T12:10:55.019609+0000","last_fullsized":"2024-03-24T12:10:55.019609+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:10.807236+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.807236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:38.972234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"429'1111","reported_seq":1145,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.307988+0000","last_change":"2024-03-24T12:10:42.558754+0000","last_active":"2024-03-24T12:10:56.307988+0000","last_peered":"2024-03-24T12:10:56.307988+0000","last_clean":"2024-03-24T12:10:56.307988+0000","last_became_active":"2024-03-24T12:10:42.558225+0000","last_became_peered":"2024-03-24T12:10:42.558225+0000","last_unstale":"2024-03-24T12:10:56.307988+0000","last_undegraded":"2024-03-24T12:10:56.307988+0000","last_fullsized":"2024-03-24T12:10:56.307988+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1111,"log_dups_size":0,"ondisk_log_size":1111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.545589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72810496,"num_objects":1111,"num_object_clones":0,"num_object_copies":3333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1111,"num_write_kb":71104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":815,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.023173+0000","last_change":"2024-03-24T12:10:37.011600+0000","last_active":"2024-03-24T12:10:55.023173+0000","last_peered":"2024-03-24T12:10:55.023173+0000","last_clean":"2024-03-24T12:10:55.023173+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:10:55.023173+0000","last_undegraded":"2024-03-24T12:10:55.023173+0000","last_fullsized":"2024-03-24T12:10:55.023173+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:37.011493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:10:37.011493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:54.370159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":1081,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019671+0000","last_change":"2024-03-24T12:08:32.823527+0000","last_active":"2024-03-24T12:10:55.019671+0000","last_peered":"2024-03-24T12:10:55.019671+0000","last_clean":"2024-03-24T12:10:55.019671+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:10:55.019671+0000","last_undegraded":"2024-03-24T12:10:55.019671+0000","last_fullsized":"2024-03-24T12:10:55.019671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:08:32.823398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:32.823398+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962854+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":1127,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.020299+0000","last_change":"2024-03-24T12:10:33.462124+0000","last_active":"2024-03-24T12:10:55.020299+0000","last_peered":"2024-03-24T12:10:55.020299+0000","last_clean":"2024-03-24T12:10:55.020299+0000","last_became_active":"2024-03-24T12:10:33.461798+0000","last_became_peered":"2024-03-24T12:10:33.461798+0000","last_unstale":"2024-03-24T12:10:55.020299+0000","last_undegraded":"2024-03-24T12:10:55.020299+0000","last_fullsized":"2024-03-24T12:10:55.020299+0000","mapping_epoch":64,"log_start":"83'130","ondisk_log_start":"83'130","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:25.984109+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:10:25.984109+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.984109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:31.103499+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":711,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014717+0000","last_change":"2024-03-24T12:08:36.169985+0000","last_active":"2024-03-24T12:10:52.014717+0000","last_peered":"2024-03-24T12:10:52.014717+0000","last_clean":"2024-03-24T12:10:52.014717+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:10:52.014717+0000","last_undegraded":"2024-03-24T12:10:52.014717+0000","last_fullsized":"2024-03-24T12:10:52.014717+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.356759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.356759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":809,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015012+0000","last_change":"2024-03-24T12:08:43.113674+0000","last_active":"2024-03-24T12:10:55.015012+0000","last_peered":"2024-03-24T12:10:55.015012+0000","last_clean":"2024-03-24T12:10:55.015012+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:10:55.015012+0000","last_undegraded":"2024-03-24T12:10:55.015012+0000","last_fullsized":"2024-03-24T12:10:55.015012+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:08:43.113510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.113510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":666,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015715+0000","last_change":"2024-03-24T12:08:36.162637+0000","last_active":"2024-03-24T12:10:52.015715+0000","last_peered":"2024-03-24T12:10:52.015715+0000","last_clean":"2024-03-24T12:10:52.015715+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:10:52.015715+0000","last_undegraded":"2024-03-24T12:10:52.015715+0000","last_fullsized":"2024-03-24T12:10:52.015715+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:10.463242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:05:10.463242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"429'1108","reported_seq":1142,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.278925+0000","last_change":"2024-03-24T12:10:42.558943+0000","last_active":"2024-03-24T12:10:56.278925+0000","last_peered":"2024-03-24T12:10:56.278925+0000","last_clean":"2024-03-24T12:10:56.278925+0000","last_became_active":"2024-03-24T12:10:42.558601+0000","last_became_peered":"2024-03-24T12:10:42.558601+0000","last_unstale":"2024-03-24T12:10:56.278925+0000","last_undegraded":"2024-03-24T12:10:56.278925+0000","last_fullsized":"2024-03-24T12:10:56.278925+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:57.508925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72613888,"num_objects":1108,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1108,"num_write_kb":70912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":193,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015474+0000","last_change":"2024-03-24T12:08:36.162550+0000","last_active":"2024-03-24T12:10:52.015474+0000","last_peered":"2024-03-24T12:10:52.015474+0000","last_clean":"2024-03-24T12:10:52.015474+0000","last_became_active":"2024-03-24T12:08:36.162188+0000","last_became_peered":"2024-03-24T12:08:36.162188+0000","last_unstale":"2024-03-24T12:10:52.015474+0000","last_undegraded":"2024-03-24T12:10:52.015474+0000","last_fullsized":"2024-03-24T12:10:52.015474+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.646040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":852,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.021424+0000","last_change":"2024-03-24T12:10:09.721155+0000","last_active":"2024-03-24T12:10:55.021424+0000","last_peered":"2024-03-24T12:10:55.021424+0000","last_clean":"2024-03-24T12:10:55.021424+0000","last_became_active":"2024-03-24T12:08:36.172879+0000","last_became_peered":"2024-03-24T12:08:36.172879+0000","last_unstale":"2024-03-24T12:10:55.021424+0000","last_undegraded":"2024-03-24T12:10:55.021424+0000","last_fullsized":"2024-03-24T12:10:55.021424+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:10:09.720983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.720983+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:26.143778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":279,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016172+0000","last_change":"2024-03-24T12:10:12.685963+0000","last_active":"2024-03-24T12:10:55.016172+0000","last_peered":"2024-03-24T12:10:55.016172+0000","last_clean":"2024-03-24T12:10:55.016172+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:10:55.016172+0000","last_undegraded":"2024-03-24T12:10:55.016172+0000","last_fullsized":"2024-03-24T12:10:55.016172+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.685815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.685815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:18.482331+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":635,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.360236+0000","last_change":"2024-03-24T12:06:55.648298+0000","last_active":"2024-03-24T12:10:55.360236+0000","last_peered":"2024-03-24T12:10:55.360236+0000","last_clean":"2024-03-24T12:10:55.360236+0000","last_became_active":"2024-03-24T12:05:44.736437+0000","last_became_peered":"2024-03-24T12:05:44.736437+0000","last_unstale":"2024-03-24T12:10:55.360236+0000","last_undegraded":"2024-03-24T12:10:55.360236+0000","last_fullsized":"2024-03-24T12:10:55.360236+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:06:55.648166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:55.648166+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962631+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":318,"num_read_kb":313,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":802,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.022201+0000","last_change":"2024-03-24T12:09:39.094894+0000","last_active":"2024-03-24T12:10:55.022201+0000","last_peered":"2024-03-24T12:10:55.022201+0000","last_clean":"2024-03-24T12:10:55.022201+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:10:55.022201+0000","last_undegraded":"2024-03-24T12:10:55.022201+0000","last_fullsized":"2024-03-24T12:10:55.022201+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:39.094777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:09:39.094777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:04.218387+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":1126,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.021202+0000","last_change":"2024-03-24T12:10:23.398610+0000","last_active":"2024-03-24T12:10:55.021202+0000","last_peered":"2024-03-24T12:10:55.021202+0000","last_clean":"2024-03-24T12:10:55.021202+0000","last_became_active":"2024-03-24T12:10:23.398177+0000","last_became_peered":"2024-03-24T12:10:23.398177+0000","last_unstale":"2024-03-24T12:10:55.021202+0000","last_undegraded":"2024-03-24T12:10:55.021202+0000","last_fullsized":"2024-03-24T12:10:55.021202+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:09:29.088421+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:09:29.088421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:58.063374+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"429'1119","reported_seq":1153,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.319242+0000","last_change":"2024-03-24T12:10:42.559860+0000","last_active":"2024-03-24T12:10:56.319242+0000","last_peered":"2024-03-24T12:10:56.319242+0000","last_clean":"2024-03-24T12:10:56.319242+0000","last_became_active":"2024-03-24T12:10:42.559548+0000","last_became_peered":"2024-03-24T12:10:42.559548+0000","last_unstale":"2024-03-24T12:10:56.319242+0000","last_undegraded":"2024-03-24T12:10:56.319242+0000","last_fullsized":"2024-03-24T12:10:56.319242+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1119,"log_dups_size":0,"ondisk_log_size":1119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:09.669545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73334784,"num_objects":1119,"num_object_clones":0,"num_object_copies":3357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1119,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":787,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.011214+0000","last_change":"2024-03-24T12:10:11.115171+0000","last_active":"2024-03-24T12:10:52.011214+0000","last_peered":"2024-03-24T12:10:52.011214+0000","last_clean":"2024-03-24T12:10:52.011214+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:10:52.011214+0000","last_undegraded":"2024-03-24T12:10:52.011214+0000","last_fullsized":"2024-03-24T12:10:52.011214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:11.115025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:10:11.115025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:13.588710+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"354'44","reported_seq":923,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015564+0000","last_change":"2024-03-24T12:10:33.726470+0000","last_active":"2024-03-24T12:10:52.015564+0000","last_peered":"2024-03-24T12:10:52.015564+0000","last_clean":"2024-03-24T12:10:52.015564+0000","last_became_active":"2024-03-24T12:08:36.152866+0000","last_became_peered":"2024-03-24T12:08:36.152866+0000","last_unstale":"2024-03-24T12:10:52.015564+0000","last_undegraded":"2024-03-24T12:10:52.015564+0000","last_fullsized":"2024-03-24T12:10:52.015564+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'44","last_scrub_stamp":"2024-03-24T12:10:33.726357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:33.726357+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:55.232205+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":675,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014539+0000","last_change":"2024-03-24T12:08:36.170514+0000","last_active":"2024-03-24T12:10:52.014539+0000","last_peered":"2024-03-24T12:10:52.014539+0000","last_clean":"2024-03-24T12:10:52.014539+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:10:52.014539+0000","last_undegraded":"2024-03-24T12:10:52.014539+0000","last_fullsized":"2024-03-24T12:10:52.014539+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:54.196749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:04:54.196749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":779,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.017215+0000","last_change":"2024-03-24T12:07:33.984169+0000","last_active":"2024-03-24T12:10:55.017215+0000","last_peered":"2024-03-24T12:10:55.017215+0000","last_clean":"2024-03-24T12:10:55.017215+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:10:55.017215+0000","last_undegraded":"2024-03-24T12:10:55.017215+0000","last_fullsized":"2024-03-24T12:10:55.017215+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:33.984042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:33.984042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":758,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014571+0000","last_change":"2024-03-24T12:10:26.959642+0000","last_active":"2024-03-24T12:10:52.014571+0000","last_peered":"2024-03-24T12:10:52.014571+0000","last_clean":"2024-03-24T12:10:52.014571+0000","last_became_active":"2024-03-24T12:10:26.959434+0000","last_became_peered":"2024-03-24T12:10:26.959434+0000","last_unstale":"2024-03-24T12:10:52.014571+0000","last_undegraded":"2024-03-24T12:10:52.014571+0000","last_fullsized":"2024-03-24T12:10:52.014571+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:10.760367+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.760367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:29.990247+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"429'1161","reported_seq":1194,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:56.649846+0000","last_change":"2024-03-24T12:10:42.558548+0000","last_active":"2024-03-24T12:10:56.649846+0000","last_peered":"2024-03-24T12:10:56.649846+0000","last_clean":"2024-03-24T12:10:56.649846+0000","last_became_active":"2024-03-24T12:10:42.558230+0000","last_became_peered":"2024-03-24T12:10:42.558230+0000","last_unstale":"2024-03-24T12:10:56.649846+0000","last_undegraded":"2024-03-24T12:10:56.649846+0000","last_fullsized":"2024-03-24T12:10:56.649846+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":1161,"log_dups_size":0,"ondisk_log_size":1161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:55.629693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76087296,"num_objects":1161,"num_object_clones":0,"num_object_copies":3483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":74304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":1030,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.010885+0000","last_change":"2024-03-24T12:08:36.673921+0000","last_active":"2024-03-24T12:10:52.010885+0000","last_peered":"2024-03-24T12:10:52.010885+0000","last_clean":"2024-03-24T12:10:52.010885+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:10:52.010885+0000","last_undegraded":"2024-03-24T12:10:52.010885+0000","last_fullsized":"2024-03-24T12:10:52.010885+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:08:36.673814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.673814+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.180943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":854,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.020079+0000","last_change":"2024-03-24T12:10:12.689604+0000","last_active":"2024-03-24T12:10:55.020079+0000","last_peered":"2024-03-24T12:10:55.020079+0000","last_clean":"2024-03-24T12:10:55.020079+0000","last_became_active":"2024-03-24T12:08:36.174663+0000","last_became_peered":"2024-03-24T12:08:36.174663+0000","last_unstale":"2024-03-24T12:10:55.020079+0000","last_undegraded":"2024-03-24T12:10:55.020079+0000","last_fullsized":"2024-03-24T12:10:55.020079+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:12.689469+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.689469+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:23.823982+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":546,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.010301+0000","last_change":"2024-03-24T12:10:08.168976+0000","last_active":"2024-03-24T12:10:52.010301+0000","last_peered":"2024-03-24T12:10:52.010301+0000","last_clean":"2024-03-24T12:10:52.010301+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:10:52.010301+0000","last_undegraded":"2024-03-24T12:10:52.010301+0000","last_fullsized":"2024-03-24T12:10:52.010301+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:08.168832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:10:08.168832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:36.496494+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":791,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.009525+0000","last_change":"2024-03-24T12:09:21.734702+0000","last_active":"2024-03-24T12:10:52.009525+0000","last_peered":"2024-03-24T12:10:52.009525+0000","last_clean":"2024-03-24T12:10:52.009525+0000","last_became_active":"2024-03-24T12:08:36.145936+0000","last_became_peered":"2024-03-24T12:08:36.145936+0000","last_unstale":"2024-03-24T12:10:52.009525+0000","last_undegraded":"2024-03-24T12:10:52.009525+0000","last_fullsized":"2024-03-24T12:10:52.009525+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:21.734396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:09:21.734396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.181069+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":879,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019613+0000","last_change":"2024-03-24T12:10:36.965706+0000","last_active":"2024-03-24T12:10:55.019613+0000","last_peered":"2024-03-24T12:10:55.019613+0000","last_clean":"2024-03-24T12:10:55.019613+0000","last_became_active":"2024-03-24T12:10:36.965307+0000","last_became_peered":"2024-03-24T12:10:36.965307+0000","last_unstale":"2024-03-24T12:10:55.019613+0000","last_undegraded":"2024-03-24T12:10:55.019613+0000","last_fullsized":"2024-03-24T12:10:55.019613+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:07:32.602779+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:07:32.602779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:57.976443+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"428'623","reported_seq":655,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:53.590177+0000","last_change":"2024-03-24T12:10:42.558716+0000","last_active":"2024-03-24T12:10:53.590177+0000","last_peered":"2024-03-24T12:10:53.590177+0000","last_clean":"2024-03-24T12:10:53.590177+0000","last_became_active":"2024-03-24T12:10:42.558369+0000","last_became_peered":"2024-03-24T12:10:42.558369+0000","last_unstale":"2024-03-24T12:10:53.590177+0000","last_undegraded":"2024-03-24T12:10:53.590177+0000","last_fullsized":"2024-03-24T12:10:53.590177+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.999822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40828928,"num_objects":623,"num_object_clones":0,"num_object_copies":1869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":623,"num_write_kb":39872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":813,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.012044+0000","last_change":"2024-03-24T12:09:36.374858+0000","last_active":"2024-03-24T12:10:55.012044+0000","last_peered":"2024-03-24T12:10:55.012044+0000","last_clean":"2024-03-24T12:10:55.012044+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:10:55.012044+0000","last_undegraded":"2024-03-24T12:10:55.012044+0000","last_fullsized":"2024-03-24T12:10:55.012044+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:36.374505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:09:36.374505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:05.979873+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":850,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015444+0000","last_change":"2024-03-24T12:08:34.327181+0000","last_active":"2024-03-24T12:10:55.015444+0000","last_peered":"2024-03-24T12:10:55.015444+0000","last_clean":"2024-03-24T12:10:55.015444+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:10:55.015444+0000","last_undegraded":"2024-03-24T12:10:55.015444+0000","last_fullsized":"2024-03-24T12:10:55.015444+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:08:34.327037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.327037+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600860+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":808,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016473+0000","last_change":"2024-03-24T12:07:45.216727+0000","last_active":"2024-03-24T12:10:55.016473+0000","last_peered":"2024-03-24T12:10:55.016473+0000","last_clean":"2024-03-24T12:10:55.016473+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:10:55.016473+0000","last_undegraded":"2024-03-24T12:10:55.016473+0000","last_fullsized":"2024-03-24T12:10:55.016473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:07:45.216581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.216581+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600731+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":779,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015101+0000","last_change":"2024-03-24T12:07:47.318834+0000","last_active":"2024-03-24T12:10:55.015101+0000","last_peered":"2024-03-24T12:10:55.015101+0000","last_clean":"2024-03-24T12:10:55.015101+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:10:55.015101+0000","last_undegraded":"2024-03-24T12:10:55.015101+0000","last_fullsized":"2024-03-24T12:10:55.015101+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:47.318040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:07:47.318040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":827,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.011568+0000","last_change":"2024-03-24T12:10:15.054613+0000","last_active":"2024-03-24T12:10:52.011568+0000","last_peered":"2024-03-24T12:10:52.011568+0000","last_clean":"2024-03-24T12:10:52.011568+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:10:52.011568+0000","last_undegraded":"2024-03-24T12:10:52.011568+0000","last_fullsized":"2024-03-24T12:10:52.011568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:10:15.054488+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:10:15.054488+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:23.002252+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":284,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.009688+0000","last_change":"2024-03-24T12:09:36.778664+0000","last_active":"2024-03-24T12:10:52.009688+0000","last_peered":"2024-03-24T12:10:52.009688+0000","last_clean":"2024-03-24T12:10:52.009688+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:10:52.009688+0000","last_undegraded":"2024-03-24T12:10:52.009688+0000","last_fullsized":"2024-03-24T12:10:52.009688+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:36.778553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:09:36.778553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.180973+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":782,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016348+0000","last_change":"2024-03-24T12:10:30.721092+0000","last_active":"2024-03-24T12:10:55.016348+0000","last_peered":"2024-03-24T12:10:55.016348+0000","last_clean":"2024-03-24T12:10:55.016348+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:10:55.016348+0000","last_undegraded":"2024-03-24T12:10:55.016348+0000","last_fullsized":"2024-03-24T12:10:55.016348+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:30.721003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:30.721003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:47.664229+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":277,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015577+0000","last_change":"2024-03-24T12:07:50.278024+0000","last_active":"2024-03-24T12:10:55.015577+0000","last_peered":"2024-03-24T12:10:55.015577+0000","last_clean":"2024-03-24T12:10:55.015577+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:10:55.015577+0000","last_undegraded":"2024-03-24T12:10:55.015577+0000","last_fullsized":"2024-03-24T12:10:55.015577+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:50.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:50.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600825+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":812,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016172+0000","last_change":"2024-03-24T12:10:25.685130+0000","last_active":"2024-03-24T12:10:55.016172+0000","last_peered":"2024-03-24T12:10:55.016172+0000","last_clean":"2024-03-24T12:10:55.016172+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:10:55.016172+0000","last_undegraded":"2024-03-24T12:10:55.016172+0000","last_fullsized":"2024-03-24T12:10:55.016172+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:10:25.685009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.685009+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:42.323270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":812,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019276+0000","last_change":"2024-03-24T12:08:30.827983+0000","last_active":"2024-03-24T12:10:55.019276+0000","last_peered":"2024-03-24T12:10:55.019276+0000","last_clean":"2024-03-24T12:10:55.019276+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:10:55.019276+0000","last_undegraded":"2024-03-24T12:10:55.019276+0000","last_fullsized":"2024-03-24T12:10:55.019276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:30.827866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:08:30.827866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962967+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":630,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015563+0000","last_change":"2024-03-24T12:10:25.718633+0000","last_active":"2024-03-24T12:10:52.015563+0000","last_peered":"2024-03-24T12:10:52.015563+0000","last_clean":"2024-03-24T12:10:52.015563+0000","last_became_active":"2024-03-24T12:08:36.165715+0000","last_became_peered":"2024-03-24T12:08:36.165715+0000","last_unstale":"2024-03-24T12:10:52.015563+0000","last_undegraded":"2024-03-24T12:10:52.015563+0000","last_fullsized":"2024-03-24T12:10:52.015563+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:25.718555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.718555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:33.522109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":804,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.021826+0000","last_change":"2024-03-24T12:10:19.030271+0000","last_active":"2024-03-24T12:10:55.021826+0000","last_peered":"2024-03-24T12:10:55.021826+0000","last_clean":"2024-03-24T12:10:55.021826+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:10:55.021826+0000","last_undegraded":"2024-03-24T12:10:55.021826+0000","last_fullsized":"2024-03-24T12:10:55.021826+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.030172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.030172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:44.266650+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":829,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016023+0000","last_change":"2024-03-24T12:10:28.696002+0000","last_active":"2024-03-24T12:10:55.016023+0000","last_peered":"2024-03-24T12:10:55.016023+0000","last_clean":"2024-03-24T12:10:55.016023+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:10:55.016023+0000","last_undegraded":"2024-03-24T12:10:55.016023+0000","last_fullsized":"2024-03-24T12:10:55.016023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:28.695890+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:10:28.695890+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:57.427147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":674,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014919+0000","last_change":"2024-03-24T12:08:36.171996+0000","last_active":"2024-03-24T12:10:52.014919+0000","last_peered":"2024-03-24T12:10:52.014919+0000","last_clean":"2024-03-24T12:10:52.014919+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:10:52.014919+0000","last_undegraded":"2024-03-24T12:10:52.014919+0000","last_fullsized":"2024-03-24T12:10:52.014919+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:59.284540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:04:59.284540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":921,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014730+0000","last_change":"2024-03-24T12:10:15.754439+0000","last_active":"2024-03-24T12:10:52.014730+0000","last_peered":"2024-03-24T12:10:52.014730+0000","last_clean":"2024-03-24T12:10:52.014730+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:10:52.014730+0000","last_undegraded":"2024-03-24T12:10:52.014730+0000","last_fullsized":"2024-03-24T12:10:52.014730+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:15.754352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:15.754352+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:39.937789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":782,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.011357+0000","last_change":"2024-03-24T12:07:58.405060+0000","last_active":"2024-03-24T12:10:52.011357+0000","last_peered":"2024-03-24T12:10:52.011357+0000","last_clean":"2024-03-24T12:10:52.011357+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:10:52.011357+0000","last_undegraded":"2024-03-24T12:10:52.011357+0000","last_fullsized":"2024-03-24T12:10:52.011357+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:58.404932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:58.404932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.180872+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":698,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015204+0000","last_change":"2024-03-24T12:08:36.154109+0000","last_active":"2024-03-24T12:10:52.015204+0000","last_peered":"2024-03-24T12:10:52.015204+0000","last_clean":"2024-03-24T12:10:52.015204+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:10:52.015204+0000","last_undegraded":"2024-03-24T12:10:52.015204+0000","last_fullsized":"2024-03-24T12:10:52.015204+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":677,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.010688+0000","last_change":"2024-03-24T12:10:34.022259+0000","last_active":"2024-03-24T12:10:52.010688+0000","last_peered":"2024-03-24T12:10:52.010688+0000","last_clean":"2024-03-24T12:10:52.010688+0000","last_became_active":"2024-03-24T12:10:30.443191+0000","last_became_peered":"2024-03-24T12:10:30.443191+0000","last_unstale":"2024-03-24T12:10:52.010688+0000","last_undegraded":"2024-03-24T12:10:52.010688+0000","last_fullsized":"2024-03-24T12:10:52.010688+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:34.022182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:34.022182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:48.597395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":1033,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014343+0000","last_change":"2024-03-24T12:10:19.664090+0000","last_active":"2024-03-24T12:10:55.014343+0000","last_peered":"2024-03-24T12:10:55.014343+0000","last_clean":"2024-03-24T12:10:55.014343+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:10:55.014343+0000","last_undegraded":"2024-03-24T12:10:55.014343+0000","last_fullsized":"2024-03-24T12:10:55.014343+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:10:19.664009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.664009+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:45.539793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":792,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.017293+0000","last_change":"2024-03-24T12:10:17.566669+0000","last_active":"2024-03-24T12:10:55.017293+0000","last_peered":"2024-03-24T12:10:55.017293+0000","last_clean":"2024-03-24T12:10:55.017293+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:10:55.017293+0000","last_undegraded":"2024-03-24T12:10:55.017293+0000","last_fullsized":"2024-03-24T12:10:55.017293+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:17.566562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:10:17.566562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:26.844131+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":592,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.018579+0000","last_change":"2024-03-24T12:07:38.607052+0000","last_active":"2024-03-24T12:10:55.018579+0000","last_peered":"2024-03-24T12:10:55.018579+0000","last_clean":"2024-03-24T12:10:55.018579+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:10:55.018579+0000","last_undegraded":"2024-03-24T12:10:55.018579+0000","last_fullsized":"2024-03-24T12:10:55.018579+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:38.606914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:38.606914+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962772+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":812,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019313+0000","last_change":"2024-03-24T12:09:37.716173+0000","last_active":"2024-03-24T12:10:55.019313+0000","last_peered":"2024-03-24T12:10:55.019313+0000","last_clean":"2024-03-24T12:10:55.019313+0000","last_became_active":"2024-03-24T12:08:36.152805+0000","last_became_peered":"2024-03-24T12:08:36.152805+0000","last_unstale":"2024-03-24T12:10:55.019313+0000","last_undegraded":"2024-03-24T12:10:55.019313+0000","last_fullsized":"2024-03-24T12:10:55.019313+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:37.716096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:09:37.716096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.963019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":788,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014992+0000","last_change":"2024-03-24T12:07:29.080020+0000","last_active":"2024-03-24T12:10:55.014992+0000","last_peered":"2024-03-24T12:10:55.014992+0000","last_clean":"2024-03-24T12:10:55.014992+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:10:55.014992+0000","last_undegraded":"2024-03-24T12:10:55.014992+0000","last_fullsized":"2024-03-24T12:10:55.014992+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:29.079435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:07:29.079435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":791,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.009408+0000","last_change":"2024-03-24T12:09:30.837253+0000","last_active":"2024-03-24T12:10:52.009408+0000","last_peered":"2024-03-24T12:10:52.009408+0000","last_clean":"2024-03-24T12:10:52.009408+0000","last_became_active":"2024-03-24T12:08:36.146995+0000","last_became_peered":"2024-03-24T12:08:36.146995+0000","last_unstale":"2024-03-24T12:10:52.009408+0000","last_undegraded":"2024-03-24T12:10:52.009408+0000","last_fullsized":"2024-03-24T12:10:52.009408+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:30.836662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:30.836662+0000","last_clean_scrub_stamp":"2024-03-24T12:09:30.836662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:56.181100+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":814,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015528+0000","last_change":"2024-03-24T12:06:50.625611+0000","last_active":"2024-03-24T12:10:55.015528+0000","last_peered":"2024-03-24T12:10:55.015528+0000","last_clean":"2024-03-24T12:10:55.015528+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:10:55.015528+0000","last_undegraded":"2024-03-24T12:10:55.015528+0000","last_fullsized":"2024-03-24T12:10:55.015528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:06:50.625428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:06:50.625428+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600421+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":830,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.020505+0000","last_change":"2024-03-24T12:10:33.780424+0000","last_active":"2024-03-24T12:10:55.020505+0000","last_peered":"2024-03-24T12:10:55.020505+0000","last_clean":"2024-03-24T12:10:55.020505+0000","last_became_active":"2024-03-24T12:08:36.174093+0000","last_became_peered":"2024-03-24T12:08:36.174093+0000","last_unstale":"2024-03-24T12:10:55.020505+0000","last_undegraded":"2024-03-24T12:10:55.020505+0000","last_fullsized":"2024-03-24T12:10:55.020505+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:33.780343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:33.780343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:42.630686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":433,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014423+0000","last_change":"2024-03-24T12:09:38.661845+0000","last_active":"2024-03-24T12:10:52.014423+0000","last_peered":"2024-03-24T12:10:52.014423+0000","last_clean":"2024-03-24T12:10:52.014423+0000","last_became_active":"2024-03-24T12:08:36.166281+0000","last_became_peered":"2024-03-24T12:08:36.166281+0000","last_unstale":"2024-03-24T12:10:52.014423+0000","last_undegraded":"2024-03-24T12:10:52.014423+0000","last_fullsized":"2024-03-24T12:10:52.014423+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.661755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.661755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":787,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014948+0000","last_change":"2024-03-24T12:10:10.700383+0000","last_active":"2024-03-24T12:10:55.014948+0000","last_peered":"2024-03-24T12:10:55.014948+0000","last_clean":"2024-03-24T12:10:55.014948+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:10:55.014948+0000","last_undegraded":"2024-03-24T12:10:55.014948+0000","last_fullsized":"2024-03-24T12:10:55.014948+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:10.700237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.700237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:34.820904+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":808,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014113+0000","last_change":"2024-03-24T12:07:56.051237+0000","last_active":"2024-03-24T12:10:55.014113+0000","last_peered":"2024-03-24T12:10:55.014113+0000","last_clean":"2024-03-24T12:10:55.014113+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:10:55.014113+0000","last_undegraded":"2024-03-24T12:10:55.014113+0000","last_fullsized":"2024-03-24T12:10:55.014113+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:07:56.051061+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.051061+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":694,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.015379+0000","last_change":"2024-03-24T12:10:12.784984+0000","last_active":"2024-03-24T12:10:52.015379+0000","last_peered":"2024-03-24T12:10:52.015379+0000","last_clean":"2024-03-24T12:10:52.015379+0000","last_became_active":"2024-03-24T12:08:36.166947+0000","last_became_peered":"2024-03-24T12:08:36.166947+0000","last_unstale":"2024-03-24T12:10:52.015379+0000","last_undegraded":"2024-03-24T12:10:52.015379+0000","last_fullsized":"2024-03-24T12:10:52.015379+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.784866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.784866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:40.172720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":803,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019096+0000","last_change":"2024-03-24T12:10:37.907768+0000","last_active":"2024-03-24T12:10:55.019096+0000","last_peered":"2024-03-24T12:10:55.019096+0000","last_clean":"2024-03-24T12:10:55.019096+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:10:55.019096+0000","last_undegraded":"2024-03-24T12:10:55.019096+0000","last_fullsized":"2024-03-24T12:10:55.019096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:37.907635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:37.907635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:47.259474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":801,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.021308+0000","last_change":"2024-03-24T12:10:35.026578+0000","last_active":"2024-03-24T12:10:55.021308+0000","last_peered":"2024-03-24T12:10:55.021308+0000","last_clean":"2024-03-24T12:10:55.021308+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:10:55.021308+0000","last_undegraded":"2024-03-24T12:10:55.021308+0000","last_fullsized":"2024-03-24T12:10:55.021308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:35.026455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.026455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:43.605666+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":817,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016324+0000","last_change":"2024-03-24T12:10:22.696939+0000","last_active":"2024-03-24T12:10:55.016324+0000","last_peered":"2024-03-24T12:10:55.016324+0000","last_clean":"2024-03-24T12:10:55.016324+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:10:55.016324+0000","last_undegraded":"2024-03-24T12:10:55.016324+0000","last_fullsized":"2024-03-24T12:10:55.016324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:10:22.696749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:22.696749+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:41.327017+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":810,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019717+0000","last_change":"2024-03-24T12:09:38.737527+0000","last_active":"2024-03-24T12:10:55.019717+0000","last_peered":"2024-03-24T12:10:55.019717+0000","last_clean":"2024-03-24T12:10:55.019717+0000","last_became_active":"2024-03-24T12:08:36.157290+0000","last_became_peered":"2024-03-24T12:08:36.157290+0000","last_unstale":"2024-03-24T12:10:55.019717+0000","last_undegraded":"2024-03-24T12:10:55.019717+0000","last_fullsized":"2024-03-24T12:10:55.019717+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.737417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":805,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.020615+0000","last_change":"2024-03-24T12:10:23.017019+0000","last_active":"2024-03-24T12:10:55.020615+0000","last_peered":"2024-03-24T12:10:55.020615+0000","last_clean":"2024-03-24T12:10:55.020615+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:10:55.020615+0000","last_undegraded":"2024-03-24T12:10:55.020615+0000","last_fullsized":"2024-03-24T12:10:55.020615+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:23.016918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.016918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:39.395453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":778,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015192+0000","last_change":"2024-03-24T12:10:13.863177+0000","last_active":"2024-03-24T12:10:55.015192+0000","last_peered":"2024-03-24T12:10:55.015192+0000","last_clean":"2024-03-24T12:10:55.015192+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:10:55.015192+0000","last_undegraded":"2024-03-24T12:10:55.015192+0000","last_fullsized":"2024-03-24T12:10:55.015192+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.862457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.862457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:41.076796+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":835,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.011891+0000","last_change":"2024-03-24T12:10:35.061280+0000","last_active":"2024-03-24T12:10:52.011891+0000","last_peered":"2024-03-24T12:10:52.011891+0000","last_clean":"2024-03-24T12:10:52.011891+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:10:52.011891+0000","last_undegraded":"2024-03-24T12:10:52.011891+0000","last_fullsized":"2024-03-24T12:10:52.011891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:35.061151+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.061151+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.815059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":646,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014027+0000","last_change":"2024-03-24T12:10:24.690361+0000","last_active":"2024-03-24T12:10:52.014027+0000","last_peered":"2024-03-24T12:10:52.014027+0000","last_clean":"2024-03-24T12:10:52.014027+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:10:52.014027+0000","last_undegraded":"2024-03-24T12:10:52.014027+0000","last_fullsized":"2024-03-24T12:10:52.014027+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.690274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.690274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:39.686088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":1033,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.020001+0000","last_change":"2024-03-24T12:07:51.843434+0000","last_active":"2024-03-24T12:10:55.020001+0000","last_peered":"2024-03-24T12:10:55.020001+0000","last_clean":"2024-03-24T12:10:55.020001+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:10:55.020001+0000","last_undegraded":"2024-03-24T12:10:55.020001+0000","last_fullsized":"2024-03-24T12:10:55.020001+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:07:51.843305+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.843305+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962823+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":820,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.023331+0000","last_change":"2024-03-24T12:10:27.009973+0000","last_active":"2024-03-24T12:10:55.023331+0000","last_peered":"2024-03-24T12:10:55.023331+0000","last_clean":"2024-03-24T12:10:55.023331+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:10:55.023331+0000","last_undegraded":"2024-03-24T12:10:55.023331+0000","last_fullsized":"2024-03-24T12:10:55.023331+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:10:27.009877+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:27.009877+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:48.532733+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":848,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.020049+0000","last_change":"2024-03-24T12:10:15.799447+0000","last_active":"2024-03-24T12:10:55.020049+0000","last_peered":"2024-03-24T12:10:55.020049+0000","last_clean":"2024-03-24T12:10:55.020049+0000","last_became_active":"2024-03-24T12:08:36.156563+0000","last_became_peered":"2024-03-24T12:08:36.156563+0000","last_unstale":"2024-03-24T12:10:55.020049+0000","last_undegraded":"2024-03-24T12:10:55.020049+0000","last_fullsized":"2024-03-24T12:10:55.020049+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:10:15.799348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:15.799348+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:25.282115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":791,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015629+0000","last_change":"2024-03-24T12:10:23.652133+0000","last_active":"2024-03-24T12:10:55.015629+0000","last_peered":"2024-03-24T12:10:55.015629+0000","last_clean":"2024-03-24T12:10:55.015629+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:10:55.015629+0000","last_undegraded":"2024-03-24T12:10:55.015629+0000","last_fullsized":"2024-03-24T12:10:55.015629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:10:23.652045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.652045+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:33.826223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":802,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014169+0000","last_change":"2024-03-24T12:10:17.070842+0000","last_active":"2024-03-24T12:10:52.014169+0000","last_peered":"2024-03-24T12:10:52.014169+0000","last_clean":"2024-03-24T12:10:52.014169+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:10:52.014169+0000","last_undegraded":"2024-03-24T12:10:52.014169+0000","last_fullsized":"2024-03-24T12:10:52.014169+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:17.070732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:10:17.070732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:39.343304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":386,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014902+0000","last_change":"2024-03-24T12:08:36.167702+0000","last_active":"2024-03-24T12:10:52.014902+0000","last_peered":"2024-03-24T12:10:52.014902+0000","last_clean":"2024-03-24T12:10:52.014902+0000","last_became_active":"2024-03-24T12:08:36.166973+0000","last_became_peered":"2024-03-24T12:08:36.166973+0000","last_unstale":"2024-03-24T12:10:52.014902+0000","last_undegraded":"2024-03-24T12:10:52.014902+0000","last_fullsized":"2024-03-24T12:10:52.014902+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:04:14.188600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":741,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014462+0000","last_change":"2024-03-24T12:07:42.113052+0000","last_active":"2024-03-24T12:10:55.014462+0000","last_peered":"2024-03-24T12:10:55.014462+0000","last_clean":"2024-03-24T12:10:55.014462+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:10:55.014462+0000","last_undegraded":"2024-03-24T12:10:55.014462+0000","last_fullsized":"2024-03-24T12:10:55.014462+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:07:42.112569+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:07:42.112569+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":803,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.018831+0000","last_change":"2024-03-24T12:06:57.653082+0000","last_active":"2024-03-24T12:10:55.018831+0000","last_peered":"2024-03-24T12:10:55.018831+0000","last_clean":"2024-03-24T12:10:55.018831+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:10:55.018831+0000","last_undegraded":"2024-03-24T12:10:55.018831+0000","last_fullsized":"2024-03-24T12:10:55.018831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:57.653004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:06:57.653004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.962661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":548,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016470+0000","last_change":"2024-03-24T12:07:53.354298+0000","last_active":"2024-03-24T12:10:55.016470+0000","last_peered":"2024-03-24T12:10:55.016470+0000","last_clean":"2024-03-24T12:10:55.016470+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:10:55.016470+0000","last_undegraded":"2024-03-24T12:10:55.016470+0000","last_fullsized":"2024-03-24T12:10:55.016470+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:07:53.354213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":797,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016405+0000","last_change":"2024-03-24T12:10:13.573691+0000","last_active":"2024-03-24T12:10:55.016405+0000","last_peered":"2024-03-24T12:10:55.016405+0000","last_clean":"2024-03-24T12:10:55.016405+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:10:55.016405+0000","last_undegraded":"2024-03-24T12:10:55.016405+0000","last_fullsized":"2024-03-24T12:10:55.016405+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.573504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.573504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:23.750544+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":841,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.019423+0000","last_change":"2024-03-24T12:07:41.142569+0000","last_active":"2024-03-24T12:10:55.019423+0000","last_peered":"2024-03-24T12:10:55.019423+0000","last_clean":"2024-03-24T12:10:55.019423+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:10:55.019423+0000","last_undegraded":"2024-03-24T12:10:55.019423+0000","last_fullsized":"2024-03-24T12:10:55.019423+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:07:41.142421+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.142421+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":796,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.014900+0000","last_change":"2024-03-24T12:07:32.002134+0000","last_active":"2024-03-24T12:10:55.014900+0000","last_peered":"2024-03-24T12:10:55.014900+0000","last_clean":"2024-03-24T12:10:55.014900+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:10:55.014900+0000","last_undegraded":"2024-03-24T12:10:55.014900+0000","last_fullsized":"2024-03-24T12:10:55.014900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:07:32.001718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:32.001718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":785,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015699+0000","last_change":"2024-03-24T12:06:42.722854+0000","last_active":"2024-03-24T12:10:55.015699+0000","last_peered":"2024-03-24T12:10:55.015699+0000","last_clean":"2024-03-24T12:10:55.015699+0000","last_became_active":"2024-03-24T12:05:44.736383+0000","last_became_peered":"2024-03-24T12:05:44.736383+0000","last_unstale":"2024-03-24T12:10:55.015699+0000","last_undegraded":"2024-03-24T12:10:55.015699+0000","last_fullsized":"2024-03-24T12:10:55.015699+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:42.722733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:42.722733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":788,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016077+0000","last_change":"2024-03-24T12:08:39.416538+0000","last_active":"2024-03-24T12:10:55.016077+0000","last_peered":"2024-03-24T12:10:55.016077+0000","last_clean":"2024-03-24T12:10:55.016077+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:10:55.016077+0000","last_undegraded":"2024-03-24T12:10:55.016077+0000","last_fullsized":"2024-03-24T12:10:55.016077+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:39.416426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:39.416426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":816,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015422+0000","last_change":"2024-03-24T12:07:40.191735+0000","last_active":"2024-03-24T12:10:55.015422+0000","last_peered":"2024-03-24T12:10:55.015422+0000","last_clean":"2024-03-24T12:10:55.015422+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:10:55.015422+0000","last_undegraded":"2024-03-24T12:10:55.015422+0000","last_fullsized":"2024-03-24T12:10:55.015422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:40.191513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.191513+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600636+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":825,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.023020+0000","last_change":"2024-03-24T12:10:25.001677+0000","last_active":"2024-03-24T12:10:55.023020+0000","last_peered":"2024-03-24T12:10:55.023020+0000","last_clean":"2024-03-24T12:10:55.023020+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:10:55.023020+0000","last_undegraded":"2024-03-24T12:10:55.023020+0000","last_fullsized":"2024-03-24T12:10:55.023020+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:25.001566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.001566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:38.605807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":707,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014619+0000","last_change":"2024-03-24T12:10:23.722476+0000","last_active":"2024-03-24T12:10:52.014619+0000","last_peered":"2024-03-24T12:10:52.014619+0000","last_clean":"2024-03-24T12:10:52.014619+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:10:52.014619+0000","last_undegraded":"2024-03-24T12:10:52.014619+0000","last_fullsized":"2024-03-24T12:10:52.014619+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:23.722321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.722321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:48.576677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":782,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.009998+0000","last_change":"2024-03-24T12:10:20.093443+0000","last_active":"2024-03-24T12:10:52.009998+0000","last_peered":"2024-03-24T12:10:52.009998+0000","last_clean":"2024-03-24T12:10:52.009998+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:10:52.009998+0000","last_undegraded":"2024-03-24T12:10:52.009998+0000","last_fullsized":"2024-03-24T12:10:52.009998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:20.093338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:10:20.093338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:42.578413+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":806,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015679+0000","last_change":"2024-03-24T12:06:56.609759+0000","last_active":"2024-03-24T12:10:55.015679+0000","last_peered":"2024-03-24T12:10:55.015679+0000","last_clean":"2024-03-24T12:10:55.015679+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:10:55.015679+0000","last_undegraded":"2024-03-24T12:10:55.015679+0000","last_fullsized":"2024-03-24T12:10:55.015679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:06:56.609621+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:06:56.609621+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600501+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":821,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.022050+0000","last_change":"2024-03-24T12:09:16.215114+0000","last_active":"2024-03-24T12:10:55.022050+0000","last_peered":"2024-03-24T12:10:55.022050+0000","last_clean":"2024-03-24T12:10:55.022050+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:10:55.022050+0000","last_undegraded":"2024-03-24T12:10:55.022050+0000","last_fullsized":"2024-03-24T12:10:55.022050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:16.214991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:09:16.214991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":813,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.021042+0000","last_change":"2024-03-24T12:09:38.012112+0000","last_active":"2024-03-24T12:10:55.021042+0000","last_peered":"2024-03-24T12:10:55.021042+0000","last_clean":"2024-03-24T12:10:55.021042+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:10:55.021042+0000","last_undegraded":"2024-03-24T12:10:55.021042+0000","last_fullsized":"2024-03-24T12:10:55.021042+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.011992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.011992+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.011992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":713,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.014366+0000","last_change":"2024-03-24T12:10:13.888112+0000","last_active":"2024-03-24T12:10:52.014366+0000","last_peered":"2024-03-24T12:10:52.014366+0000","last_clean":"2024-03-24T12:10:52.014366+0000","last_became_active":"2024-03-24T12:08:36.164166+0000","last_became_peered":"2024-03-24T12:08:36.164166+0000","last_unstale":"2024-03-24T12:10:52.014366+0000","last_undegraded":"2024-03-24T12:10:52.014366+0000","last_fullsized":"2024-03-24T12:10:52.014366+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.888002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.888002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:20.276623+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":820,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015202+0000","last_change":"2024-03-24T12:07:51.280566+0000","last_active":"2024-03-24T12:10:55.015202+0000","last_peered":"2024-03-24T12:10:55.015202+0000","last_clean":"2024-03-24T12:10:55.015202+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:10:55.015202+0000","last_undegraded":"2024-03-24T12:10:55.015202+0000","last_fullsized":"2024-03-24T12:10:55.015202+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:07:51.280424+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:07:51.280424+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:59.600582+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":801,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.017107+0000","last_change":"2024-03-24T12:08:40.183479+0000","last_active":"2024-03-24T12:10:55.017107+0000","last_peered":"2024-03-24T12:10:55.017107+0000","last_clean":"2024-03-24T12:10:55.017107+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:10:55.017107+0000","last_undegraded":"2024-03-24T12:10:55.017107+0000","last_fullsized":"2024-03-24T12:10:55.017107+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.183311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.183311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":278,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.018750+0000","last_change":"2024-03-24T12:10:24.707449+0000","last_active":"2024-03-24T12:10:55.018750+0000","last_peered":"2024-03-24T12:10:55.018750+0000","last_clean":"2024-03-24T12:10:55.018750+0000","last_became_active":"2024-03-24T12:08:36.152717+0000","last_became_peered":"2024-03-24T12:08:36.152717+0000","last_unstale":"2024-03-24T12:10:55.018750+0000","last_undegraded":"2024-03-24T12:10:55.018750+0000","last_fullsized":"2024-03-24T12:10:55.018750+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.707367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.707367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:37.505000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":792,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015379+0000","last_change":"2024-03-24T12:10:12.547927+0000","last_active":"2024-03-24T12:10:55.015379+0000","last_peered":"2024-03-24T12:10:55.015379+0000","last_clean":"2024-03-24T12:10:55.015379+0000","last_became_active":"2024-03-24T12:08:36.165334+0000","last_became_peered":"2024-03-24T12:08:36.165334+0000","last_unstale":"2024-03-24T12:10:55.015379+0000","last_undegraded":"2024-03-24T12:10:55.015379+0000","last_fullsized":"2024-03-24T12:10:55.015379+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.547803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.547803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:16.347328+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":867,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.016350+0000","last_change":"2024-03-24T12:08:47.101220+0000","last_active":"2024-03-24T12:10:55.016350+0000","last_peered":"2024-03-24T12:10:55.016350+0000","last_clean":"2024-03-24T12:10:55.016350+0000","last_became_active":"2024-03-24T12:08:36.154117+0000","last_became_peered":"2024-03-24T12:08:36.154117+0000","last_unstale":"2024-03-24T12:10:55.016350+0000","last_undegraded":"2024-03-24T12:10:55.016350+0000","last_fullsized":"2024-03-24T12:10:55.016350+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:47.101082+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:08:47.101082+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":795,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-03-24T12:10:52.011561+0000","last_change":"2024-03-24T12:10:24.957781+0000","last_active":"2024-03-24T12:10:52.011561+0000","last_peered":"2024-03-24T12:10:52.011561+0000","last_clean":"2024-03-24T12:10:52.011561+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:10:52.011561+0000","last_undegraded":"2024-03-24T12:10:52.011561+0000","last_fullsized":"2024-03-24T12:10:52.011561+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.957699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.957699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:33.644775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":261,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.013761+0000","last_change":"2024-03-24T12:08:33.256620+0000","last_active":"2024-03-24T12:10:55.013761+0000","last_peered":"2024-03-24T12:10:55.013761+0000","last_clean":"2024-03-24T12:10:55.013761+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:10:55.013761+0000","last_undegraded":"2024-03-24T12:10:55.013761+0000","last_fullsized":"2024-03-24T12:10:55.013761+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:33.255322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:08:33.255322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":778,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-24T12:10:55.015949+0000","last_change":"2024-03-24T12:07:48.265595+0000","last_active":"2024-03-24T12:10:55.015949+0000","last_peered":"2024-03-24T12:10:55.015949+0000","last_clean":"2024-03-24T12:10:55.015949+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:10:55.015949+0000","last_undegraded":"2024-03-24T12:10:55.015949+0000","last_fullsized":"2024-03-24T12:10:55.015949+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:48.265295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:48.265295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1150156800,"num_objects":17550,"num_object_clones":0,"num_object_copies":52650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17550,"num_write_kb":1123200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3385851904,"data_stored":3385851904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17550,"ondisk_log_size":17550,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":390,"ondisk_log_size":390,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":318,"num_read_kb":313,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":792,"ondisk_log_size":792,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005284,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080424,"kb_used_data":417568,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92664728,"statfs":{"total":95995035648,"available":94888681472,"internally_reserved":0,"allocated":427589632,"data_stored":426765211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":678754762},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332331,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1187132,"kb_used_data":538160,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92558020,"statfs":{"total":95995035648,"available":94779412480,"internally_reserved":0,"allocated":551075840,"data_stored":550235321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":664533446},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575107,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1482868,"kb_used_data":775592,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92262284,"statfs":{"total":95995035648,"available":94476578816,"internally_reserved":0,"allocated":794206208,"data_stored":793372992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738632,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1026092,"kb_used_data":218844,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92719060,"statfs":{"total":95995035648,"available":94944317440,"internally_reserved":0,"allocated":224096256,"data_stored":223268213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411583,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732304,"kb_used_data":151364,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93012848,"statfs":{"total":95995035648,"available":95245156352,"internally_reserved":0,"allocated":154996736,"data_stored":154200258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117241,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265992,"kb_used_data":454588,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92479160,"statfs":{"total":95995035648,"available":94698659840,"internally_reserved":0,"allocated":465498112,"data_stored":464645810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":830863815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921012,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1070200,"kb_used_data":367084,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92674952,"statfs":{"total":95995035648,"available":94899150848,"internally_reserved":0,"allocated":375894016,"data_stored":375056739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455517,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":460716,"kb_used_data":171428,"kb_used_omap":7,"kb_used_meta":289272,"kb_avail":93284436,"statfs":{"total":95995035648,"available":95523262464,"internally_reserved":0,"allocated":175542272,"data_stored":174706882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":296214640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246743040,"data_stored":246743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":801898496,"data_stored":801898496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161415168,"data_stored":161415168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":506462208,"data_stored":506462208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":381550592,"data_stored":381550592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":203751424,"data_stored":203751424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":618332160,"data_stored":618332160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":465698816,"data_stored":465698816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:11:04.029 INFO:tasks.thrashosds.thrasher:Setting 5.1c to [0, 1, 3, 2, 6, 7] 2024-03-24T12:11:04.029 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.1c', '0', '1', '3', '2', '6', '7'] 2024-03-24T12:11:04.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 5.1c 0 1 3 2 6 7 2024-03-24T12:11:04.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:04.081 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:04.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:04.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:04.227 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:04.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:04.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:04.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:04 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:11:04.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:04 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:11:04.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:04 smithi064 ceph-mon[42243]: osdmap e432: 8 total, 8 up, 8 in 2024-03-24T12:11:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:04 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:11:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:04 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:11:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:04 smithi064 ceph-mon[30223]: osdmap e432: 8 total, 8 up, 8 in 2024-03-24T12:11:04.318 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 14 16 34428 34412 153.608 118.438 0.00529122 0.00650437 2024-03-24T12:11:04.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:04 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:11:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:04 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:11:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:04 smithi193 ceph-mon[33593]: osdmap e432: 8 total, 8 up, 8 in 2024-03-24T12:11:04.367 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:04.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:04.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:04.492 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:04.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:04.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:04.638 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:04.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:04.741 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:04.776 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:04.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:04.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:04.917 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:04.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:05.017 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:05.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:05.094 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:05.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:05.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:05.222 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:05 smithi064 ceph-mon[30223]: pgmap v898: 158 pgs: 158 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-24T12:11:05.222 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:05 smithi064 ceph-mon[42243]: pgmap v898: 158 pgs: 158 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-24T12:11:05.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:05.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:05.319 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 15 16 36238 36222 150.908 113.125 0.00330452 0.00662025 2024-03-24T12:11:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:05 smithi193 ceph-mon[33593]: pgmap v898: 158 pgs: 158 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-24T12:11:05.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:05.396 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:05.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:05.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:05.498 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:05.533 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:05.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:05.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:05.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:05.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:05.815 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:05.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:05.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:05.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:06.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:06.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:06.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:06.197 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:06.228 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:06.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:06.319 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 16 16 38124 38108 148.843 117.875 0.00608681 0.00671307 2024-03-24T12:11:06.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:06.358 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:06.365 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:06.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:06.448 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:11:06.449 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:11:06.467 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:06.508 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:06.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:06.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:11:06.610 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:06.646 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:06.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:06.747 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:06.794 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:06.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:06.895 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:06.931 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:06.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:07.033 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:07.068 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:07.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:07.170 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:07.210 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:07.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:07.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:07.320 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 17 16 39933 39917 146.726 113.062 0.00834044 0.00680886 2024-03-24T12:11:07.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:07.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:07.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:07 smithi064 ceph-mon[30223]: pgmap v899: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 684 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-24T12:11:07.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:07 smithi064 ceph-mon[30223]: osdmap e433: 8 total, 8 up, 8 in 2024-03-24T12:11:07.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:07 smithi064 ceph-mon[42243]: pgmap v899: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 684 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-24T12:11:07.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:07 smithi064 ceph-mon[42243]: osdmap e433: 8 total, 8 up, 8 in 2024-03-24T12:11:07.464 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:07.501 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:07.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:07 smithi193 ceph-mon[33593]: pgmap v899: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 684 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-24T12:11:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:07 smithi193 ceph-mon[33593]: osdmap e433: 8 total, 8 up, 8 in 2024-03-24T12:11:07.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:07.645 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:07.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:07.747 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:07.778 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:07.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:07.804 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:11:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:11:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:07.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:07.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:11:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:07.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:08.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:08.068 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:08.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:08.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:08.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:08.321 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 18 16 41740 41724 144.845 112.938 0.00713301 0.00689945 2024-03-24T12:11:08.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:08.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:11:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:11:08.360 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:08.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:08.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:08.633 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:08.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.777 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:08.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:08.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:08.915 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:09.017 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:09.044 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:09.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:09.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:09.185 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:09.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:09.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:09.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:09.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:09.321 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 19 16 43370 43354 142.583 101.875 0.00346215 0.00700813 2024-03-24T12:11:09.416 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:09 smithi064 ceph-mon[30223]: pgmap v901: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-24T12:11:09.416 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:09.417 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:09 smithi064 ceph-mon[42243]: pgmap v901: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-24T12:11:09.417 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:09.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:09.469 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:09.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:09.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:09.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:09 smithi193 ceph-mon[33593]: pgmap v901: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-24T12:11:09.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:09.604 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:09.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:09.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:09.742 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:09.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:09.845 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:09.870 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:09.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:09.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:10.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:10.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:11:10.136 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:10.171 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:10.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:10.193 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3096203157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:11:10.193 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3096203157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:11:10.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:10.307 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:10.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:10.321 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:11:10.322881+0000 min lat: 0.00124462 max lat: 0.101006 avg lat: 0.00713503 2024-03-24T12:11:10.321 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:11:10.322 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 20 16 44837 44821 140.038 91.6875 0.00877194 0.00713503 2024-03-24T12:11:10.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:10.452 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:10.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:10.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3096203157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:11:10.597 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:10.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:10.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:10.738 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:10.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:10.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:10.876 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:10.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:10.978 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:11.010 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:11.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:11.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:11.154 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:11.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:11.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:11.293 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:11.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:11.321 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 21 16 46586 46570 138.575 109.312 0.00669834 0.00721103 2024-03-24T12:11:11.395 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:11.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:11 smithi064 ceph-mon[30223]: pgmap v902: 158 pgs: 158 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-24T12:11:11.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:11 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:11.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:11 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3096203157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:11:11.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:11 smithi064 ceph-mon[30223]: osdmap e434: 8 total, 8 up, 8 in 2024-03-24T12:11:11.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:11 smithi064 ceph-mon[42243]: pgmap v902: 158 pgs: 158 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-24T12:11:11.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:11 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:11.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:11 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3096203157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:11:11.397 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:11 smithi064 ceph-mon[42243]: osdmap e434: 8 total, 8 up, 8 in 2024-03-24T12:11:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:11.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:11.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:11.534 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:11.569 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:11.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:11 smithi193 ceph-mon[33593]: pgmap v902: 158 pgs: 158 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-24T12:11:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:11 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3096203157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:11:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:11 smithi193 ceph-mon[33593]: osdmap e434: 8 total, 8 up, 8 in 2024-03-24T12:11:11.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:11.706 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:11.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:11.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:11.850 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:11.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:11.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:11.992 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:11.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:12.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:12.128 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:12.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:12.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:12.267 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:12.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:12.321 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 22 16 48300 48284 137.145 107.125 0.00475935 0.00728626 2024-03-24T12:11:12.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:12.407 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:12.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:12.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:12.546 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:12.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:12.648 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:12.682 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:12.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:12.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:12.819 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:12.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:12.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:12.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:12.965 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:12.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:13.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:13.102 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:13.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:13.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:12 smithi064 ceph-mon[30223]: pgmap v904: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-24T12:11:13.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:12 smithi064 ceph-mon[30223]: osdmap e435: 8 total, 8 up, 8 in 2024-03-24T12:11:13.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:12 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:13.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:12 smithi064 ceph-mon[42243]: pgmap v904: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-24T12:11:13.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:12 smithi064 ceph-mon[42243]: osdmap e435: 8 total, 8 up, 8 in 2024-03-24T12:11:13.154 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:12 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:13.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:13.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:11:13.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:13.241 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:13.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:13.322 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 23 16 49942 49926 135.639 102.625 0.0132635 0.00736704 2024-03-24T12:11:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:12 smithi193 ceph-mon[33593]: pgmap v904: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-24T12:11:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:12 smithi193 ceph-mon[33593]: osdmap e435: 8 total, 8 up, 8 in 2024-03-24T12:11:13.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:12 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:13.343 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:13.377 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:13.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:13.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:13.520 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:13.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:13.622 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:13.653 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:13.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:13.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:13.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:13.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:13.903 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:13.935 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:13.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:14.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:14.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:13 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:11:14.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:13 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:11:14.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:13 smithi064 ceph-mon[30223]: 6.d scrub starts 2024-03-24T12:11:14.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:13 smithi064 ceph-mon[30223]: 6.d scrub ok 2024-03-24T12:11:14.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:13 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:11:14.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:13 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:11:14.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:13 smithi064 ceph-mon[42243]: 6.d scrub starts 2024-03-24T12:11:14.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:13 smithi064 ceph-mon[42243]: 6.d scrub ok 2024-03-24T12:11:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:14.073 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:14.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:14.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:14.256 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:14.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:14.322 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 24 16 51412 51396 133.816 91.875 0.0183214 0.00746808 2024-03-24T12:11:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:13 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:11:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:13 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:11:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:13 smithi193 ceph-mon[33593]: 6.d scrub starts 2024-03-24T12:11:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:13 smithi193 ceph-mon[33593]: 6.d scrub ok 2024-03-24T12:11:14.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:14.390 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:14.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:14.492 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:14.526 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:14.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:14.629 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:14.662 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:14.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:14.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:14.802 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:14.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:14.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:14.930 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:14.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:14.993 INFO:teuthology.orchestra.run.smithi064.stderr:set 5.1c pg_upmap_items mapping to [0->1,3->2,6->7] 2024-03-24T12:11:15.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:15.066 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:15.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:15.168 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:15.201 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:15.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:15.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:15 smithi064 ceph-mon[42243]: pgmap v906: 158 pgs: 158 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 780 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:15.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:15 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T12:11:15.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:15 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T12:11:15.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:15 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3350532276' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1c", "id": [0, 1, 3, 2, 6, 7]}]: dispatch 2024-03-24T12:11:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:14 smithi064 ceph-mon[30223]: pgmap v906: 158 pgs: 158 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 780 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:14 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T12:11:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:14 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T12:11:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3350532276' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1c", "id": [0, 1, 3, 2, 6, 7]}]: dispatch 2024-03-24T12:11:15.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:15.323 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 25 16 52900 52884 132.183 93 0.00854959 0.00756038 2024-03-24T12:11:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:14 smithi193 ceph-mon[33593]: pgmap v906: 158 pgs: 158 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 780 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:14 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T12:11:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:14 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T12:11:15.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3350532276' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1c", "id": [0, 1, 3, 2, 6, 7]}]: dispatch 2024-03-24T12:11:15.338 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:15.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:15.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:15.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:15.481 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:15.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:15.582 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:15.617 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:15.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:15.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:15.757 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:15.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:15.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:15.908 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:15.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:16.010 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:16.013 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3350532276' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1c", "id": [0, 1, 3, 2, 6, 7]}]': finished 2024-03-24T12:11:16.014 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[30223]: osdmap e436: 8 total, 8 up, 8 in 2024-03-24T12:11:16.014 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[30223]: 6.1d scrub starts 2024-03-24T12:11:16.014 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[30223]: 6.1d scrub ok 2024-03-24T12:11:16.014 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3350532276' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1c", "id": [0, 1, 3, 2, 6, 7]}]': finished 2024-03-24T12:11:16.014 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[42243]: osdmap e436: 8 total, 8 up, 8 in 2024-03-24T12:11:16.014 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[42243]: 6.1d scrub starts 2024-03-24T12:11:16.014 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[42243]: 6.1d scrub ok 2024-03-24T12:11:16.014 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:11:16.042 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:16.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:16.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:16.186 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:16.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:16.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:16.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:11:16.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:11:16.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:16 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:11:16.322 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 26 16 54455 54439 130.837 97.1875 0.0167563 0.00763768 2024-03-24T12:11:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:16 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3350532276' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1c", "id": [0, 1, 3, 2, 6, 7]}]': finished 2024-03-24T12:11:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:16 smithi193 ceph-mon[33593]: osdmap e436: 8 total, 8 up, 8 in 2024-03-24T12:11:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:16 smithi193 ceph-mon[33593]: 6.1d scrub starts 2024-03-24T12:11:16.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:16 smithi193 ceph-mon[33593]: 6.1d scrub ok 2024-03-24T12:11:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:16 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:11:16.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:16 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:11:16.344 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:16.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:16.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:16.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:16.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:16.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:16.622 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:16.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:16.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:16.779 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:16.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:16.883 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:16.917 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:16.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:17.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:17.095 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:17.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:17.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:17.234 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:17.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-mon[30223]: pgmap v908: 158 pgs: 158 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-24T12:11:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T12:11:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T12:11:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-mon[30223]: osdmap e437: 8 total, 8 up, 8 in 2024-03-24T12:11:17.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-mon[42243]: pgmap v908: 158 pgs: 158 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-24T12:11:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T12:11:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T12:11:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-mon[42243]: osdmap e437: 8 total, 8 up, 8 in 2024-03-24T12:11:17.322 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 27 15 56017 56002 129.609 97.6875 0.008668 0.00771034 2024-03-24T12:11:17.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:17 smithi193 ceph-mon[33593]: pgmap v908: 158 pgs: 158 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-24T12:11:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:17 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T12:11:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:17 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T12:11:17.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:17 smithi193 ceph-mon[33593]: osdmap e437: 8 total, 8 up, 8 in 2024-03-24T12:11:17.371 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:17.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:17.474 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:17.507 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:17.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:17.609 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:17.647 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:17.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:17.672 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:11:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:11:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:17.747 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:17.795 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:17.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:17.797 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:11:17.797 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:11:17.813 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:17.897 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:17.934 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:17.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:17.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:18.036 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:18.067 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:18.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:18.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:18.190 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1707435196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T12:11:18.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1707435196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:18.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:18.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:18.322 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 28 16 57346 57330 127.944 83 0.0145167 0.00781096 2024-03-24T12:11:18.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:11:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:11:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:18 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T12:11:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:18 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T12:11:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:18 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T12:11:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:18 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T12:11:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1707435196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:18.354 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:18.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:18.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:18.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:18.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:18.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:18.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:18.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:18.768 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:18.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:18.870 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:18.908 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:18.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:19.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:19.053 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:19.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:19.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:19.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:19.202 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:19.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:19.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:11:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T12:11:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T12:11:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[30223]: pgmap v910: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-24T12:11:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T12:11:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T12:11:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T12:11:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T12:11:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[42243]: pgmap v910: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-24T12:11:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T12:11:19.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:19 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T12:11:19.303 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:19.323 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 29 16 58755 58739 126.568 88.0625 0.00905761 0.00789559 2024-03-24T12:11:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:19 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T12:11:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:19 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T12:11:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:19 smithi193 ceph-mon[33593]: pgmap v910: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-24T12:11:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:19 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T12:11:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:19 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T12:11:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:19.337 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:19.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:19.373 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:11:19.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:11:19.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:11:19.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:11:19.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:11:19.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:11:19.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:11:19.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:11:19.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:11:19.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:11:19.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:11:19.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:11:19.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:11:19.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:11:19.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:11:19.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:11:19.438 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:19.471 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:19.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:19.572 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:19.604 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:19.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:19.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:19.740 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:19.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:19.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:19.883 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:19.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:19.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:20.020 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:20.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:20.121 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:20.148 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:20.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:20.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:20.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T12:11:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T12:11:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[30223]: 1.7 scrub starts 2024-03-24T12:11:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[30223]: 1.7 scrub ok 2024-03-24T12:11:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[30223]: 2.2 scrub starts 2024-03-24T12:11:20.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[30223]: 2.2 scrub ok 2024-03-24T12:11:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T12:11:20.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T12:11:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[42243]: 1.7 scrub starts 2024-03-24T12:11:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[42243]: 1.7 scrub ok 2024-03-24T12:11:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[42243]: 2.2 scrub starts 2024-03-24T12:11:20.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:20 smithi064 ceph-mon[42243]: 2.2 scrub ok 2024-03-24T12:11:20.323 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 30 15 60153 60138 125.263 87.4375 0.0091167 0.00797799 2024-03-24T12:11:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:20 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T12:11:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:20 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T12:11:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:20 smithi193 ceph-mon[33593]: 1.7 scrub starts 2024-03-24T12:11:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:20 smithi193 ceph-mon[33593]: 1.7 scrub ok 2024-03-24T12:11:20.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:20 smithi193 ceph-mon[33593]: 2.2 scrub starts 2024-03-24T12:11:20.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:20 smithi193 ceph-mon[33593]: 2.2 scrub ok 2024-03-24T12:11:20.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:20.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:20.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:20.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:20.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:20.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:20.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:20.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:20.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:20.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:20.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:20.843 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:20.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:20.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:20.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:21.084 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:21.118 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:21.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:21.220 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:21.260 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:21.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:21.323 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 31 16 61559 61543 124.055 87.8125 0.00588034 0.00805604 2024-03-24T12:11:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:21 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T12:11:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:21 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T12:11:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:21 smithi193 ceph-mon[33593]: pgmap v911: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 677 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T12:11:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:21 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:11:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:21 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:11:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:21 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T12:11:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:21 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T12:11:21.346 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[30223]: pgmap v911: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 677 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[42243]: pgmap v911: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 677 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T12:11:21.347 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:11:21.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:11:21.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T12:11:21.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:21 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T12:11:21.363 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:21.398 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:21.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:21.499 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:21.535 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:21.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:21.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:21.672 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:21.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:21.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:21.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:11:21.806 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:11:21.806 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-24T12:11:21.806 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T12:11:21.806 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T12:11:21.810 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-24T12:11:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:21.812 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:21.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:21.913 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:21.952 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:21.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:22.054 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:22.092 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:22.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:22.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:11:22.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T12:11:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T12:11:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T12:11:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T12:11:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[30223]: 4.e scrub starts 2024-03-24T12:11:22.204 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[30223]: 4.e scrub ok 2024-03-24T12:11:22.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T12:11:22.204 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T12:11:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T12:11:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T12:11:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[42243]: 4.e scrub starts 2024-03-24T12:11:22.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:22 smithi064 ceph-mon[42243]: 4.e scrub ok 2024-03-24T12:11:22.227 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:22.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:22.323 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 32 16 62955 62939 122.903 87.25 0.0151654 0.00813106 2024-03-24T12:11:22.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:22 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T12:11:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:22 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T12:11:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:22 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T12:11:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:22 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T12:11:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:22 smithi193 ceph-mon[33593]: 4.e scrub starts 2024-03-24T12:11:22.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:22 smithi193 ceph-mon[33593]: 4.e scrub ok 2024-03-24T12:11:22.361 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:22.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:22.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:22.518 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:22.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:22.620 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:22.662 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:22.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:22.764 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:22.798 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:11:22.817 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:22.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:22.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:22.958 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:22.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:23.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:23.095 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:23.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:23.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:23.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:23 smithi064 ceph-mon[30223]: pgmap v912: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:11:23.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:23 smithi064 ceph-mon[42243]: pgmap v912: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:11:23.237 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:23 smithi193 ceph-mon[33593]: pgmap v912: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:11:23.242 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:23.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:23.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 33 16 64302 64286 121.73 84.1875 0.0131557 0.00820962 2024-03-24T12:11:23.343 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:23.382 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:23.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:23.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:23.520 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:23.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:23.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:23.667 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:23.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:23.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:23.818 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:23.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:23.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:23.964 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:23.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:24.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:24.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:24.112 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:24.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:24.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:24 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T12:11:24.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:24 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T12:11:24.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:24 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T12:11:24.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:24 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T12:11:24.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:24 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T12:11:24.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:24 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T12:11:24.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:24 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T12:11:24.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:24 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T12:11:24.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:24.254 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:24.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:24.323 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 34 16 65444 65428 120.249 71.375 0.0109852 0.00831144 2024-03-24T12:11:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:24 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T12:11:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:24 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T12:11:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:24 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T12:11:24.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:24 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T12:11:24.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:24.395 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:24.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:11:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:11:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:11:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:11:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200634 9.2 e296/246 9:4deab3a2:::benchmark_data_smithi064_148274_object74821:head v 296'12490, mlcod=296'12487)", 2024-03-24T12:11:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 353.847990724, 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.1099523, 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247519+0000", 2024-03-24T12:11:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247521+0000", 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8390000000000001e-06 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247528+0000", 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.374e-06 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247540+0000", 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1324e-05 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.247548+0000", 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3110000000000006e-06 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.248032+0000", 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000483993 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.248052+0000", 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0441999999999998e-05 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357431+0000", 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10937805 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357472+0000", 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0967000000000003e-05 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200636 9.0 e296/246 9:059695cb:::benchmark_data_smithi064_148274_object74823:head v 296'12488, mlcod=296'12486)", 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 353.84547144599998, 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12092386400000001, 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250038+0000", 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250040+0000", 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.250050+0000", 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7540000000000001e-06 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346892+0000", 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096842284000000001 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346904+0000", 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1429e-05 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.349515+0000", 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026114250000000001 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.349570+0000", 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5211999999999999e-05 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.370781+0000", 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021210360000000001 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.370962+0000", 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001818 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.35306.0:200640 9.6 e296/246 9:66c6e6e1:::benchmark_data_smithi064_148274_object74827:head v 296'12828, mlcod=296'12826)", 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:11:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 353.84308839900001, 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10521612800000001, 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252422+0000", 2024-03-24T12:11:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252423+0000", 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.575e-06 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.252432+0000", 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9539999999999993e-06 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346205+0000", 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093772446999999995 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.346228+0000", 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3819999999999999e-05 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.348705+0000", 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024768310000000001 2024-03-24T12:11:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.348731+0000", 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6122000000000002e-05 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357526+0000", 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087951639999999994 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:30.357638+0000", 2024-03-24T12:11:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111215 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.24379.0:3048 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94338803870976 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:06:26.485518+0000", 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 297.60999189699999, 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52480097400000003, 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.24379", 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3048 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485518+0000", 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485518+0000", 2024-03-24T12:11:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485520+0000", 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485527+0000", 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0299999999999999e-06 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485538+0000", 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1627000000000001e-05 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485549+0000", 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0404999999999999e-05 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485629+0000", 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0885999999999998e-05 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:26.485640+0000", 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0437e-05 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:27.008076+0000", 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52243590100000004 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:27.008259+0000", 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018303300000000001 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:06:27.010319+0000", 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020602189999999999 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:155011 10.5 10:a267356f:::benchmark_data_smithi064_174908_object10594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:18.647169+0000", 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 185.44834112000001, 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111205293, 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 155011 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647169+0000", 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647169+0000", 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647171+0000", 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.88e-06 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647176+0000", 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5419999999999996e-06 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647188+0000", 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2035000000000001e-05 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647196+0000", 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.391e-06 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647252+0000", 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6671e-05 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647364+0000", 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112157 2024-03-24T12:11:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.647463+0000", 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8146000000000005e-05 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.668390+0000", 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020927484 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.668405+0000", 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4596000000000001e-05 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.758044+0000", 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089639265999999995 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.758228+0000", 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018375499999999999 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.758374+0000", 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014637 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:155016 10.5 10:a7b45954:::benchmark_data_smithi064_174908_object10599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:18.649766+0000", 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 185.44574408700001, 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121993776, 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 155016 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.649766+0000", 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.649766+0000", 2024-03-24T12:11:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.649768+0000", 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1110000000000002e-06 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.649774+0000", 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1469999999999998e-06 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.649786+0000", 2024-03-24T12:11:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1994000000000001e-05 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.649795+0000", 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1859999999999998e-06 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.656122+0000", 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006326974 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.656247+0000", 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124263 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.656359+0000", 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112519 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.668411+0000", 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012051845 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.668434+0000", 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3572999999999999e-05 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.771649+0000", 2024-03-24T12:11:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10321501600000001 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.771706+0000", 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6591000000000001e-05 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.771760+0000", 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3557000000000002e-05 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:155020 10.5 10:a2223ed9:::benchmark_data_smithi064_174908_object10603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:18.652763+0000", 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 185.44274737200001, 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13839219899999999, 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 155020 2024-03-24T12:11:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.652763+0000", 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.652763+0000", 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.652765+0000", 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2809999999999998e-06 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.652772+0000", 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0249999999999997e-06 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.652786+0000", 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4134999999999999e-05 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.652797+0000", 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0535e-05 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.661527+0000", 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087303539999999992 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.661623+0000", 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5851000000000001e-05 2024-03-24T12:11:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.661700+0000", 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7070999999999994e-05 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.668447+0000", 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067474850000000001 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.668841+0000", 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000394001 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.790789+0000", 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121947574 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.790905+0000", 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115648 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.791155+0000", 2024-03-24T12:11:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025023899999999997 2024-03-24T12:11:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:155023 10.5 10:ade550c4:::benchmark_data_smithi064_174908_object10606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:18.654808+0000", 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 185.44070177699999, 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997187999999999, 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 155023 2024-03-24T12:11:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.654808+0000", 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.654808+0000", 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.654810+0000", 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2689999999999998e-06 2024-03-24T12:11:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.654816+0000", 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.823e-06 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.654830+0000", 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3947999999999999e-05 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.654838+0000", 2024-03-24T12:11:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1799999999999996e-06 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.664952+0000", 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010113733 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.665068+0000", 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116196 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.665174+0000", 2024-03-24T12:11:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106092 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.668502+0000", 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003328003 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.668952+0000", 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044962000000000002 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.794510+0000", 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12555827 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.794717+0000", 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020670399999999999 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:18.794780+0000", 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3041999999999997e-05 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.24379.0:3310 5.d 5:b4812045:::notify.4:head [watch ping cookie 94338803870976 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:35.391076+0000", 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 168.704434169, 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77058675799999998, 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:11:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.24379", 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3310 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391076+0000", 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391076+0000", 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391077+0000", 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0869999999999999e-06 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391082+0000", 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9030000000000003e-06 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391090+0000", 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391099+0000", 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6010000000000001e-06 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391255+0000", 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000155577 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391277+0000", 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1865000000000002e-05 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.161339+0000", 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.770062358 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.161546+0000", 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206982 2024-03-24T12:11:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.161663+0000", 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011647800000000001 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.24379.0:3320 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94338803870976 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:35.391556+0000", 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 168.703954157, 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.78178044499999999, 2024-03-24T12:11:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.24379", 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3099653627", 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3320 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391556+0000", 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391556+0000", 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391558+0000", 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.747e-06 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391565+0000", 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.892e-06 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391586+0000", 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0687e-05 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391596+0000", 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8030000000000008e-06 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391654+0000", 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7966e-05 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:35.391667+0000", 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3569e-05 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.161692+0000", 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77002432700000001 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.161783+0000", 2024-03-24T12:11:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0965999999999996e-05 2024-03-24T12:11:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.173336+0000", 2024-03-24T12:11:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011553488000000001 2024-03-24T12:11:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231434 10.5 10:a86799ca:::benchmark_data_smithi064_174908_object87017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.267545+0000", 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.827964929, 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14317793500000001, 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231434 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267545+0000", 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267545+0000", 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267547+0000", 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2079999999999999e-06 2024-03-24T12:11:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267553+0000", 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267569+0000", 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6018000000000001e-05 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267582+0000", 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2248e-05 2024-03-24T12:11:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267649+0000", 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7772000000000001e-05 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267793+0000", 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014411099999999999 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.267936+0000", 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014231199999999999 2024-03-24T12:11:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.272526+0000", 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045903589999999996 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273324+0000", 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079782699999999998 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410648+0000", 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13732403900000001 2024-03-24T12:11:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410687+0000", 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8523999999999999e-05 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410723+0000", 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6421000000000002e-05 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231438 10.5 10:a240586a:::benchmark_data_smithi064_174908_object87021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.273282+0000", 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.822228012, 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137586978, 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231438 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273282+0000", 2024-03-24T12:11:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273282+0000", 2024-03-24T12:11:24.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273284+0000", 2024-03-24T12:11:24.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4219999999999999e-06 2024-03-24T12:11:24.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273290+0000", 2024-03-24T12:11:24.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0480000000000004e-06 2024-03-24T12:11:24.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273302+0000", 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2041e-05 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273311+0000", 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0979999999999994e-06 2024-03-24T12:11:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273359+0000", 2024-03-24T12:11:24.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8573999999999999e-05 2024-03-24T12:11:24.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273504+0000", 2024-03-24T12:11:24.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014531400000000001 2024-03-24T12:11:24.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.273631+0000", 2024-03-24T12:11:24.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012695299999999999 2024-03-24T12:11:24.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.277785+0000", 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041535529999999999 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.277932+0000", 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014692899999999999 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410785+0000", 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13285281600000001 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410817+0000", 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2160999999999999e-05 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410869+0000", 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2068999999999999e-05 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231453 10.5 10:a6e1c401:::benchmark_data_smithi064_174908_object87036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.288884+0000", 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.806626307, 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.122163596, 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231453 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.288884+0000", 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.288884+0000", 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.288885+0000", 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7120000000000001e-06 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.288890+0000", 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.741e-06 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.288902+0000", 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1619e-05 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.288908+0000", 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.251e-06 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.288968+0000", 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.289101+0000", 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132989 2024-03-24T12:11:24.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.289235+0000", 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013366300000000001 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.293344+0000", 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041090129999999999 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.293633+0000", 2024-03-24T12:11:24.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028907200000000003 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410955+0000", 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.117321774 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.411017+0000", 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2130000000000003e-05 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.411047+0000", 2024-03-24T12:11:24.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0386999999999999e-05 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231566 10.5 10:ae3171e2:::benchmark_data_smithi064_174908_object87149:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.476300+0000", 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.619209955, 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12418311, 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231566 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476300+0000", 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476300+0000", 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476302+0000", 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.736e-06 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476306+0000", 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2320000000000003e-06 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476314+0000", 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2679999999999994e-06 2024-03-24T12:11:24.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476320+0000", 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8300000000000001e-06 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476364+0000", 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3671000000000003e-05 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476468+0000", 2024-03-24T12:11:24.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104467 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.476551+0000", 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3188000000000004e-05 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.480548+0000", 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039968460000000001 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.588239+0000", 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107691206 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.600364+0000", 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012124932 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.600429+0000", 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4412999999999995e-05 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.600483+0000", 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4320999999999997e-05 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:231577 10.5 10:a7154441:::benchmark_data_smithi064_174908_object87160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.487937+0000", 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.607573284, 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.117078089, 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 231577 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.487937+0000", 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.487937+0000", 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.487938+0000", 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.609e-06 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.487943+0000", 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3780000000000003e-06 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.487952+0000", 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1560000000000001e-06 2024-03-24T12:11:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.487958+0000", 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8760000000000002e-06 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.488000+0000", 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1931000000000002e-05 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.488079+0000", 2024-03-24T12:11:24.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9095000000000001e-05 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.488147+0000", 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8654999999999999e-05 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.491739+0000", 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035912700000000001 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.588443+0000", 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096704693999999994 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.604852+0000", 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016409136000000001 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.604954+0000", 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010136000000000001 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.605015+0000", 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0928999999999999e-05 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:255676 10.8 e377/320 10:1a45541d:::benchmark_data_smithi064_174908_object111259:head v 377'15883, mlcod=377'15879)", 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:49.461966+0000", 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.633543915999994, 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17729510500000001, 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.510 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461966+0000", 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461966+0000", 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461967+0000", 2024-03-24T12:11:24.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.421e-06 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461975+0000", 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0260000000000007e-06 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461986+0000", 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0974e-05 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.461994+0000", 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5220000000000001e-06 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.462047+0000", 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2809000000000002e-05 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.462061+0000", 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3936e-05 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.464002+0000", 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019416100000000001 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639261+0000", 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17525880699999999 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:255681 10.8 e377/320 10:1d81bdc4:::benchmark_data_smithi064_174908_object111264:head v 377'15884, mlcod=377'15881)", 2024-03-24T12:11:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:49.464251+0000", 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.631259404000005, 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18867756099999999, 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.464251+0000", 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.464251+0000", 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.464252+0000", 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2950000000000001e-06 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.464258+0000", 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7640000000000002e-06 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639268+0000", 2024-03-24T12:11:24.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17501061500000001 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639277+0000", 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6600000000000001e-06 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639425+0000", 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014830900000000001 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639448+0000", 2024-03-24T12:11:24.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2767000000000001e-05 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.652894+0000", 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013446125999999999 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.652928+0000", 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4025e-05 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:255745 10.5 10:a2144aa7:::benchmark_data_smithi064_174908_object111328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:49.517536+0000", 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.577973612999997, 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.133653472, 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 255745 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.517536+0000", 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.517536+0000", 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.517540+0000", 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4010000000000001e-06 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.517548+0000", 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5760000000000006e-06 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639231+0000", 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121682657 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639248+0000", 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7521999999999999e-05 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639311+0000", 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2162000000000006e-05 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639446+0000", 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013569 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.639573+0000", 2024-03-24T12:11:24.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012712499999999999 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.649608+0000", 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01003427 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.650460+0000", 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085222100000000001 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.651069+0000", 2024-03-24T12:11:24.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060920899999999999 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.651136+0000", 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7331999999999998e-05 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:49.651190+0000", 2024-03-24T12:11:24.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3307000000000003e-05 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:257297 10.5 10:adb92daf:::benchmark_data_smithi064_174908_object112880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:50.794743+0000", 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.300766952999993, 2024-03-24T12:11:24.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091245701999999998, 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 257297 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.794743+0000", 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.794743+0000", 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.794745+0000", 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1720000000000001e-06 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.794751+0000", 2024-03-24T12:11:24.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2079999999999997e-06 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.794763+0000", 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2118e-05 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.794770+0000", 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4729999999999997e-06 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.794826+0000", 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6067999999999999e-05 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.794968+0000", 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014164200000000001 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.795093+0000", 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012485099999999999 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.803904+0000", 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088115580000000006 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.803917+0000", 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.885881+0000", 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081963379000000003 2024-03-24T12:11:24.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.885934+0000", 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2975e-05 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.885989+0000", 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4858999999999998e-05 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.531 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:257299 10.5 10:a0da3e77:::benchmark_data_smithi064_174908_object112882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:50.795263+0000", 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.300246555000001, 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091032518000000007, 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 257299 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:11:24.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.795263+0000", 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.795263+0000", 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.795265+0000", 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.438e-06 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:11:24.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.795270+0000", 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7650000000000001e-06 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.795281+0000", 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1826e-05 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.795290+0000", 2024-03-24T12:11:24.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1110000000000008e-06 2024-03-24T12:11:24.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:11:24.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.799983+0000", 2024-03-24T12:11:24.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004693086 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.800073+0000", 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0242999999999996e-05 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.800148+0000", 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5030000000000005e-05 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.804035+0000", 2024-03-24T12:11:24.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038872669999999998 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.804254+0000", 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021876 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.886185+0000", 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081930893000000005 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.886255+0000", 2024-03-24T12:11:24.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9889999999999997e-05 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:50.886296+0000", 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1208999999999998e-05 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:24.538 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:11:24.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:24.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:24.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:24.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:24.720 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:24.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:24.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:24.857 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:24.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:24.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:24.982 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:24.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:25.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:25.118 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:11:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:25.132 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:25.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:25.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:25.274 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:25.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:25.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 35 16 66601 66585 118.879 72.3125 0.0128619 0.00840625 2024-03-24T12:11:25.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:25.416 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:25.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:25.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T12:11:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T12:11:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: pgmap v913: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T12:11:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T12:11:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T12:11:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3518077923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: from='client.25879 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: pgmap v913: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T12:11:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T12:11:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:11:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:11:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3518077923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: from='client.25879 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:25 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:11:25.556 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:25.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: pgmap v913: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3518077923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: from='client.25879 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:25 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:11:25.659 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:25.687 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:25.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:25.789 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:25.829 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:25.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:25.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:25.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:25.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:26.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:26.079 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:11:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T12:11:26.080 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:11:26.106 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:26.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:26.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:26.244 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:26.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:26.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 36 16 67789 67773 117.639 74.25 0.0191143 0.00849474 2024-03-24T12:11:26.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:26 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:11:26.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:26 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:11:26.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:26 smithi193 ceph-mon[33593]: from='client.25879 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:11:26.338 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:26 smithi193 ceph-mon[33593]: osdmap e438: 8 total, 8 up, 8 in 2024-03-24T12:11:26.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:26.394 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:26.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:26.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:26.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:26 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:11:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:26 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:11:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:26 smithi064 ceph-mon[30223]: from='client.25879 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:11:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:26 smithi064 ceph-mon[30223]: osdmap e438: 8 total, 8 up, 8 in 2024-03-24T12:11:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:26 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:11:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:26 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:11:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:26 smithi064 ceph-mon[42243]: from='client.25879 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:11:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:26 smithi064 ceph-mon[42243]: osdmap e438: 8 total, 8 up, 8 in 2024-03-24T12:11:26.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:26.682 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:26.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:26.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:26.819 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:26.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:26.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:26.959 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:26.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:27.061 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:27.100 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:27.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:27.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:27.236 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:27.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:27.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 37 16 69086 69070 116.65 81.0625 0.00961156 0.00856777 2024-03-24T12:11:27.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:27.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:27.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[30223]: pgmap v915: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T12:11:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:11:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:11:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[30223]: osdmap e439: 8 total, 8 up, 8 in 2024-03-24T12:11:27.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[42243]: pgmap v915: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T12:11:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:11:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:11:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-mon[42243]: osdmap e439: 8 total, 8 up, 8 in 2024-03-24T12:11:27.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:27.505 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:27.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:27 smithi193 ceph-mon[33593]: pgmap v915: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T12:11:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:11:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:11:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:27 smithi193 ceph-mon[33593]: osdmap e439: 8 total, 8 up, 8 in 2024-03-24T12:11:27.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:27.650 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:27.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:27.670 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:11:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:11:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:27.751 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:27.781 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:27.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:27.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:27.911 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:27.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:28.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:28.036 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:28.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:28.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:28.175 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:28.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:28.276 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:28.308 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:11:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:11:28.311 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:28.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:28.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 38 16 70182 70166 115.383 68.5 0.00876015 0.00866166 2024-03-24T12:11:28.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:28.450 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:28.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:28.551 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:28.576 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:28.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:28.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 6.0 scrub starts 2024-03-24T12:11:28.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 6.0 scrub ok 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: pgmap v917: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:28.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:28 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T12:11:28.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:28.715 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:28.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 6.0 scrub starts 2024-03-24T12:11:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 6.0 scrub ok 2024-03-24T12:11:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:11:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T12:11:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: pgmap v917: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 6.0 scrub starts 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 6.0 scrub ok 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: pgmap v917: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:28 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T12:11:28.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:28.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:28.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:28.995 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:28.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:29.097 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:29.132 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:29.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:29.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:29.267 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:29.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:29.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 39 16 71527 71511 114.58 84.0625 0.0119747 0.00872151 2024-03-24T12:11:29.369 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:29.404 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:29.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:29.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:29.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:29 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T12:11:29.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:29 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T12:11:29.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:29 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T12:11:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:29.705 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:29.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:29 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T12:11:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:29 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T12:11:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:29 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T12:11:29.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:29 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T12:11:29.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:29 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T12:11:29.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:29 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T12:11:29.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:29.843 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:29.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:29.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:29.977 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:29.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:30.079 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:30.112 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:30.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:30.213 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:30.251 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:30.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:30.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:11:30.326212+0000 min lat: 0.00124462 max lat: 0.182416 avg lat: 0.00878526 2024-03-24T12:11:30.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:11:30.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 40 16 72835 72819 113.759 81.75 0.0191089 0.00878526 2024-03-24T12:11:30.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:30.391 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:30.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:30.492 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:30.519 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:30.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:30 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T12:11:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:30 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T12:11:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:30 smithi193 ceph-mon[33593]: pgmap v918: 158 pgs: 158 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-24T12:11:30.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:30.660 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:30.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:30.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:30.762 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:30.796 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:30.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:30 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T12:11:30.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:30 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T12:11:30.797 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:30 smithi064 ceph-mon[30223]: pgmap v918: 158 pgs: 158 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-24T12:11:30.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:30 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T12:11:30.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:30 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T12:11:30.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:30 smithi064 ceph-mon[42243]: pgmap v918: 158 pgs: 158 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-24T12:11:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:30.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:30.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:30.946 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:30.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:31.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:31.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:31.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:31.230 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:31.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:31.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 41 16 74169 74153 113.018 83.375 0.00988785 0.00884266 2024-03-24T12:11:31.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:31.368 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:31.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:31.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:31.508 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:31.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:31 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T12:11:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:31 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T12:11:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:31 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:11:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:31 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:11:31.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:31.648 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:31.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:31.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:31.791 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:31.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:31.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:31.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:31 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T12:11:31.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:31 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T12:11:31.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:31 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:11:31.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:31 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:11:31.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:31 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T12:11:31.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:31 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T12:11:31.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:31 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:11:31.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:31 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:11:31.893 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:31.899 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:11:31.899 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:11:31.899 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-24T12:11:31.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:11:31.927 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:31.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:32.029 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:32.067 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:32.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:32.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:11:32.159 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:32.168 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:32.199 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:32.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:32.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:32.301 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:32.324 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 42 16 75356 75340 112.093 74.1875 0.00984561 0.00891523 2024-03-24T12:11:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:32.338 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:32.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:32.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:32.474 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:32.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:32.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:32 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:11:32.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:32 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:11:32.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:32 smithi064 ceph-mon[30223]: pgmap v919: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:11:32.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:32 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:11:32.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:32 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:11:32.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:32 smithi064 ceph-mon[42243]: pgmap v919: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:11:32.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:32 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:11:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:32 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:11:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:32 smithi193 ceph-mon[33593]: pgmap v919: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:11:32.627 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:32.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:32.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:32.781 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:32.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:32.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:32.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:11:32.916 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:32.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:33.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:33.049 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:33.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:33.118 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:33.150 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:33.178 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:33.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:33.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:33.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:33.319 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:33.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:33.326 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 43 16 76521 76505 111.179 72.8125 0.0153558 0.00898904 2024-03-24T12:11:33.420 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:33.455 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:33.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:33.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:33.599 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:33.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:33.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:33.740 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:33.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 6.10 scrub starts 2024-03-24T12:11:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 6.10 scrub ok 2024-03-24T12:11:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T12:11:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 2.12 scrub starts 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 2.12 scrub ok 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 5.1 scrub starts 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[30223]: 5.1 scrub ok 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 6.10 scrub starts 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 6.10 scrub ok 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T12:11:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T12:11:33.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 2.12 scrub starts 2024-03-24T12:11:33.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 2.12 scrub ok 2024-03-24T12:11:33.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 5.1 scrub starts 2024-03-24T12:11:33.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:33 smithi064 ceph-mon[42243]: 5.1 scrub ok 2024-03-24T12:11:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 6.10 scrub starts 2024-03-24T12:11:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 6.10 scrub ok 2024-03-24T12:11:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T12:11:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T12:11:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T12:11:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T12:11:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 2.12 scrub starts 2024-03-24T12:11:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 2.12 scrub ok 2024-03-24T12:11:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 5.1 scrub starts 2024-03-24T12:11:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:33 smithi193 ceph-mon[33593]: 5.1 scrub ok 2024-03-24T12:11:33.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:33.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:33.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:34.031 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:34.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:34.132 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:34.170 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:34.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:34.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:34.309 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:34.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:34.325 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 44 16 77821 77805 110.498 81.25 0.00675062 0.00904505 2024-03-24T12:11:34.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:34.454 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:34 smithi064 ceph-mon[42243]: pgmap v920: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:34.454 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:34 smithi064 ceph-mon[30223]: pgmap v920: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:34.456 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:34.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:34.557 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:34.593 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:34.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:34.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:34.729 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:34.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:34.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:34.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:34 smithi193 ceph-mon[33593]: pgmap v920: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:34.870 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:34.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:34.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:35.008 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:35.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:35.110 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:35.141 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:35.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:35.242 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:35.274 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:35.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:35.325 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 45 16 79145 79129 109.882 82.75 0.0182359 0.00909565 2024-03-24T12:11:35.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:35.414 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:35.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:35.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:35 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:11:35.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:35 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:11:35.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:35 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T12:11:35.432 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:35 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T12:11:35.433 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:35 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:11:35.433 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:35 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:11:35.433 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:35 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T12:11:35.433 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:35 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T12:11:35.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:35.555 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:35.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:35.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:35.689 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:35.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:35.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:35.831 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:35.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:35 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:11:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:35 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:11:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:35 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T12:11:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:35 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T12:11:35.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:35.972 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:35.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:36.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:36.105 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:36.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:36.207 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:36.239 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:36.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:36.326 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 46 16 80438 80422 109.25 80.8125 0.0103205 0.00914833 2024-03-24T12:11:36.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:36.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:36.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:36.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:36.517 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:36.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:36.618 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:36.654 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:36.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:36.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:36.794 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:36.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:36.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T12:11:36.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T12:11:36.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[30223]: 6.7 scrub starts 2024-03-24T12:11:36.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[30223]: 6.7 scrub ok 2024-03-24T12:11:36.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[30223]: pgmap v921: 158 pgs: 158 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T12:11:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T12:11:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[42243]: 6.7 scrub starts 2024-03-24T12:11:36.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[42243]: 6.7 scrub ok 2024-03-24T12:11:36.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:36 smithi064 ceph-mon[42243]: pgmap v921: 158 pgs: 158 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:36 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T12:11:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:36 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T12:11:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:36 smithi193 ceph-mon[33593]: 6.7 scrub starts 2024-03-24T12:11:36.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:36 smithi193 ceph-mon[33593]: 6.7 scrub ok 2024-03-24T12:11:36.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:36 smithi193 ceph-mon[33593]: pgmap v921: 158 pgs: 158 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:36.896 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:36.932 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:36.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:37.034 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:37.068 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:37.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:37.169 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:37.193 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:37.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:37.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:37.325 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 47 16 81691 81675 108.591 78.3125 0.00849845 0.00920407 2024-03-24T12:11:37.325 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:37.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:37.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:37.465 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:37.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:37.567 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:37.603 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:37.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:37.705 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:37.734 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:37.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:37.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:37 smithi064 ceph-mon[30223]: 4.1b scrub starts 2024-03-24T12:11:37.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:37 smithi064 ceph-mon[30223]: 4.1b scrub ok 2024-03-24T12:11:37.750 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:11:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:11:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:37.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:37 smithi064 ceph-mon[42243]: 4.1b scrub starts 2024-03-24T12:11:37.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:37 smithi064 ceph-mon[42243]: 4.1b scrub ok 2024-03-24T12:11:37.835 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:37 smithi193 ceph-mon[33593]: 4.1b scrub starts 2024-03-24T12:11:37.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:37 smithi193 ceph-mon[33593]: 4.1b scrub ok 2024-03-24T12:11:37.865 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:37.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:37.967 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:37.998 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:37.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:38.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:38.138 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:38.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:38.240 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:38.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:38.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:38.327 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 48 16 82962 82946 107.979 79.4375 0.00937118 0.00925635 2024-03-24T12:11:38.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:11:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:11:38.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:38.427 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:38.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:38.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[30223]: 4.3 deep-scrub starts 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[30223]: 4.3 deep-scrub ok 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[30223]: pgmap v922: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 969 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[42243]: 4.3 deep-scrub starts 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[42243]: 4.3 deep-scrub ok 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[42243]: pgmap v922: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 969 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T12:11:38.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:38.528 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:38.566 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:38.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:38.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:38.695 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:38.717 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:38.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:38.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:38.823 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:11:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:38 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T12:11:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:38 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T12:11:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:38 smithi193 ceph-mon[33593]: 4.3 deep-scrub starts 2024-03-24T12:11:38.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:38 smithi193 ceph-mon[33593]: 4.3 deep-scrub ok 2024-03-24T12:11:38.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:38 smithi193 ceph-mon[33593]: pgmap v922: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 969 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T12:11:38.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:38.870 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:11:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:38.873 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:38.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:38.913 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:11:38.953 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:11:38.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:38.995 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:11:39.028 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:39.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:39.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:39.035 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:11:39.081 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:11:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:11:39.130 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:39.145 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:11:39.145 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:11:39.145 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:11:39.146 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:11:39.146 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:11:39.146 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:11:39.146 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:11:39.146 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:11:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:39.161 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:39.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:39.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:39.294 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:39.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:39.328 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 49 15 84094 84079 107.221 70.8125 0.0165704 0.009322 2024-03-24T12:11:39.396 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:39.433 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:39.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:39.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:39.567 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:39.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:39.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:39.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T12:11:39.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T12:11:39.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T12:11:39.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T12:11:39.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T12:11:39.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T12:11:39.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/759730971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T12:11:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T12:11:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T12:11:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T12:11:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T12:11:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T12:11:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:39 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/759730971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:39.673 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:11:39.673 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:11:39.674 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 439, 2024-03-24T12:11:39.674 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "06f23456-e9d4-11ee-95cd-87774f69a715", 2024-03-24T12:11:39.674 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-03-24T11:46:40.657065+0000", 2024-03-24T12:11:39.674 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-03-24T12:11:26.970481+0000", 2024-03-24T12:11:39.674 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-03-24T12:08:35.127870+0000", 2024-03-24T12:11:39.674 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-03-24T11:51:43.667453+0000", 2024-03-24T12:11:39.674 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:11:39.674 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5799936, 2024-03-24T12:11:39.675 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-03-24T12:11:39.675 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-03-24T12:11:39.675 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 19, 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 11, 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-03-24T12:11:39.676 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:49:56.369674+0000", 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:11:39.677 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 11, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 11, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 11, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 11, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 11, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "1.b", 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 426, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 425, 2024-03-24T12:11:39.678 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 425, 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "83'130", 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "83'130" 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "426", 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "426", 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "424", 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:11:39.679 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:11:39.680 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.681 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:11:39.682 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.1800000667572021, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.1800000667572021, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:11:39.683 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".rgw.root", 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:13.871974+0000", 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:11:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:11:39.685 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "68", 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:11:39.686 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:11:39.687 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:11:39.688 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.5, 2024-03-24T12:11:39.689 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.5, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.5, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.5, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "datapool", 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:14.117018+0000", 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-03-24T12:11:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 3, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 3, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 3, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 3, 2024-03-24T12:11:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 3, 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "71", 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:11:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 2, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 71, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:11:39.693 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:11:39.694 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.6700000762939453, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.6700000762939453, 2024-03-24T12:11:39.695 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-03-24T12:11:39.696 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T12:11:39.696 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T12:11:39.696 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:11:39.696 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:11:39.696 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:11:39.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 4, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:15.453744+0000", 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:11:39.697 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:11:39.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "70", 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:11:39.699 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:11:39.700 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:11:39.701 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.25, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.25, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.25, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.25, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:11:39.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 5, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:17.611004+0000", 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:11:39.703 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:11:39.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "72", 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:11:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:11:39.706 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:11:39.707 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.75, 2024-03-24T12:11:39.708 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.75, 2024-03-24T12:11:39.709 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:11:39.709 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-03-24T12:11:39.709 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-03-24T12:11:39.709 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:11:39.709 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 6, 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T11:52:19.701416+0000", 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:11:39.710 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:11:39.711 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:11:39.711 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:11:39.711 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:11:39.711 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:11:39.711 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:11:39.711 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:11:39.711 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:11:39.711 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "74", 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:11:39.712 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:11:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:11:39.714 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_bias": 4 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:11:39.715 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": {} 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2, 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2, 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 11, 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "unique_pool_4", 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-03-24T12:10:41.539113+0000", 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-03-24T12:11:39.717 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 16, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 16, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 16, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 16, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 16, 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-03-24T12:11:39.718 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "424", 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:11:39.719 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:11:39.720 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:11:39.721 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:11:39.721 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-03-24T12:11:39.721 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-03-24T12:11:39.721 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-03-24T12:11:39.721 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-03-24T12:11:39.721 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-03-24T12:11:39.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.721 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-03-24T12:11:39.722 INFO:teuthology.orchestra.run.smithi064.stdout: "rados": {} 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3, 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3, 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 3, 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 3, 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89453697204589844, 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:11:39.723 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "bc14a64f-9e6c-41f1-901e-c49fabffb53d", 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:11:39.724 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 8, 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-03-24T12:11:39.725 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:11:39.726 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:11:39.727 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675992571 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/2675992571", 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6802/2675992571", 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6804/2675992571", 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6803/2675992571", 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:11:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "74587332-f0b2-4905-9888-7bdb604cc0e9", 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.2537689208984375, 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:11:39.729 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 385, 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.730 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.732 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-03-24T12:11:39.732 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:11:39.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.732 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.732 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 841667271 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6805/841667271", 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6806/841667271", 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6808/841667271", 2024-03-24T12:11:39.733 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6807/841667271", 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a0c0ed85-9f33-4198-a1f2-447eb17c364b", 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:11:39.734 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.3165130615234375, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.90252685546875, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 18, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.735 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:11:39.737 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 435147390 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/435147390", 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6810/435147390", 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6812/435147390", 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6811/435147390", 2024-03-24T12:11:39.738 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1396f66c-3e0c-4a01-b0a7-6f763989a479", 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:11:39.739 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 27, 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 423, 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.742 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-03-24T12:11:39.742 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:11:39.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.742 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:11:39.742 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3355318038 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6813/3355318038", 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6814/3355318038", 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6816/3355318038", 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6815/3355318038", 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:11:39.743 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50d65d8b-3e90-46a4-b19d-50c3015af40b", 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 33, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 436, 2024-03-24T12:11:39.744 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6800", 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.745 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6801", 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6803", 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.746 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6802", 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3851339176 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6800/3851339176", 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6801/3851339176", 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3851339176", 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3851339176", 2024-03-24T12:11:39.747 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "50c5d50e-7e68-487c-9f68-1abb8c927ff6", 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 41, 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 317, 2024-03-24T12:11:39.748 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 353, 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 318, 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6804", 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6805", 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:39.752 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:39.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6807", 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6806", 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1531860860 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6804/1531860860", 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6805/1531860860", 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6807/1531860860", 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6806/1531860860", 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7e74fe39-5707-4c90-ad38-1083cd7a6996", 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 51, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 426, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:11:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6808", 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6809", 2024-03-24T12:11:39.756 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6811", 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:11:39.757 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6810", 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2408237278 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6808/2408237278", 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6809/2408237278", 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6811/2408237278", 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6810/2408237278", 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:11:39.758 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e568df64-cc9a-43f8-a432-8628d1da1015", 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 61, 2024-03-24T12:11:39.759 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 417, 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6812", 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-03-24T12:11:39.760 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6813", 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6815", 2024-03-24T12:11:39.761 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.193:6814", 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2649989609 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.762 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.193:6812/2649989609", 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.193:6813/2649989609", 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.193:6815/2649989609", 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.193:6814/2649989609", 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.763 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:11:02.451289+0000", 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:11:39.764 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:11:15.630952+0000", 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:10:59.935657+0000", 2024-03-24T12:11:39.765 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:10:34.546015+0000", 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.766 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:10:41.332483+0000", 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-03-24T12:11:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:10:45.871448+0000", 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 317 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-03-24T12:11:39.768 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:10:57.475453+0000", 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-03-24T12:11:39.769 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-03-24T12:10:54.455564+0000", 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.0", 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [ 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.b", 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.4", 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 5 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.9", 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.772 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.1d", 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.6", 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 7, 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.13", 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.18", 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 4, 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 3 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.775 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.19", 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 1, 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 6 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "5.1c", 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: "mappings": [ 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:11:39.776 INFO:teuthology.orchestra.run.smithi064.stdout: "from": 3, 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "to": 2 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": { 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/3538503390": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1277626303": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3407451558": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3753895773": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:11:39.777 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/842059688": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1610565583": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/2730170969": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3257265142": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/1459540658": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/629947599": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/520335082": "2024-03-25T11:47:08.661692+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/3351953651": "2024-03-25T11:52:55.117233+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:6800/4027721305": "2024-03-25T11:47:36.035658+0000", 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "172.21.15.64:0/4028269665": "2024-03-25T11:47:36.035658+0000" 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-03-24T12:11:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-03-24T12:11:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:11:39.780 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:11:39.780 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-03-24T12:11:39.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:11:39.780 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:11:39.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:39 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T12:11:39.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:39 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T12:11:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:39 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T12:11:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:39 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T12:11:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:39 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T12:11:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:39 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T12:11:39.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:39 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/759730971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:39.853 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:39.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:39.888 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:39.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:39.989 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:40.024 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:40.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:40.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:40.167 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:40.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:40.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:40.307 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:40.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:40.327 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 50 16 85325 85309 106.614 76.875 0.0154934 0.00937481 2024-03-24T12:11:40.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:40.454 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:40.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:40.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:40.594 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:40.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:40.696 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:40.732 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:40.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:40.771 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:11:40.771 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:11:40.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:40 smithi064 ceph-mon[30223]: pgmap v923: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:11:40.772 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:40 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3024704332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:11:40.772 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:40 smithi064 ceph-mon[42243]: pgmap v923: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:11:40.772 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:40 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3024704332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:11:40.833 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:40.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:40 smithi193 ceph-mon[33593]: pgmap v923: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:11:40.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:40 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3024704332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:11:40.869 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:40.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:40.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:41.013 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:41.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:41.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:41.153 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:41.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:41.255 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:41.290 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:41.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:41.330 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 51 16 86625 86609 106.116 81.25 0.00833608 0.00941878 2024-03-24T12:11:41.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:41.432 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:41.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:41.534 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:41.572 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:41.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:41.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:11:41.663 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:41 smithi064 ceph-mon[30223]: 6.4 deep-scrub starts 2024-03-24T12:11:41.663 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:41 smithi064 ceph-mon[30223]: 6.4 deep-scrub ok 2024-03-24T12:11:41.663 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:41 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T12:11:41.664 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:41 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T12:11:41.664 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:41 smithi064 ceph-mon[42243]: 6.4 deep-scrub starts 2024-03-24T12:11:41.664 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:41 smithi064 ceph-mon[42243]: 6.4 deep-scrub ok 2024-03-24T12:11:41.664 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:41 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T12:11:41.665 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:41 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T12:11:41.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:41.717 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:41.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:41.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:41.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:41 smithi193 ceph-mon[33593]: 6.4 deep-scrub starts 2024-03-24T12:11:41.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:41 smithi193 ceph-mon[33593]: 6.4 deep-scrub ok 2024-03-24T12:11:41.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:41 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T12:11:41.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:41 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T12:11:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:41.856 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:41.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:41.956 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:41.988 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:41.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:41.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:42.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:42.141 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:42.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:42.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:42.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:42.328 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 52 16 87861 87845 105.561 77.25 0.00913637 0.00946814 2024-03-24T12:11:42.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:42.416 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:42.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:42.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:42.558 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:42.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:42.660 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:42.694 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:42.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:42.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:42.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:42.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[30223]: 4.1d scrub starts 2024-03-24T12:11:42.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[30223]: 4.1d scrub ok 2024-03-24T12:11:42.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T12:11:42.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[30223]: pgmap v924: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[42243]: 4.1d scrub starts 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[42243]: 4.1d scrub ok 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[42243]: pgmap v924: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:42 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:42.833 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:42.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:42.839 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:42 smithi193 ceph-mon[33593]: 4.1d scrub starts 2024-03-24T12:11:42.839 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:42 smithi193 ceph-mon[33593]: 4.1d scrub ok 2024-03-24T12:11:42.839 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:42 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T12:11:42.839 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:42 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T12:11:42.839 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:42 smithi193 ceph-mon[33593]: pgmap v924: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:42.839 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:42 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:42.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:42.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:43.078 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:43.111 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:43.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:43.114 INFO:tasks.thrashosds.thrasher:j is {'epoch': 439, 'fsid': '06f23456-e9d4-11ee-95cd-87774f69a715', 'created': '2024-03-24T11:46:40.657065+0000', 'modified': '2024-03-24T12:11:26.970481+0000', 'last_up_change': '2024-03-24T12:08:35.127870+0000', 'last_in_change': '2024-03-24T11:51:43.667453+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:49:56.369674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 426, 'last_epoch_started': 425, 'last_epoch_clean': 425, 'source_version': "83'130", 'target_version': "83'130"}, 'last_change': '426', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '426', 'last_force_op_resend_preluminous': '424', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T11:52:13.871974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-24T11:52:14.117018+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T11:52:15.453744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T11:52:17.611004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T11:52:19.701416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-24T12:10:41.539113+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '424', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8945369720458984, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'bc14a64f-9e6c-41f1-901e-c49fabffb53d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 2675992571}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6802', 'nonce': 2675992571}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6804', 'nonce': 2675992571}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 2675992571}]}, 'public_addr': '172.21.15.64:6801/2675992571', 'cluster_addr': '172.21.15.64:6802/2675992571', 'heartbeat_back_addr': '172.21.15.64:6804/2675992571', 'heartbeat_front_addr': '172.21.15.64:6803/2675992571', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '74587332-f0b2-4905-9888-7bdb604cc0e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2537689208984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 841667271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6806', 'nonce': 841667271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6808', 'nonce': 841667271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 841667271}]}, 'public_addr': '172.21.15.64:6805/841667271', 'cluster_addr': '172.21.15.64:6806/841667271', 'heartbeat_back_addr': '172.21.15.64:6808/841667271', 'heartbeat_front_addr': '172.21.15.64:6807/841667271', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a0c0ed85-9f33-4198-a1f2-447eb17c364b', 'up': 1, 'in': 1, 'weight': 0.3165130615234375, 'primary_affinity': 0.90252685546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 435147390}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6810', 'nonce': 435147390}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6812', 'nonce': 435147390}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 435147390}]}, 'public_addr': '172.21.15.64:6809/435147390', 'cluster_addr': '172.21.15.64:6810/435147390', 'heartbeat_back_addr': '172.21.15.64:6812/435147390', 'heartbeat_front_addr': '172.21.15.64:6811/435147390', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1396f66c-3e0c-4a01-b0a7-6f763989a479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 423, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3355318038}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6814', 'nonce': 3355318038}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6816', 'nonce': 3355318038}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3355318038}]}, 'public_addr': '172.21.15.64:6813/3355318038', 'cluster_addr': '172.21.15.64:6814/3355318038', 'heartbeat_back_addr': '172.21.15.64:6816/3355318038', 'heartbeat_front_addr': '172.21.15.64:6815/3355318038', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50d65d8b-3e90-46a4-b19d-50c3015af40b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 436, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3851339176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3851339176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3851339176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3851339176}]}, 'public_addr': '172.21.15.193:6800/3851339176', 'cluster_addr': '172.21.15.193:6801/3851339176', 'heartbeat_back_addr': '172.21.15.193:6803/3851339176', 'heartbeat_front_addr': '172.21.15.193:6802/3851339176', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '50c5d50e-7e68-487c-9f68-1abb8c927ff6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 317, 'up_from': 353, 'up_thru': 417, 'down_at': 318, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 1531860860}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 1531860860}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 1531860860}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 1531860860}]}, 'public_addr': '172.21.15.193:6804/1531860860', 'cluster_addr': '172.21.15.193:6805/1531860860', 'heartbeat_back_addr': '172.21.15.193:6807/1531860860', 'heartbeat_front_addr': '172.21.15.193:6806/1531860860', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '7e74fe39-5707-4c90-ad38-1083cd7a6996', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 426, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 2408237278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 2408237278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 2408237278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 2408237278}]}, 'public_addr': '172.21.15.193:6808/2408237278', 'cluster_addr': '172.21.15.193:6809/2408237278', 'heartbeat_back_addr': '172.21.15.193:6811/2408237278', 'heartbeat_front_addr': '172.21.15.193:6810/2408237278', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e568df64-cc9a-43f8-a432-8628d1da1015', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 2649989609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 2649989609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 2649989609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 2649989609}]}, 'public_addr': '172.21.15.193:6812/2649989609', 'cluster_addr': '172.21.15.193:6813/2649989609', 'heartbeat_back_addr': '172.21.15.193:6815/2649989609', 'heartbeat_front_addr': '172.21.15.193:6814/2649989609', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:11:02.451289+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:11:15.630952+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:10:59.935657+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:10:34.546015+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:10:41.332483+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:10:45.871448+0000', 'dead_epoch': 317}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:10:57.475453+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:10:54.455564+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [0, 1, 7]}], 'pg_upmap_items': [{'pgid': '2.b', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.6', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.1c', 'mappings': [{'from': 3, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.64:6800/3538503390': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/1277626303': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3407451558': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/3753895773': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/842059688': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/1610565583': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/2730170969': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3257265142': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/1459540658': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:0/629947599': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:6800/520335082': '2024-03-25T11:47:08.661692+0000', '172.21.15.64:0/3351953651': '2024-03-25T11:52:55.117233+0000', '172.21.15.64:6800/4027721305': '2024-03-25T11:47:36.035658+0000', '172.21.15.64:0/4028269665': '2024-03-25T11:47:36.035658+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:11:43.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:11:43.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:43.268 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:43.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:43.330 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 53 16 89156 89140 105.092 80.9375 0.0112047 0.00950982 2024-03-24T12:11:43.370 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:43.400 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:43.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:43.502 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:43.531 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:43.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:43.633 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:43.671 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:43.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:43.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:43 smithi064 ceph-mon[30223]: 1.9 scrub starts 2024-03-24T12:11:43.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:43 smithi064 ceph-mon[30223]: 1.9 scrub ok 2024-03-24T12:11:43.687 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:43 smithi064 ceph-mon[42243]: 1.9 scrub starts 2024-03-24T12:11:43.687 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:43 smithi064 ceph-mon[42243]: 1.9 scrub ok 2024-03-24T12:11:43.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:43.808 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:43.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:43.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:43 smithi193 ceph-mon[33593]: 1.9 scrub starts 2024-03-24T12:11:43.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:43 smithi193 ceph-mon[33593]: 1.9 scrub ok 2024-03-24T12:11:43.910 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:43.945 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:43.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:44.040 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:44.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:44.074 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:44.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:44.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:44.204 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:44.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:44.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:44.330 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 54 16 90358 90342 104.537 75.125 0.00668028 0.00955963 2024-03-24T12:11:44.333 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:44.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:44.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:44.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:44.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:44.625 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:44.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:44.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:44 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T12:11:44.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:44 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T12:11:44.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:44 smithi064 ceph-mon[30223]: pgmap v925: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T12:11:44.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:44 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T12:11:44.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:44 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T12:11:44.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:44 smithi064 ceph-mon[42243]: pgmap v925: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T12:11:44.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:44.765 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:44.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:44 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T12:11:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:44 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T12:11:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:44 smithi193 ceph-mon[33593]: pgmap v925: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T12:11:44.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:44.918 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:44.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:45.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:45.052 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:45.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:45.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:45.190 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:45.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:45.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:45.329 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:45.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:45.330 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 55 16 91593 91577 104.04 77.1875 0.00611905 0.00960675 2024-03-24T12:11:45.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:45.453 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:11:45.474 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:45.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:45.576 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:45.613 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:45.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:45.714 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:45.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:11:45.757 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:45.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:45.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:45.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[30223]: 5.c scrub starts 2024-03-24T12:11:45.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[30223]: 5.c scrub ok 2024-03-24T12:11:45.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:11:45.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1076585448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[30223]: from='client.35909 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[42243]: 5.c scrub starts 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[42243]: 5.c scrub ok 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T12:11:45.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1076585448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:11:45.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:45 smithi064 ceph-mon[42243]: from='client.35909 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:11:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:45 smithi193 ceph-mon[33593]: 5.c scrub starts 2024-03-24T12:11:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:45 smithi193 ceph-mon[33593]: 5.c scrub ok 2024-03-24T12:11:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:45 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:11:45.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:45 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:11:45.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:45 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T12:11:45.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:45 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T12:11:45.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:45 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1076585448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:11:45.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:45 smithi193 ceph-mon[33593]: from='client.35909 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:11:45.859 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:45.897 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:45.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:45.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:46.040 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:46.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:46.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:46.142 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:46.209 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:46.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:46.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:46.330 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 56 16 92795 92779 103.524 75.125 0.0216195 0.00965526 2024-03-24T12:11:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:46.353 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:46.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:46.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T12:11:46.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:46.498 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:46.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:46.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:46.650 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:46.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:46.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T12:11:46.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T12:11:46.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T12:11:46.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T12:11:46.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 1.6 deep-scrub starts 2024-03-24T12:11:46.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 1.6 deep-scrub ok 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: from='client.35909 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: osdmap e440: 8 total, 8 up, 8 in 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[30223]: pgmap v927: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T12:11:46.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 1.6 deep-scrub starts 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 1.6 deep-scrub ok 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: from='client.35909 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: osdmap e440: 8 total, 8 up, 8 in 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T12:11:46.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:46 smithi064 ceph-mon[42243]: pgmap v927: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:46.752 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:46.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:46.785 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:46.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T12:11:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T12:11:46.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 1.6 deep-scrub starts 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 1.6 deep-scrub ok 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: from='client.35909 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: osdmap e440: 8 total, 8 up, 8 in 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T12:11:46.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:46 smithi193 ceph-mon[33593]: pgmap v927: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:46.887 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:46.918 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:46.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:47.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:47.050 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:47.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:47.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:47.196 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:47.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:11:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:11:47.298 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:47.330 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:47.330 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 57 16 93984 93968 103.011 74.3125 0.00905545 0.00970241 2024-03-24T12:11:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:47.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:47.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:47.473 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:47.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:47.504 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:47.576 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:47.612 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:47.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:47.657 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:11:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:11:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:47.714 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:47.754 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:47.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:47.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:47.898 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:47.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:47.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:48.032 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:48.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:11:48.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:48.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:48.167 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:48.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:48.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:11:48.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:47 smithi064 ceph-mon[30223]: osdmap e441: 8 total, 8 up, 8 in 2024-03-24T12:11:48.236 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:48 smithi064 ceph-mon[42243]: osdmap e441: 8 total, 8 up, 8 in 2024-03-24T12:11:48.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:48.298 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:48.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:48.331 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 58 16 95037 95021 102.368 65.8125 0.00959727 0.00976249 2024-03-24T12:11:48.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:11:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:11:48.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:47 smithi193 ceph-mon[33593]: osdmap e441: 8 total, 8 up, 8 in 2024-03-24T12:11:48.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:48.428 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:48.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:48.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:48.578 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:48.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:48.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:48.725 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:48.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:48.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:48.870 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:48.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:48.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:49.007 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:49.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:49.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:49.145 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:49.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:49.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:49.275 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:49.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:49.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:49 smithi064 ceph-mon[30223]: pgmap v929: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-24T12:11:49.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:49.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:49 smithi064 ceph-mon[42243]: pgmap v929: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-24T12:11:49.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:49.331 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 59 16 96206 96190 101.871 73.0625 0.00861383 0.00981194 2024-03-24T12:11:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:49 smithi193 ceph-mon[33593]: pgmap v929: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-24T12:11:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:49.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:49.415 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:49.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:49.517 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:49.553 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:49.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:49.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:49.719 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:49.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:49.820 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:49.854 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:49.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:49.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:49.993 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:49.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:49.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:50.095 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:50.126 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:50.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:50.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:50.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:50.267 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:50.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:50.331 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:11:50.333701+0000 min lat: 0.00124462 max lat: 0.182416 avg lat: 0.00984869 2024-03-24T12:11:50.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:11:50.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 60 16 97463 97447 101.482 78.5625 0.00533148 0.00984869 2024-03-24T12:11:50.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:50.404 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:50.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:50.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:50.547 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:50.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:50.649 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:50.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:50.709 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:50.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:50.714 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:11:50.810 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:50.847 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:50.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:50.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:50.979 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:50.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:51.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:51.087 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:51 smithi064 ceph-mon[30223]: pgmap v930: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:51.087 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:51 smithi064 ceph-mon[42243]: pgmap v930: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:51.131 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:51.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:51.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:51.264 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:51.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:51.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 61 16 98653 98637 101.038 74.375 0.00945874 0.00989249 2024-03-24T12:11:51.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:51 smithi193 ceph-mon[33593]: pgmap v930: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:11:51.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:51.406 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:51.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:51.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:51.540 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:51.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:51.642 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:51.677 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:51.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:51.778 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:51.806 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:51.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:51.909 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:51.941 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:51.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:51.976 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:11:51.979 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:11:52.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:52.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:52.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:52.219 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:52.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:52.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:52.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 62 16 99843 99827 100.608 74.375 0.0167918 0.00993479 2024-03-24T12:11:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:52.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:52.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:52.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:52.495 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:52.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:52.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:52.627 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:52.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:52.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:52.766 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:52.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:52.868 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:52.900 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:52.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:53.002 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:53.038 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:53.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:53.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:53.182 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:53.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:53.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:53.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:53 smithi064 ceph-mon[30223]: pgmap v931: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-24T12:11:53.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:53 smithi064 ceph-mon[30223]: from='client.35915 v1:172.21.15.64:0/147298184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:11:53.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:53 smithi064 ceph-mon[42243]: pgmap v931: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-24T12:11:53.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:53 smithi064 ceph-mon[42243]: from='client.35915 v1:172.21.15.64:0/147298184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:11:53.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:53.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:53.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 63 16 101098 101082 100.256 78.4375 0.0153451 0.00996957 2024-03-24T12:11:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:53 smithi193 ceph-mon[33593]: pgmap v931: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-24T12:11:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:53 smithi193 ceph-mon[33593]: from='client.35915 v1:172.21.15.64:0/147298184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:11:53.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:53.448 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:53.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:53.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:53.592 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:53.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:53.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:53.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:53.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:53.791 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:11:53.829 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:53.836 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:11:53.855 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:53.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:53.876 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:11:53.919 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:11:53.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:53.966 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:11:53.992 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:53.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:54.008 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:11:54.050 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:11:54.094 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:11:54.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:54.099 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:11:54.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:11:54.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:11:54.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:11:54.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:11:54.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:11:54.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:11:54.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:11:54.132 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:54.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:54.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:54.271 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:54.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:54.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2540960462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:54.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2540960462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:54.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 64 16 102219 102203 99.7841 70.0625 0.00985176 0.0100169 2024-03-24T12:11:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2540960462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:11:54.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:54.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:54.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:54.557 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:54.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:54.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:54.697 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:54.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:54.798 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:54.833 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:54.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:54.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:54.959 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:54.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:55.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:55.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:55 smithi064 ceph-mon[30223]: pgmap v932: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:55 smithi064 ceph-mon[42243]: pgmap v932: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:55.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 65 15 103389 103374 99.3749 73.1875 0.00541185 0.0100581 2024-03-24T12:11:55.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:55 smithi193 ceph-mon[33593]: pgmap v932: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:11:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:55.375 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:55.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:55.476 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:55.521 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:55.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:55.622 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:11:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:55.663 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:55.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:11:55.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:55.797 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:55.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:55.879 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":931,"stamp":"2024-03-24T12:11:51.544886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6273898846,"num_objects":95921,"num_object_clones":0,"num_object_copies":287763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95921,"num_whiteouts":0,"num_read":1507,"num_read_kb":1259,"num_write":96445,"num_write_kb":6126317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":329608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75897,"ondisk_log_size":75897,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23696928,"kb_used_data":18124876,"kb_used_omap":18,"kb_used_meta":5571949,"kb_avail":726264288,"statfs":{"total":767960285184,"available":743694630912,"internally_reserved":0,"allocated":18559873024,"data_stored":18552734774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":5705675995},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":596770816,"num_objects":9106,"num_object_clones":0,"num_object_copies":27318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9106,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9106,"num_write_kb":582784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003223"},"pg_stats":[{"pgid":"11.7","version":"441'5977","reported_seq":6035,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:49.283209+0000","last_change":"2024-03-24T12:10:42.552347+0000","last_active":"2024-03-24T12:11:49.283209+0000","last_peered":"2024-03-24T12:11:49.283209+0000","last_clean":"2024-03-24T12:11:49.283209+0000","last_became_active":"2024-03-24T12:10:42.552023+0000","last_became_peered":"2024-03-24T12:10:42.552023+0000","last_unstale":"2024-03-24T12:11:49.283209+0000","last_undegraded":"2024-03-24T12:11:49.283209+0000","last_fullsized":"2024-03-24T12:11:49.283209+0000","mapping_epoch":417,"log_start":"429'1100","ondisk_log_start":"429'1100","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4877,"log_dups_size":0,"ondisk_log_size":4877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:07.500025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391708672,"num_objects":5977,"num_object_clones":0,"num_object_copies":17931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":382528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":1080,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442178+0000","last_change":"2024-03-24T12:10:29.922054+0000","last_active":"2024-03-24T12:11:45.442178+0000","last_peered":"2024-03-24T12:11:45.442178+0000","last_clean":"2024-03-24T12:11:45.442178+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:11:45.442178+0000","last_undegraded":"2024-03-24T12:11:45.442178+0000","last_fullsized":"2024-03-24T12:11:45.442178+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:10:29.921946+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.921946+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'27","reported_seq":743,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.005311+0000","last_change":"2024-03-24T12:08:36.154379+0000","last_active":"2024-03-24T12:11:47.005311+0000","last_peered":"2024-03-24T12:11:47.005311+0000","last_clean":"2024-03-24T12:11:47.005311+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:11:47.005311+0000","last_undegraded":"2024-03-24T12:11:47.005311+0000","last_fullsized":"2024-03-24T12:11:47.005311+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:05:17.253630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:17.253630+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519646+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":303,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991774+0000","last_change":"2024-03-24T12:08:32.282623+0000","last_active":"2024-03-24T12:11:46.991774+0000","last_peered":"2024-03-24T12:11:46.991774+0000","last_clean":"2024-03-24T12:11:46.991774+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:11:46.991774+0000","last_undegraded":"2024-03-24T12:11:46.991774+0000","last_fullsized":"2024-03-24T12:11:46.991774+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:32.282513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:32.282513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":735,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.005663+0000","last_change":"2024-03-24T12:10:19.713100+0000","last_active":"2024-03-24T12:11:47.005663+0000","last_peered":"2024-03-24T12:11:47.005663+0000","last_clean":"2024-03-24T12:11:47.005663+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:11:47.005663+0000","last_undegraded":"2024-03-24T12:11:47.005663+0000","last_fullsized":"2024-03-24T12:11:47.005663+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.713016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.713016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519950+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"440'5955","reported_seq":6011,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.736866+0000","last_change":"2024-03-24T12:10:42.558953+0000","last_active":"2024-03-24T12:11:46.736866+0000","last_peered":"2024-03-24T12:11:46.736866+0000","last_clean":"2024-03-24T12:11:46.736866+0000","last_became_active":"2024-03-24T12:10:42.558408+0000","last_became_peered":"2024-03-24T12:10:42.558408+0000","last_unstale":"2024-03-24T12:11:46.736866+0000","last_undegraded":"2024-03-24T12:11:46.736866+0000","last_fullsized":"2024-03-24T12:11:46.736866+0000","mapping_epoch":417,"log_start":"430'1600","ondisk_log_start":"430'1600","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.445780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390266880,"num_objects":5955,"num_object_clones":0,"num_object_copies":17865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":381120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"241'15","reported_seq":850,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991104+0000","last_change":"2024-03-24T12:11:18.368554+0000","last_active":"2024-03-24T12:11:46.991104+0000","last_peered":"2024-03-24T12:11:46.991104+0000","last_clean":"2024-03-24T12:11:46.991104+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:11:46.991104+0000","last_undegraded":"2024-03-24T12:11:46.991104+0000","last_fullsized":"2024-03-24T12:11:46.991104+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:11:18.368416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.368416+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:29.701377+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":862,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.019294+0000","last_change":"2024-03-24T12:11:33.850001+0000","last_active":"2024-03-24T12:11:47.019294+0000","last_peered":"2024-03-24T12:11:47.019294+0000","last_clean":"2024-03-24T12:11:47.019294+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:11:47.019294+0000","last_undegraded":"2024-03-24T12:11:47.019294+0000","last_fullsized":"2024-03-24T12:11:47.019294+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:33.849353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:11:33.849353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:51.989463+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":307,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446256+0000","last_change":"2024-03-24T12:11:37.592094+0000","last_active":"2024-03-24T12:11:45.446256+0000","last_peered":"2024-03-24T12:11:45.446256+0000","last_clean":"2024-03-24T12:11:45.446256+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:11:45.446256+0000","last_undegraded":"2024-03-24T12:11:45.446256+0000","last_fullsized":"2024-03-24T12:11:45.446256+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:37.591971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.591971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:53.358009+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":815,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990939+0000","last_change":"2024-03-24T12:10:26.001526+0000","last_active":"2024-03-24T12:11:46.990939+0000","last_peered":"2024-03-24T12:11:46.990939+0000","last_clean":"2024-03-24T12:11:46.990939+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:11:46.990939+0000","last_undegraded":"2024-03-24T12:11:46.990939+0000","last_fullsized":"2024-03-24T12:11:46.990939+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:26.001400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:26.001400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.491453+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"440'5807","reported_seq":5863,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.739668+0000","last_change":"2024-03-24T12:10:42.558826+0000","last_active":"2024-03-24T12:11:46.739668+0000","last_peered":"2024-03-24T12:11:46.739668+0000","last_clean":"2024-03-24T12:11:46.739668+0000","last_became_active":"2024-03-24T12:10:42.558497+0000","last_became_peered":"2024-03-24T12:10:42.558497+0000","last_unstale":"2024-03-24T12:11:46.739668+0000","last_undegraded":"2024-03-24T12:11:46.739668+0000","last_fullsized":"2024-03-24T12:11:46.739668+0000","mapping_epoch":417,"log_start":"430'1400","ondisk_log_start":"430'1400","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:09.648782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380567552,"num_objects":5807,"num_object_clones":0,"num_object_copies":17421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":371648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"296'44","reported_seq":894,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440356+0000","last_change":"2024-03-24T12:11:20.907053+0000","last_active":"2024-03-24T12:11:45.440356+0000","last_peered":"2024-03-24T12:11:45.440356+0000","last_clean":"2024-03-24T12:11:45.440356+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:11:45.440356+0000","last_undegraded":"2024-03-24T12:11:45.440356+0000","last_fullsized":"2024-03-24T12:11:45.440356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:11:20.906884+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.906884+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:39.544494+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":205,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.006226+0000","last_change":"2024-03-24T12:11:27.439531+0000","last_active":"2024-03-24T12:11:47.006226+0000","last_peered":"2024-03-24T12:11:47.006226+0000","last_clean":"2024-03-24T12:11:47.006226+0000","last_became_active":"2024-03-24T12:08:36.164337+0000","last_became_peered":"2024-03-24T12:08:36.164337+0000","last_unstale":"2024-03-24T12:11:47.006226+0000","last_undegraded":"2024-03-24T12:11:47.006226+0000","last_fullsized":"2024-03-24T12:11:47.006226+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:27.439409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.439409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:27.467179+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":318,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990694+0000","last_change":"2024-03-24T12:11:22.405559+0000","last_active":"2024-03-24T12:11:46.990694+0000","last_peered":"2024-03-24T12:11:46.990694+0000","last_clean":"2024-03-24T12:11:46.990694+0000","last_became_active":"2024-03-24T12:08:36.155105+0000","last_became_peered":"2024-03-24T12:08:36.155105+0000","last_unstale":"2024-03-24T12:11:46.990694+0000","last_undegraded":"2024-03-24T12:11:46.990694+0000","last_fullsized":"2024-03-24T12:11:46.990694+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:22.404955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:22.404955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:31.732903+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":838,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.441700+0000","last_change":"2024-03-24T12:11:32.175174+0000","last_active":"2024-03-24T12:11:45.441700+0000","last_peered":"2024-03-24T12:11:45.441700+0000","last_clean":"2024-03-24T12:11:45.441700+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:11:45.441700+0000","last_undegraded":"2024-03-24T12:11:45.441700+0000","last_fullsized":"2024-03-24T12:11:45.441700+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.175048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.175048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:00.349651+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"441'6209","reported_seq":6267,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:51.524269+0000","last_change":"2024-03-24T12:10:42.555659+0000","last_active":"2024-03-24T12:11:51.524269+0000","last_peered":"2024-03-24T12:11:51.524269+0000","last_clean":"2024-03-24T12:11:51.524269+0000","last_became_active":"2024-03-24T12:10:42.555322+0000","last_became_peered":"2024-03-24T12:10:42.555322+0000","last_unstale":"2024-03-24T12:11:51.524269+0000","last_undegraded":"2024-03-24T12:11:51.524269+0000","last_fullsized":"2024-03-24T12:11:51.524269+0000","mapping_epoch":417,"log_start":"431'1800","ondisk_log_start":"431'1800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:54.319589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406913024,"num_objects":6209,"num_object_clones":0,"num_object_copies":18627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6209,"num_write_kb":397376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"241'39","reported_seq":895,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446529+0000","last_change":"2024-03-24T12:07:41.143174+0000","last_active":"2024-03-24T12:11:45.446529+0000","last_peered":"2024-03-24T12:11:45.446529+0000","last_clean":"2024-03-24T12:11:45.446529+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:11:45.446529+0000","last_undegraded":"2024-03-24T12:11:45.446529+0000","last_fullsized":"2024-03-24T12:11:45.446529+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:07:41.143037+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:07:41.143037+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":608,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446334+0000","last_change":"2024-03-24T12:09:04.230431+0000","last_active":"2024-03-24T12:11:45.446334+0000","last_peered":"2024-03-24T12:11:45.446334+0000","last_clean":"2024-03-24T12:11:45.446334+0000","last_became_active":"2024-03-24T12:09:04.230111+0000","last_became_peered":"2024-03-24T12:09:04.230111+0000","last_unstale":"2024-03-24T12:11:45.446334+0000","last_undegraded":"2024-03-24T12:11:45.446334+0000","last_fullsized":"2024-03-24T12:11:45.446334+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:08:42.842178+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.842178+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":571,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446505+0000","last_change":"2024-03-24T12:11:43.684235+0000","last_active":"2024-03-24T12:11:45.446505+0000","last_peered":"2024-03-24T12:11:45.446505+0000","last_clean":"2024-03-24T12:11:45.446505+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:11:45.446505+0000","last_undegraded":"2024-03-24T12:11:45.446505+0000","last_fullsized":"2024-03-24T12:11:45.446505+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.684070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.684070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:05.544597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":302,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992174+0000","last_change":"2024-03-24T12:08:34.445597+0000","last_active":"2024-03-24T12:11:46.992174+0000","last_peered":"2024-03-24T12:11:46.992174+0000","last_clean":"2024-03-24T12:11:46.992174+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:11:46.992174+0000","last_undegraded":"2024-03-24T12:11:46.992174+0000","last_fullsized":"2024-03-24T12:11:46.992174+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:34.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.445477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":765,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.005525+0000","last_change":"2024-03-24T12:10:09.762381+0000","last_active":"2024-03-24T12:11:47.005525+0000","last_peered":"2024-03-24T12:11:47.005525+0000","last_clean":"2024-03-24T12:11:47.005525+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:11:47.005525+0000","last_undegraded":"2024-03-24T12:11:47.005525+0000","last_fullsized":"2024-03-24T12:11:47.005525+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:09.762247+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.762247+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519850+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"440'5980","reported_seq":6036,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.790916+0000","last_change":"2024-03-24T12:10:42.557591+0000","last_active":"2024-03-24T12:11:46.790916+0000","last_peered":"2024-03-24T12:11:46.790916+0000","last_clean":"2024-03-24T12:11:46.790916+0000","last_became_active":"2024-03-24T12:10:42.557222+0000","last_became_peered":"2024-03-24T12:10:42.557222+0000","last_unstale":"2024-03-24T12:11:46.790916+0000","last_undegraded":"2024-03-24T12:11:46.790916+0000","last_fullsized":"2024-03-24T12:11:46.790916+0000","mapping_epoch":417,"log_start":"429'1200","ondisk_log_start":"429'1200","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:54.567849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391905280,"num_objects":5980,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5980,"num_write_kb":382720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"241'25","reported_seq":849,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446414+0000","last_change":"2024-03-24T12:07:56.328634+0000","last_active":"2024-03-24T12:11:45.446414+0000","last_peered":"2024-03-24T12:11:45.446414+0000","last_clean":"2024-03-24T12:11:45.446414+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:11:45.446414+0000","last_undegraded":"2024-03-24T12:11:45.446414+0000","last_fullsized":"2024-03-24T12:11:45.446414+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:56.328393+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.328393+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":573,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990618+0000","last_change":"2024-03-24T12:10:31.694887+0000","last_active":"2024-03-24T12:11:46.990618+0000","last_peered":"2024-03-24T12:11:46.990618+0000","last_clean":"2024-03-24T12:11:46.990618+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:11:46.990618+0000","last_undegraded":"2024-03-24T12:11:46.990618+0000","last_fullsized":"2024-03-24T12:11:46.990618+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:31.694756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:10:31.694756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:56.762052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":1091,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.997761+0000","last_change":"2024-03-24T12:11:45.062251+0000","last_active":"2024-03-24T12:11:46.997761+0000","last_peered":"2024-03-24T12:11:46.997761+0000","last_clean":"2024-03-24T12:11:46.997761+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:11:46.997761+0000","last_undegraded":"2024-03-24T12:11:46.997761+0000","last_fullsized":"2024-03-24T12:11:46.997761+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:11:45.062114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.062114+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.519407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":826,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446940+0000","last_change":"2024-03-24T12:10:09.602488+0000","last_active":"2024-03-24T12:11:45.446940+0000","last_peered":"2024-03-24T12:11:45.446940+0000","last_clean":"2024-03-24T12:11:45.446940+0000","last_became_active":"2024-03-24T12:08:36.161088+0000","last_became_peered":"2024-03-24T12:08:36.161088+0000","last_unstale":"2024-03-24T12:11:45.446940+0000","last_undegraded":"2024-03-24T12:11:45.446940+0000","last_fullsized":"2024-03-24T12:11:45.446940+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:10:09.602371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.602371+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":1159,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.441871+0000","last_change":"2024-03-24T12:11:42.368078+0000","last_active":"2024-03-24T12:11:45.441871+0000","last_peered":"2024-03-24T12:11:45.441871+0000","last_clean":"2024-03-24T12:11:45.441871+0000","last_became_active":"2024-03-24T12:08:36.161668+0000","last_became_peered":"2024-03-24T12:08:36.161668+0000","last_unstale":"2024-03-24T12:11:45.441871+0000","last_undegraded":"2024-03-24T12:11:45.441871+0000","last_fullsized":"2024-03-24T12:11:45.441871+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:42.367917+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:11:42.367917+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.913748+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"441'6164","reported_seq":6221,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:51.528651+0000","last_change":"2024-03-24T12:10:42.559881+0000","last_active":"2024-03-24T12:11:51.528651+0000","last_peered":"2024-03-24T12:11:51.528651+0000","last_clean":"2024-03-24T12:11:51.528651+0000","last_became_active":"2024-03-24T12:10:42.559566+0000","last_became_peered":"2024-03-24T12:10:42.559566+0000","last_unstale":"2024-03-24T12:11:51.528651+0000","last_undegraded":"2024-03-24T12:11:51.528651+0000","last_fullsized":"2024-03-24T12:11:51.528651+0000","mapping_epoch":417,"log_start":"431'1800","ondisk_log_start":"431'1800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:08.554116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403963904,"num_objects":6164,"num_object_clones":0,"num_object_copies":18492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6164,"num_write_kb":394496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"242'34","reported_seq":892,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.026836+0000","last_change":"2024-03-24T12:10:32.769377+0000","last_active":"2024-03-24T12:11:47.026836+0000","last_peered":"2024-03-24T12:11:47.026836+0000","last_clean":"2024-03-24T12:11:47.026836+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:11:47.026836+0000","last_undegraded":"2024-03-24T12:11:47.026836+0000","last_fullsized":"2024-03-24T12:11:47.026836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:10:32.769281+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:10:32.769281+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:58.251612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":854,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.013972+0000","last_change":"2024-03-24T12:11:43.926041+0000","last_active":"2024-03-24T12:11:47.013972+0000","last_peered":"2024-03-24T12:11:47.013972+0000","last_clean":"2024-03-24T12:11:47.013972+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:11:47.013972+0000","last_undegraded":"2024-03-24T12:11:47.013972+0000","last_fullsized":"2024-03-24T12:11:47.013972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.925905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.925905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:02.893804+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":1067,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446935+0000","last_change":"2024-03-24T12:10:14.509080+0000","last_active":"2024-03-24T12:11:45.446935+0000","last_peered":"2024-03-24T12:11:45.446935+0000","last_clean":"2024-03-24T12:11:45.446935+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:11:45.446935+0000","last_undegraded":"2024-03-24T12:11:45.446935+0000","last_fullsized":"2024-03-24T12:11:45.446935+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:14.508959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.508959+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":743,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.007465+0000","last_change":"2024-03-24T12:11:13.744756+0000","last_active":"2024-03-24T12:11:47.007465+0000","last_peered":"2024-03-24T12:11:47.007465+0000","last_clean":"2024-03-24T12:11:47.007465+0000","last_became_active":"2024-03-24T12:08:36.166897+0000","last_became_peered":"2024-03-24T12:08:36.166897+0000","last_unstale":"2024-03-24T12:11:47.007465+0000","last_undegraded":"2024-03-24T12:11:47.007465+0000","last_fullsized":"2024-03-24T12:11:47.007465+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:13.744672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.744672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:34.151849+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":765,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.008026+0000","last_change":"2024-03-24T12:10:11.797368+0000","last_active":"2024-03-24T12:11:47.008026+0000","last_peered":"2024-03-24T12:11:47.008026+0000","last_clean":"2024-03-24T12:11:47.008026+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:11:47.008026+0000","last_undegraded":"2024-03-24T12:11:47.008026+0000","last_fullsized":"2024-03-24T12:11:47.008026+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:11.797287+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:11.797287+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"440'5813","reported_seq":5869,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.799144+0000","last_change":"2024-03-24T12:10:42.557448+0000","last_active":"2024-03-24T12:11:46.799144+0000","last_peered":"2024-03-24T12:11:46.799144+0000","last_clean":"2024-03-24T12:11:46.799144+0000","last_became_active":"2024-03-24T12:10:42.557136+0000","last_became_peered":"2024-03-24T12:10:42.557136+0000","last_unstale":"2024-03-24T12:11:46.799144+0000","last_undegraded":"2024-03-24T12:11:46.799144+0000","last_fullsized":"2024-03-24T12:11:46.799144+0000","mapping_epoch":417,"log_start":"429'1100","ondisk_log_start":"429'1100","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4713,"log_dups_size":0,"ondisk_log_size":4713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:10.219799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380960768,"num_objects":5813,"num_object_clones":0,"num_object_copies":17439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":372032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"241'24","reported_seq":850,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446753+0000","last_change":"2024-03-24T12:11:21.537427+0000","last_active":"2024-03-24T12:11:45.446753+0000","last_peered":"2024-03-24T12:11:45.446753+0000","last_clean":"2024-03-24T12:11:45.446753+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:11:45.446753+0000","last_undegraded":"2024-03-24T12:11:45.446753+0000","last_fullsized":"2024-03-24T12:11:45.446753+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:11:21.537273+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:11:21.537273+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:37.951661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":845,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.441857+0000","last_change":"2024-03-24T12:11:18.149890+0000","last_active":"2024-03-24T12:11:45.441857+0000","last_peered":"2024-03-24T12:11:45.441857+0000","last_clean":"2024-03-24T12:11:45.441857+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:11:45.441857+0000","last_undegraded":"2024-03-24T12:11:45.441857+0000","last_fullsized":"2024-03-24T12:11:45.441857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:18.149775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.149775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:31.562020+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":731,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.007008+0000","last_change":"2024-03-24T12:08:36.163502+0000","last_active":"2024-03-24T12:11:47.007008+0000","last_peered":"2024-03-24T12:11:47.007008+0000","last_clean":"2024-03-24T12:11:47.007008+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:11:47.007008+0000","last_undegraded":"2024-03-24T12:11:47.007008+0000","last_fullsized":"2024-03-24T12:11:47.007008+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:15.334211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:15.334211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":820,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990477+0000","last_change":"2024-03-24T12:10:14.749596+0000","last_active":"2024-03-24T12:11:46.990477+0000","last_peered":"2024-03-24T12:11:46.990477+0000","last_clean":"2024-03-24T12:11:46.990477+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:11:46.990477+0000","last_undegraded":"2024-03-24T12:11:46.990477+0000","last_fullsized":"2024-03-24T12:11:46.990477+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:14.749468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.749468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"440'5771","reported_seq":5827,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.672448+0000","last_change":"2024-03-24T12:10:42.552380+0000","last_active":"2024-03-24T12:11:46.672448+0000","last_peered":"2024-03-24T12:11:46.672448+0000","last_clean":"2024-03-24T12:11:46.672448+0000","last_became_active":"2024-03-24T12:10:42.552054+0000","last_became_peered":"2024-03-24T12:10:42.552054+0000","last_unstale":"2024-03-24T12:11:46.672448+0000","last_undegraded":"2024-03-24T12:11:46.672448+0000","last_fullsized":"2024-03-24T12:11:46.672448+0000","mapping_epoch":417,"log_start":"428'800","ondisk_log_start":"428'800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:08.753590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378208256,"num_objects":5771,"num_object_clones":0,"num_object_copies":17313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5771,"num_write_kb":369344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":1060,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991776+0000","last_change":"2024-03-24T12:11:38.190269+0000","last_active":"2024-03-24T12:11:46.991776+0000","last_peered":"2024-03-24T12:11:46.991776+0000","last_clean":"2024-03-24T12:11:46.991776+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:11:46.991776+0000","last_undegraded":"2024-03-24T12:11:46.991776+0000","last_fullsized":"2024-03-24T12:11:46.991776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:11:38.190121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:38.190121+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.913586+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"242'35","reported_seq":873,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446053+0000","last_change":"2024-03-24T12:10:19.407534+0000","last_active":"2024-03-24T12:11:45.446053+0000","last_peered":"2024-03-24T12:11:45.446053+0000","last_clean":"2024-03-24T12:11:45.446053+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:11:45.446053+0000","last_undegraded":"2024-03-24T12:11:45.446053+0000","last_fullsized":"2024-03-24T12:11:45.446053+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:10:19.407433+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.407433+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":835,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440485+0000","last_change":"2024-03-24T12:11:16.896924+0000","last_active":"2024-03-24T12:11:45.440485+0000","last_peered":"2024-03-24T12:11:45.440485+0000","last_clean":"2024-03-24T12:11:45.440485+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:11:45.440485+0000","last_undegraded":"2024-03-24T12:11:45.440485+0000","last_fullsized":"2024-03-24T12:11:45.440485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:16.896801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:11:16.896801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:23.741850+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":839,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442719+0000","last_change":"2024-03-24T12:10:18.089858+0000","last_active":"2024-03-24T12:11:45.442719+0000","last_peered":"2024-03-24T12:11:45.442719+0000","last_clean":"2024-03-24T12:11:45.442719+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:11:45.442719+0000","last_undegraded":"2024-03-24T12:11:45.442719+0000","last_fullsized":"2024-03-24T12:11:45.442719+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:18.089744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:18.089744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":899,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990691+0000","last_change":"2024-03-24T12:10:47.972288+0000","last_active":"2024-03-24T12:11:46.990691+0000","last_peered":"2024-03-24T12:11:46.990691+0000","last_clean":"2024-03-24T12:11:46.990691+0000","last_became_active":"2024-03-24T12:10:47.971906+0000","last_became_peered":"2024-03-24T12:10:47.971906+0000","last_unstale":"2024-03-24T12:11:46.990691+0000","last_undegraded":"2024-03-24T12:11:46.990691+0000","last_fullsized":"2024-03-24T12:11:46.990691+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:36.201565+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.201565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"440'5999","reported_seq":6053,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.722145+0000","last_change":"2024-03-24T12:10:42.562451+0000","last_active":"2024-03-24T12:11:46.722145+0000","last_peered":"2024-03-24T12:11:46.722145+0000","last_clean":"2024-03-24T12:11:46.722145+0000","last_became_active":"2024-03-24T12:10:42.562110+0000","last_became_peered":"2024-03-24T12:10:42.562110+0000","last_unstale":"2024-03-24T12:11:46.722145+0000","last_undegraded":"2024-03-24T12:11:46.722145+0000","last_fullsized":"2024-03-24T12:11:46.722145+0000","mapping_epoch":417,"log_start":"430'1600","ondisk_log_start":"430'1600","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4399,"log_dups_size":0,"ondisk_log_size":4399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:08.384408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393150464,"num_objects":5999,"num_object_clones":0,"num_object_copies":17997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5999,"num_write_kb":383936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":811,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446160+0000","last_change":"2024-03-24T12:11:32.519689+0000","last_active":"2024-03-24T12:11:45.446160+0000","last_peered":"2024-03-24T12:11:45.446160+0000","last_clean":"2024-03-24T12:11:45.446160+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:11:45.446160+0000","last_undegraded":"2024-03-24T12:11:45.446160+0000","last_fullsized":"2024-03-24T12:11:45.446160+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.519559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.519559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:57.052977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"241'38","reported_seq":878,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991262+0000","last_change":"2024-03-24T12:10:29.733172+0000","last_active":"2024-03-24T12:11:46.991262+0000","last_peered":"2024-03-24T12:11:46.991262+0000","last_clean":"2024-03-24T12:11:46.991262+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:11:46.991262+0000","last_undegraded":"2024-03-24T12:11:46.991262+0000","last_fullsized":"2024-03-24T12:11:46.991262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_deep_scrub":"241'38","last_deep_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.733073+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":587,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440624+0000","last_change":"2024-03-24T12:11:27.010877+0000","last_active":"2024-03-24T12:11:45.440624+0000","last_peered":"2024-03-24T12:11:45.440624+0000","last_clean":"2024-03-24T12:11:45.440624+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:11:45.440624+0000","last_undegraded":"2024-03-24T12:11:45.440624+0000","last_fullsized":"2024-03-24T12:11:45.440624+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:11:27.010737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.010737+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:34.851382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":829,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446264+0000","last_change":"2024-03-24T12:10:38.401132+0000","last_active":"2024-03-24T12:11:45.446264+0000","last_peered":"2024-03-24T12:11:45.446264+0000","last_clean":"2024-03-24T12:11:45.446264+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:11:45.446264+0000","last_undegraded":"2024-03-24T12:11:45.446264+0000","last_fullsized":"2024-03-24T12:11:45.446264+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:38.401028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:10:38.401028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:07.118106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":903,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446006+0000","last_change":"2024-03-24T12:10:45.574511+0000","last_active":"2024-03-24T12:11:45.446006+0000","last_peered":"2024-03-24T12:11:45.446006+0000","last_clean":"2024-03-24T12:11:45.446006+0000","last_became_active":"2024-03-24T12:10:45.573960+0000","last_became_peered":"2024-03-24T12:10:45.573960+0000","last_unstale":"2024-03-24T12:11:45.446006+0000","last_undegraded":"2024-03-24T12:11:45.446006+0000","last_fullsized":"2024-03-24T12:11:45.446006+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:25.359883+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.359883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:48.227960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"441'5781","reported_seq":5839,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.218440+0000","last_change":"2024-03-24T12:10:42.552647+0000","last_active":"2024-03-24T12:11:47.218440+0000","last_peered":"2024-03-24T12:11:47.218440+0000","last_clean":"2024-03-24T12:11:47.218440+0000","last_became_active":"2024-03-24T12:10:42.552346+0000","last_became_peered":"2024-03-24T12:10:42.552346+0000","last_unstale":"2024-03-24T12:11:47.218440+0000","last_undegraded":"2024-03-24T12:11:47.218440+0000","last_fullsized":"2024-03-24T12:11:47.218440+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378863616,"num_objects":5781,"num_object_clones":0,"num_object_copies":17343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5781,"num_write_kb":369984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":828,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440554+0000","last_change":"2024-03-24T12:10:14.821427+0000","last_active":"2024-03-24T12:11:45.440554+0000","last_peered":"2024-03-24T12:11:45.440554+0000","last_clean":"2024-03-24T12:11:45.440554+0000","last_became_active":"2024-03-24T12:08:36.161658+0000","last_became_peered":"2024-03-24T12:08:36.161658+0000","last_unstale":"2024-03-24T12:11:45.440554+0000","last_undegraded":"2024-03-24T12:11:45.440554+0000","last_fullsized":"2024-03-24T12:11:45.440554+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:14.821157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.821157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"369'37","reported_seq":886,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990846+0000","last_change":"2024-03-24T12:11:37.405546+0000","last_active":"2024-03-24T12:11:46.990846+0000","last_peered":"2024-03-24T12:11:46.990846+0000","last_clean":"2024-03-24T12:11:46.990846+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:11:46.990846+0000","last_undegraded":"2024-03-24T12:11:46.990846+0000","last_fullsized":"2024-03-24T12:11:46.990846+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'37","last_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_deep_scrub":"369'37","last_deep_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.405400+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:04.594252+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":853,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.443318+0000","last_change":"2024-03-24T12:11:17.183566+0000","last_active":"2024-03-24T12:11:45.443318+0000","last_peered":"2024-03-24T12:11:45.443318+0000","last_clean":"2024-03-24T12:11:45.443318+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:11:45.443318+0000","last_undegraded":"2024-03-24T12:11:45.443318+0000","last_fullsized":"2024-03-24T12:11:45.443318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:17.183126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:11:17.183126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.643290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":577,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442644+0000","last_change":"2024-03-24T12:10:26.791811+0000","last_active":"2024-03-24T12:11:45.442644+0000","last_peered":"2024-03-24T12:11:45.442644+0000","last_clean":"2024-03-24T12:11:45.442644+0000","last_became_active":"2024-03-24T12:05:44.738926+0000","last_became_peered":"2024-03-24T12:05:44.738926+0000","last_unstale":"2024-03-24T12:11:45.442644+0000","last_undegraded":"2024-03-24T12:11:45.442644+0000","last_fullsized":"2024-03-24T12:11:45.442644+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:26.791681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:26.791681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":917,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440850+0000","last_change":"2024-03-24T12:11:45.081667+0000","last_active":"2024-03-24T12:11:45.440850+0000","last_peered":"2024-03-24T12:11:45.440850+0000","last_clean":"2024-03-24T12:11:45.440850+0000","last_became_active":"2024-03-24T12:10:41.554328+0000","last_became_peered":"2024-03-24T12:10:41.554328+0000","last_unstale":"2024-03-24T12:11:45.440850+0000","last_undegraded":"2024-03-24T12:11:45.440850+0000","last_fullsized":"2024-03-24T12:11:45.440850+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.081517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.235760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"441'6117","reported_seq":6175,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:51.492089+0000","last_change":"2024-03-24T12:10:42.558775+0000","last_active":"2024-03-24T12:11:51.492089+0000","last_peered":"2024-03-24T12:11:51.492089+0000","last_clean":"2024-03-24T12:11:51.492089+0000","last_became_active":"2024-03-24T12:10:42.558462+0000","last_became_peered":"2024-03-24T12:10:42.558462+0000","last_unstale":"2024-03-24T12:11:51.492089+0000","last_undegraded":"2024-03-24T12:11:51.492089+0000","last_fullsized":"2024-03-24T12:11:51.492089+0000","mapping_epoch":417,"log_start":"431'1700","ondisk_log_start":"431'1700","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4417,"log_dups_size":0,"ondisk_log_size":4417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:00.091227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400883712,"num_objects":6117,"num_object_clones":0,"num_object_copies":18351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6117,"num_write_kb":391488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":697,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.007259+0000","last_change":"2024-03-24T12:08:36.165575+0000","last_active":"2024-03-24T12:11:47.007259+0000","last_peered":"2024-03-24T12:11:47.007259+0000","last_clean":"2024-03-24T12:11:47.007259+0000","last_became_active":"2024-03-24T12:08:36.165265+0000","last_became_peered":"2024-03-24T12:08:36.165265+0000","last_unstale":"2024-03-24T12:11:47.007259+0000","last_undegraded":"2024-03-24T12:11:47.007259+0000","last_fullsized":"2024-03-24T12:11:47.007259+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:28.232417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:05:28.232417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"241'30","reported_seq":888,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992228+0000","last_change":"2024-03-24T12:11:23.383853+0000","last_active":"2024-03-24T12:11:46.992228+0000","last_peered":"2024-03-24T12:11:46.992228+0000","last_clean":"2024-03-24T12:11:46.992228+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:11:46.992228+0000","last_undegraded":"2024-03-24T12:11:46.992228+0000","last_fullsized":"2024-03-24T12:11:46.992228+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:11:23.383616+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.383616+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:33.198353+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":666,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991348+0000","last_change":"2024-03-24T12:10:16.036972+0000","last_active":"2024-03-24T12:11:46.991348+0000","last_peered":"2024-03-24T12:11:46.991348+0000","last_clean":"2024-03-24T12:11:46.991348+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:11:46.991348+0000","last_undegraded":"2024-03-24T12:11:46.991348+0000","last_fullsized":"2024-03-24T12:11:46.991348+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:16.036850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:16.036850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.491426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":840,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990122+0000","last_change":"2024-03-24T12:11:46.990122+0000","last_active":"2024-03-24T12:11:46.990122+0000","last_peered":"2024-03-24T12:11:46.990122+0000","last_clean":"2024-03-24T12:11:46.990122+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:11:46.990122+0000","last_undegraded":"2024-03-24T12:11:46.990122+0000","last_fullsized":"2024-03-24T12:11:46.990122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:29.995266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.995266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:50.460345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":844,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.020445+0000","last_change":"2024-03-24T12:11:39.903255+0000","last_active":"2024-03-24T12:11:47.020445+0000","last_peered":"2024-03-24T12:11:47.020445+0000","last_clean":"2024-03-24T12:11:47.020445+0000","last_became_active":"2024-03-24T12:08:36.173357+0000","last_became_peered":"2024-03-24T12:08:36.173357+0000","last_unstale":"2024-03-24T12:11:47.020445+0000","last_undegraded":"2024-03-24T12:11:47.020445+0000","last_fullsized":"2024-03-24T12:11:47.020445+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_clean_scrub_stamp":"2024-03-24T12:11:39.903133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:04.706316+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":919,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442078+0000","last_change":"2024-03-24T12:10:50.999143+0000","last_active":"2024-03-24T12:11:45.442078+0000","last_peered":"2024-03-24T12:11:45.442078+0000","last_clean":"2024-03-24T12:11:45.442078+0000","last_became_active":"2024-03-24T12:10:50.998877+0000","last_became_peered":"2024-03-24T12:10:50.998877+0000","last_unstale":"2024-03-24T12:11:45.442078+0000","last_undegraded":"2024-03-24T12:11:45.442078+0000","last_fullsized":"2024-03-24T12:11:45.442078+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:10.807236+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.807236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"441'6152","reported_seq":6210,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:51.483709+0000","last_change":"2024-03-24T12:10:42.558754+0000","last_active":"2024-03-24T12:11:51.483709+0000","last_peered":"2024-03-24T12:11:51.483709+0000","last_clean":"2024-03-24T12:11:51.483709+0000","last_became_active":"2024-03-24T12:10:42.558225+0000","last_became_peered":"2024-03-24T12:10:42.558225+0000","last_unstale":"2024-03-24T12:11:51.483709+0000","last_undegraded":"2024-03-24T12:11:51.483709+0000","last_fullsized":"2024-03-24T12:11:51.483709+0000","mapping_epoch":417,"log_start":"431'1800","ondisk_log_start":"431'1800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.545589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403177472,"num_objects":6152,"num_object_clones":0,"num_object_copies":18456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":393728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":838,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.443018+0000","last_change":"2024-03-24T12:10:37.011600+0000","last_active":"2024-03-24T12:11:45.443018+0000","last_peered":"2024-03-24T12:11:45.443018+0000","last_clean":"2024-03-24T12:11:45.443018+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:11:45.443018+0000","last_undegraded":"2024-03-24T12:11:45.443018+0000","last_fullsized":"2024-03-24T12:11:45.443018+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:37.011493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:10:37.011493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:54.370159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"241'28","reported_seq":1122,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440428+0000","last_change":"2024-03-24T12:11:32.027906+0000","last_active":"2024-03-24T12:11:45.440428+0000","last_peered":"2024-03-24T12:11:45.440428+0000","last_clean":"2024-03-24T12:11:45.440428+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:11:45.440428+0000","last_undegraded":"2024-03-24T12:11:45.440428+0000","last_fullsized":"2024-03-24T12:11:45.440428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:11:32.027804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.027804+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:44.058819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":1157,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442139+0000","last_change":"2024-03-24T12:11:35.238528+0000","last_active":"2024-03-24T12:11:45.442139+0000","last_peered":"2024-03-24T12:11:45.442139+0000","last_clean":"2024-03-24T12:11:45.442139+0000","last_became_active":"2024-03-24T12:10:33.461798+0000","last_became_peered":"2024-03-24T12:10:33.461798+0000","last_unstale":"2024-03-24T12:11:45.442139+0000","last_undegraded":"2024-03-24T12:11:45.442139+0000","last_fullsized":"2024-03-24T12:11:45.442139+0000","mapping_epoch":64,"log_start":"83'130","ondisk_log_start":"83'130","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:35.238422+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:10:25.984109+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.238422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.501130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":737,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.006136+0000","last_change":"2024-03-24T12:08:36.169985+0000","last_active":"2024-03-24T12:11:47.006136+0000","last_peered":"2024-03-24T12:11:47.006136+0000","last_clean":"2024-03-24T12:11:47.006136+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:11:47.006136+0000","last_undegraded":"2024-03-24T12:11:47.006136+0000","last_fullsized":"2024-03-24T12:11:47.006136+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.356759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.356759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":834,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990575+0000","last_change":"2024-03-24T12:08:43.113674+0000","last_active":"2024-03-24T12:11:46.990575+0000","last_peered":"2024-03-24T12:11:46.990575+0000","last_clean":"2024-03-24T12:11:46.990575+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:11:46.990575+0000","last_undegraded":"2024-03-24T12:11:46.990575+0000","last_fullsized":"2024-03-24T12:11:46.990575+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:08:43.113510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.113510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":700,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.007164+0000","last_change":"2024-03-24T12:11:35.488009+0000","last_active":"2024-03-24T12:11:47.007164+0000","last_peered":"2024-03-24T12:11:47.007164+0000","last_clean":"2024-03-24T12:11:47.007164+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:11:47.007164+0000","last_undegraded":"2024-03-24T12:11:47.007164+0000","last_fullsized":"2024-03-24T12:11:47.007164+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:35.487872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.487872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:03.379847+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"441'6170","reported_seq":6228,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:51.521079+0000","last_change":"2024-03-24T12:10:42.558943+0000","last_active":"2024-03-24T12:11:51.521079+0000","last_peered":"2024-03-24T12:11:51.521079+0000","last_clean":"2024-03-24T12:11:51.521079+0000","last_became_active":"2024-03-24T12:10:42.558601+0000","last_became_peered":"2024-03-24T12:10:42.558601+0000","last_unstale":"2024-03-24T12:11:51.521079+0000","last_undegraded":"2024-03-24T12:11:51.521079+0000","last_fullsized":"2024-03-24T12:11:51.521079+0000","mapping_epoch":417,"log_start":"431'1800","ondisk_log_start":"431'1800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:57.508925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404357120,"num_objects":6170,"num_object_clones":0,"num_object_copies":18510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":394880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":219,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.006774+0000","last_change":"2024-03-24T12:08:36.162550+0000","last_active":"2024-03-24T12:11:47.006774+0000","last_peered":"2024-03-24T12:11:47.006774+0000","last_clean":"2024-03-24T12:11:47.006774+0000","last_became_active":"2024-03-24T12:08:36.162188+0000","last_became_peered":"2024-03-24T12:08:36.162188+0000","last_unstale":"2024-03-24T12:11:47.006774+0000","last_undegraded":"2024-03-24T12:11:47.006774+0000","last_fullsized":"2024-03-24T12:11:47.006774+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:05:19.646040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"241'17","reported_seq":885,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.026777+0000","last_change":"2024-03-24T12:11:27.012324+0000","last_active":"2024-03-24T12:11:47.026777+0000","last_peered":"2024-03-24T12:11:47.026777+0000","last_clean":"2024-03-24T12:11:47.026777+0000","last_became_active":"2024-03-24T12:08:36.172879+0000","last_became_peered":"2024-03-24T12:08:36.172879+0000","last_unstale":"2024-03-24T12:11:47.026777+0000","last_undegraded":"2024-03-24T12:11:47.026777+0000","last_fullsized":"2024-03-24T12:11:47.026777+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:11:27.012204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.012204+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.287575+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":303,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991796+0000","last_change":"2024-03-24T12:10:12.685963+0000","last_active":"2024-03-24T12:11:46.991796+0000","last_peered":"2024-03-24T12:11:46.991796+0000","last_clean":"2024-03-24T12:11:46.991796+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:11:46.991796+0000","last_undegraded":"2024-03-24T12:11:46.991796+0000","last_fullsized":"2024-03-24T12:11:46.991796+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.685815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.685815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":716,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.425274+0000","last_change":"2024-03-24T12:11:13.936448+0000","last_active":"2024-03-24T12:11:46.425274+0000","last_peered":"2024-03-24T12:11:46.425274+0000","last_clean":"2024-03-24T12:11:46.425274+0000","last_became_active":"2024-03-24T12:05:44.736437+0000","last_became_peered":"2024-03-24T12:05:44.736437+0000","last_unstale":"2024-03-24T12:11:46.425274+0000","last_undegraded":"2024-03-24T12:11:46.425274+0000","last_fullsized":"2024-03-24T12:11:46.425274+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:11:13.936256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.936256+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:31.155826+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":368,"num_read_kb":363,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":833,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.443039+0000","last_change":"2024-03-24T12:11:31.240744+0000","last_active":"2024-03-24T12:11:45.443039+0000","last_peered":"2024-03-24T12:11:45.443039+0000","last_clean":"2024-03-24T12:11:45.443039+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:11:45.443039+0000","last_undegraded":"2024-03-24T12:11:45.443039+0000","last_fullsized":"2024-03-24T12:11:45.443039+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:31.240650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:31.240650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:46.619564+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":1156,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442458+0000","last_change":"2024-03-24T12:11:30.174297+0000","last_active":"2024-03-24T12:11:45.442458+0000","last_peered":"2024-03-24T12:11:45.442458+0000","last_clean":"2024-03-24T12:11:45.442458+0000","last_became_active":"2024-03-24T12:10:23.398177+0000","last_became_peered":"2024-03-24T12:10:23.398177+0000","last_unstale":"2024-03-24T12:11:45.442458+0000","last_undegraded":"2024-03-24T12:11:45.442458+0000","last_fullsized":"2024-03-24T12:11:45.442458+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:30.173569+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:11:30.173569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:40.507040+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"441'6242","reported_seq":6300,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:51.523852+0000","last_change":"2024-03-24T12:10:42.559860+0000","last_active":"2024-03-24T12:11:51.523852+0000","last_peered":"2024-03-24T12:11:51.523852+0000","last_clean":"2024-03-24T12:11:51.523852+0000","last_became_active":"2024-03-24T12:10:42.559548+0000","last_became_peered":"2024-03-24T12:10:42.559548+0000","last_unstale":"2024-03-24T12:11:51.523852+0000","last_undegraded":"2024-03-24T12:11:51.523852+0000","last_fullsized":"2024-03-24T12:11:51.523852+0000","mapping_epoch":417,"log_start":"431'1800","ondisk_log_start":"431'1800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:09.669545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409075712,"num_objects":6242,"num_object_clones":0,"num_object_copies":18726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":399488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":822,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991769+0000","last_change":"2024-03-24T12:11:38.437189+0000","last_active":"2024-03-24T12:11:46.991769+0000","last_peered":"2024-03-24T12:11:46.991769+0000","last_clean":"2024-03-24T12:11:46.991769+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:11:46.991769+0000","last_undegraded":"2024-03-24T12:11:46.991769+0000","last_fullsized":"2024-03-24T12:11:46.991769+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:38.437087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:11:38.437087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:46.086372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"354'44","reported_seq":960,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.006522+0000","last_change":"2024-03-24T12:10:33.726470+0000","last_active":"2024-03-24T12:11:47.006522+0000","last_peered":"2024-03-24T12:11:47.006522+0000","last_clean":"2024-03-24T12:11:47.006522+0000","last_became_active":"2024-03-24T12:08:36.152866+0000","last_became_peered":"2024-03-24T12:08:36.152866+0000","last_unstale":"2024-03-24T12:11:47.006522+0000","last_undegraded":"2024-03-24T12:11:47.006522+0000","last_fullsized":"2024-03-24T12:11:47.006522+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'44","last_scrub_stamp":"2024-03-24T12:10:33.726357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:33.726357+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:55.232205+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":709,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.005485+0000","last_change":"2024-03-24T12:11:19.470678+0000","last_active":"2024-03-24T12:11:47.005485+0000","last_peered":"2024-03-24T12:11:47.005485+0000","last_clean":"2024-03-24T12:11:47.005485+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:11:47.005485+0000","last_undegraded":"2024-03-24T12:11:47.005485+0000","last_fullsized":"2024-03-24T12:11:47.005485+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:19.470485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.470485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:38.111645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":812,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992151+0000","last_change":"2024-03-24T12:11:28.156543+0000","last_active":"2024-03-24T12:11:46.992151+0000","last_peered":"2024-03-24T12:11:46.992151+0000","last_clean":"2024-03-24T12:11:46.992151+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:11:46.992151+0000","last_undegraded":"2024-03-24T12:11:46.992151+0000","last_fullsized":"2024-03-24T12:11:46.992151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:28.156411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.156411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:30.950628+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":784,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.005307+0000","last_change":"2024-03-24T12:10:26.959642+0000","last_active":"2024-03-24T12:11:47.005307+0000","last_peered":"2024-03-24T12:11:47.005307+0000","last_clean":"2024-03-24T12:11:47.005307+0000","last_became_active":"2024-03-24T12:10:26.959434+0000","last_became_peered":"2024-03-24T12:10:26.959434+0000","last_unstale":"2024-03-24T12:11:47.005307+0000","last_undegraded":"2024-03-24T12:11:47.005307+0000","last_fullsized":"2024-03-24T12:11:47.005307+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:10.760367+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.760367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.520011+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"440'5780","reported_seq":5836,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.646495+0000","last_change":"2024-03-24T12:10:42.558548+0000","last_active":"2024-03-24T12:11:46.646495+0000","last_peered":"2024-03-24T12:11:46.646495+0000","last_clean":"2024-03-24T12:11:46.646495+0000","last_became_active":"2024-03-24T12:10:42.558230+0000","last_became_peered":"2024-03-24T12:10:42.558230+0000","last_unstale":"2024-03-24T12:11:46.646495+0000","last_undegraded":"2024-03-24T12:11:46.646495+0000","last_fullsized":"2024-03-24T12:11:46.646495+0000","mapping_epoch":417,"log_start":"428'800","ondisk_log_start":"428'800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:55.629693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378798080,"num_objects":5780,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5780,"num_write_kb":369920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":1076,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991205+0000","last_change":"2024-03-24T12:11:19.363311+0000","last_active":"2024-03-24T12:11:46.991205+0000","last_peered":"2024-03-24T12:11:46.991205+0000","last_clean":"2024-03-24T12:11:46.991205+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:11:46.991205+0000","last_undegraded":"2024-03-24T12:11:46.991205+0000","last_fullsized":"2024-03-24T12:11:46.991205+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:11:19.362922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.362922+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:43.620435+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"241'25","reported_seq":887,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.013924+0000","last_change":"2024-03-24T12:11:24.029766+0000","last_active":"2024-03-24T12:11:47.013924+0000","last_peered":"2024-03-24T12:11:47.013924+0000","last_clean":"2024-03-24T12:11:47.013924+0000","last_became_active":"2024-03-24T12:08:36.174663+0000","last_became_peered":"2024-03-24T12:08:36.174663+0000","last_unstale":"2024-03-24T12:11:47.013924+0000","last_undegraded":"2024-03-24T12:11:47.013924+0000","last_fullsized":"2024-03-24T12:11:47.013924+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:11:24.029563+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:11:24.029563+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.448106+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":572,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990636+0000","last_change":"2024-03-24T12:10:08.168976+0000","last_active":"2024-03-24T12:11:46.990636+0000","last_peered":"2024-03-24T12:11:46.990636+0000","last_clean":"2024-03-24T12:11:46.990636+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:11:46.990636+0000","last_undegraded":"2024-03-24T12:11:46.990636+0000","last_fullsized":"2024-03-24T12:11:46.990636+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:08.168832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:10:08.168832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:50.445336+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":825,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990451+0000","last_change":"2024-03-24T12:11:26.507311+0000","last_active":"2024-03-24T12:11:46.990451+0000","last_peered":"2024-03-24T12:11:46.990451+0000","last_clean":"2024-03-24T12:11:46.990451+0000","last_became_active":"2024-03-24T12:08:36.145936+0000","last_became_peered":"2024-03-24T12:08:36.145936+0000","last_unstale":"2024-03-24T12:11:46.990451+0000","last_undegraded":"2024-03-24T12:11:46.990451+0000","last_fullsized":"2024-03-24T12:11:46.990451+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:26.507184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:11:26.507184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.897133+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":909,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440247+0000","last_change":"2024-03-24T12:11:18.911425+0000","last_active":"2024-03-24T12:11:45.440247+0000","last_peered":"2024-03-24T12:11:45.440247+0000","last_clean":"2024-03-24T12:11:45.440247+0000","last_became_active":"2024-03-24T12:10:36.965307+0000","last_became_peered":"2024-03-24T12:10:36.965307+0000","last_unstale":"2024-03-24T12:11:45.440247+0000","last_undegraded":"2024-03-24T12:11:45.440247+0000","last_fullsized":"2024-03-24T12:11:45.440247+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:18.911296+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.911296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:22.192298+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"441'5806","reported_seq":5864,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:48.767253+0000","last_change":"2024-03-24T12:10:42.558716+0000","last_active":"2024-03-24T12:11:48.767253+0000","last_peered":"2024-03-24T12:11:48.767253+0000","last_clean":"2024-03-24T12:11:48.767253+0000","last_became_active":"2024-03-24T12:10:42.558369+0000","last_became_peered":"2024-03-24T12:10:42.558369+0000","last_unstale":"2024-03-24T12:11:48.767253+0000","last_undegraded":"2024-03-24T12:11:48.767253+0000","last_fullsized":"2024-03-24T12:11:48.767253+0000","mapping_epoch":417,"log_start":"428'800","ondisk_log_start":"428'800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.999822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380502016,"num_objects":5806,"num_object_clones":0,"num_object_copies":17418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":371584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":845,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.997161+0000","last_change":"2024-03-24T12:11:19.790253+0000","last_active":"2024-03-24T12:11:46.997161+0000","last_peered":"2024-03-24T12:11:46.997161+0000","last_clean":"2024-03-24T12:11:46.997161+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:11:46.997161+0000","last_undegraded":"2024-03-24T12:11:46.997161+0000","last_fullsized":"2024-03-24T12:11:46.997161+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:19.790086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.790086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.399995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"241'18","reported_seq":872,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446098+0000","last_change":"2024-03-24T12:08:34.327181+0000","last_active":"2024-03-24T12:11:45.446098+0000","last_peered":"2024-03-24T12:11:45.446098+0000","last_clean":"2024-03-24T12:11:45.446098+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:11:45.446098+0000","last_undegraded":"2024-03-24T12:11:45.446098+0000","last_fullsized":"2024-03-24T12:11:45.446098+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:08:34.327037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.327037+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":830,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446620+0000","last_change":"2024-03-24T12:07:45.216727+0000","last_active":"2024-03-24T12:11:45.446620+0000","last_peered":"2024-03-24T12:11:45.446620+0000","last_clean":"2024-03-24T12:11:45.446620+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:11:45.446620+0000","last_undegraded":"2024-03-24T12:11:45.446620+0000","last_fullsized":"2024-03-24T12:11:45.446620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:07:45.216581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:07:45.216581+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":803,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990966+0000","last_change":"2024-03-24T12:07:47.318834+0000","last_active":"2024-03-24T12:11:46.990966+0000","last_peered":"2024-03-24T12:11:46.990966+0000","last_clean":"2024-03-24T12:11:46.990966+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:11:46.990966+0000","last_undegraded":"2024-03-24T12:11:46.990966+0000","last_fullsized":"2024-03-24T12:11:46.990966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:47.318040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:07:47.318040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"241'19","reported_seq":862,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992285+0000","last_change":"2024-03-24T12:11:40.464348+0000","last_active":"2024-03-24T12:11:46.992285+0000","last_peered":"2024-03-24T12:11:46.992285+0000","last_clean":"2024-03-24T12:11:46.992285+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:11:46.992285+0000","last_undegraded":"2024-03-24T12:11:46.992285+0000","last_fullsized":"2024-03-24T12:11:46.992285+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:40.464206+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:11:40.464206+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:55.591744+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":318,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991190+0000","last_change":"2024-03-24T12:11:20.390445+0000","last_active":"2024-03-24T12:11:46.991190+0000","last_peered":"2024-03-24T12:11:46.991190+0000","last_clean":"2024-03-24T12:11:46.991190+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:11:46.991190+0000","last_undegraded":"2024-03-24T12:11:46.991190+0000","last_fullsized":"2024-03-24T12:11:46.991190+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:20.390312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.390312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:44.725416+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":807,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992022+0000","last_change":"2024-03-24T12:10:30.721092+0000","last_active":"2024-03-24T12:11:46.992022+0000","last_peered":"2024-03-24T12:11:46.992022+0000","last_clean":"2024-03-24T12:11:46.992022+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:11:46.992022+0000","last_undegraded":"2024-03-24T12:11:46.992022+0000","last_fullsized":"2024-03-24T12:11:46.992022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:30.721003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:30.721003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:47.664229+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":299,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446314+0000","last_change":"2024-03-24T12:07:50.278024+0000","last_active":"2024-03-24T12:11:45.446314+0000","last_peered":"2024-03-24T12:11:45.446314+0000","last_clean":"2024-03-24T12:11:45.446314+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:11:45.446314+0000","last_undegraded":"2024-03-24T12:11:45.446314+0000","last_fullsized":"2024-03-24T12:11:45.446314+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:50.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:50.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"241'12","reported_seq":837,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991916+0000","last_change":"2024-03-24T12:10:25.685130+0000","last_active":"2024-03-24T12:11:46.991916+0000","last_peered":"2024-03-24T12:11:46.991916+0000","last_clean":"2024-03-24T12:11:46.991916+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:11:46.991916+0000","last_undegraded":"2024-03-24T12:11:46.991916+0000","last_fullsized":"2024-03-24T12:11:46.991916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:10:25.685009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.685009+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":842,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.443105+0000","last_change":"2024-03-24T12:11:43.032559+0000","last_active":"2024-03-24T12:11:45.443105+0000","last_peered":"2024-03-24T12:11:45.443105+0000","last_clean":"2024-03-24T12:11:45.443105+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:11:45.443105+0000","last_undegraded":"2024-03-24T12:11:45.443105+0000","last_fullsized":"2024-03-24T12:11:45.443105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.032403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.032403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:12.416512+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":657,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.006940+0000","last_change":"2024-03-24T12:10:25.718633+0000","last_active":"2024-03-24T12:11:47.006940+0000","last_peered":"2024-03-24T12:11:47.006940+0000","last_clean":"2024-03-24T12:11:47.006940+0000","last_became_active":"2024-03-24T12:08:36.165715+0000","last_became_peered":"2024-03-24T12:08:36.165715+0000","last_unstale":"2024-03-24T12:11:47.006940+0000","last_undegraded":"2024-03-24T12:11:47.006940+0000","last_fullsized":"2024-03-24T12:11:47.006940+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:25.718555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.718555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.520040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":827,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442854+0000","last_change":"2024-03-24T12:10:19.030271+0000","last_active":"2024-03-24T12:11:45.442854+0000","last_peered":"2024-03-24T12:11:45.442854+0000","last_clean":"2024-03-24T12:11:45.442854+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:11:45.442854+0000","last_undegraded":"2024-03-24T12:11:45.442854+0000","last_fullsized":"2024-03-24T12:11:45.442854+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.030172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.030172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'25","reported_seq":854,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990581+0000","last_change":"2024-03-24T12:10:28.696002+0000","last_active":"2024-03-24T12:11:46.990581+0000","last_peered":"2024-03-24T12:11:46.990581+0000","last_clean":"2024-03-24T12:11:46.990581+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:11:46.990581+0000","last_undegraded":"2024-03-24T12:11:46.990581+0000","last_fullsized":"2024-03-24T12:11:46.990581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:28.695890+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:10:28.695890+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:57.427147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":708,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.007852+0000","last_change":"2024-03-24T12:11:32.439386+0000","last_active":"2024-03-24T12:11:47.007852+0000","last_peered":"2024-03-24T12:11:47.007852+0000","last_clean":"2024-03-24T12:11:47.007852+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:11:47.007852+0000","last_undegraded":"2024-03-24T12:11:47.007852+0000","last_fullsized":"2024-03-24T12:11:47.007852+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.439301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.439301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:32.556201+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":959,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.007218+0000","last_change":"2024-03-24T12:10:15.754439+0000","last_active":"2024-03-24T12:11:47.007218+0000","last_peered":"2024-03-24T12:11:47.007218+0000","last_clean":"2024-03-24T12:11:47.007218+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:11:47.007218+0000","last_undegraded":"2024-03-24T12:11:47.007218+0000","last_fullsized":"2024-03-24T12:11:47.007218+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:15.754352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:15.754352+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.520095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":817,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992084+0000","last_change":"2024-03-24T12:11:17.508810+0000","last_active":"2024-03-24T12:11:46.992084+0000","last_peered":"2024-03-24T12:11:46.992084+0000","last_clean":"2024-03-24T12:11:46.992084+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:11:46.992084+0000","last_undegraded":"2024-03-24T12:11:46.992084+0000","last_fullsized":"2024-03-24T12:11:46.992084+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:17.508697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:17.508697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:21.358451+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'15","reported_seq":725,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.007661+0000","last_change":"2024-03-24T12:08:36.154109+0000","last_active":"2024-03-24T12:11:47.007661+0000","last_peered":"2024-03-24T12:11:47.007661+0000","last_clean":"2024-03-24T12:11:47.007661+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:11:47.007661+0000","last_undegraded":"2024-03-24T12:11:47.007661+0000","last_fullsized":"2024-03-24T12:11:47.007661+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":703,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991012+0000","last_change":"2024-03-24T12:10:34.022259+0000","last_active":"2024-03-24T12:11:46.991012+0000","last_peered":"2024-03-24T12:11:46.991012+0000","last_clean":"2024-03-24T12:11:46.991012+0000","last_became_active":"2024-03-24T12:10:30.443191+0000","last_became_peered":"2024-03-24T12:10:30.443191+0000","last_unstale":"2024-03-24T12:11:46.991012+0000","last_undegraded":"2024-03-24T12:11:46.991012+0000","last_fullsized":"2024-03-24T12:11:46.991012+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:34.022182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:34.022182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:48.597395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":1069,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990907+0000","last_change":"2024-03-24T12:10:19.664090+0000","last_active":"2024-03-24T12:11:46.990907+0000","last_peered":"2024-03-24T12:11:46.990907+0000","last_clean":"2024-03-24T12:11:46.990907+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:11:46.990907+0000","last_undegraded":"2024-03-24T12:11:46.990907+0000","last_fullsized":"2024-03-24T12:11:46.990907+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:10:19.664009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.664009+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:54.261421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":815,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446799+0000","last_change":"2024-03-24T12:10:17.566669+0000","last_active":"2024-03-24T12:11:45.446799+0000","last_peered":"2024-03-24T12:11:45.446799+0000","last_clean":"2024-03-24T12:11:45.446799+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:11:45.446799+0000","last_undegraded":"2024-03-24T12:11:45.446799+0000","last_fullsized":"2024-03-24T12:11:45.446799+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:17.566562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:10:17.566562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"241'15","reported_seq":623,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440202+0000","last_change":"2024-03-24T12:11:28.054589+0000","last_active":"2024-03-24T12:11:45.440202+0000","last_peered":"2024-03-24T12:11:45.440202+0000","last_clean":"2024-03-24T12:11:45.440202+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:11:45.440202+0000","last_undegraded":"2024-03-24T12:11:45.440202+0000","last_fullsized":"2024-03-24T12:11:45.440202+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:11:28.054480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.054480+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.923586+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":835,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440171+0000","last_change":"2024-03-24T12:09:37.716173+0000","last_active":"2024-03-24T12:11:45.440171+0000","last_peered":"2024-03-24T12:11:45.440171+0000","last_clean":"2024-03-24T12:11:45.440171+0000","last_became_active":"2024-03-24T12:08:36.152805+0000","last_became_peered":"2024-03-24T12:08:36.152805+0000","last_unstale":"2024-03-24T12:11:45.440171+0000","last_undegraded":"2024-03-24T12:11:45.440171+0000","last_fullsized":"2024-03-24T12:11:45.440171+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:37.716096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:09:37.716096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":821,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990860+0000","last_change":"2024-03-24T12:11:37.185727+0000","last_active":"2024-03-24T12:11:46.990860+0000","last_peered":"2024-03-24T12:11:46.990860+0000","last_clean":"2024-03-24T12:11:46.990860+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:11:46.990860+0000","last_undegraded":"2024-03-24T12:11:46.990860+0000","last_fullsized":"2024-03-24T12:11:46.990860+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:37.185629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.185629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:37.902893+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":826,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990911+0000","last_change":"2024-03-24T12:11:31.446015+0000","last_active":"2024-03-24T12:11:46.990911+0000","last_peered":"2024-03-24T12:11:46.990911+0000","last_clean":"2024-03-24T12:11:46.990911+0000","last_became_active":"2024-03-24T12:08:36.146995+0000","last_became_peered":"2024-03-24T12:08:36.146995+0000","last_unstale":"2024-03-24T12:11:46.990911+0000","last_undegraded":"2024-03-24T12:11:46.990911+0000","last_fullsized":"2024-03-24T12:11:46.990911+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:31.445925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:30.836662+0000","last_clean_scrub_stamp":"2024-03-24T12:11:31.445925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.589782+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":845,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.445990+0000","last_change":"2024-03-24T12:11:28.574992+0000","last_active":"2024-03-24T12:11:45.445990+0000","last_peered":"2024-03-24T12:11:45.445990+0000","last_clean":"2024-03-24T12:11:45.445990+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:11:45.445990+0000","last_undegraded":"2024-03-24T12:11:45.445990+0000","last_fullsized":"2024-03-24T12:11:45.445990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:28.574841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.574841+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:40.417840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":855,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.019119+0000","last_change":"2024-03-24T12:10:33.780424+0000","last_active":"2024-03-24T12:11:47.019119+0000","last_peered":"2024-03-24T12:11:47.019119+0000","last_clean":"2024-03-24T12:11:47.019119+0000","last_became_active":"2024-03-24T12:08:36.174093+0000","last_became_peered":"2024-03-24T12:08:36.174093+0000","last_unstale":"2024-03-24T12:11:47.019119+0000","last_undegraded":"2024-03-24T12:11:47.019119+0000","last_fullsized":"2024-03-24T12:11:47.019119+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:33.780343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:33.780343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.871663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":459,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.004933+0000","last_change":"2024-03-24T12:09:38.661845+0000","last_active":"2024-03-24T12:11:47.004933+0000","last_peered":"2024-03-24T12:11:47.004933+0000","last_clean":"2024-03-24T12:11:47.004933+0000","last_became_active":"2024-03-24T12:08:36.166281+0000","last_became_peered":"2024-03-24T12:08:36.166281+0000","last_unstale":"2024-03-24T12:11:47.004933+0000","last_undegraded":"2024-03-24T12:11:47.004933+0000","last_fullsized":"2024-03-24T12:11:47.004933+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.661755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.661755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":812,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991168+0000","last_change":"2024-03-24T12:10:10.700383+0000","last_active":"2024-03-24T12:11:46.991168+0000","last_peered":"2024-03-24T12:11:46.991168+0000","last_clean":"2024-03-24T12:11:46.991168+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:11:46.991168+0000","last_undegraded":"2024-03-24T12:11:46.991168+0000","last_fullsized":"2024-03-24T12:11:46.991168+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:10.700237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.700237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":833,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991165+0000","last_change":"2024-03-24T12:07:56.051237+0000","last_active":"2024-03-24T12:11:46.991165+0000","last_peered":"2024-03-24T12:11:46.991165+0000","last_clean":"2024-03-24T12:11:46.991165+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:11:46.991165+0000","last_undegraded":"2024-03-24T12:11:46.991165+0000","last_fullsized":"2024-03-24T12:11:46.991165+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:07:56.051061+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.051061+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":721,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.006677+0000","last_change":"2024-03-24T12:10:12.784984+0000","last_active":"2024-03-24T12:11:47.006677+0000","last_peered":"2024-03-24T12:11:47.006677+0000","last_clean":"2024-03-24T12:11:47.006677+0000","last_became_active":"2024-03-24T12:08:36.166947+0000","last_became_peered":"2024-03-24T12:08:36.166947+0000","last_unstale":"2024-03-24T12:11:47.006677+0000","last_undegraded":"2024-03-24T12:11:47.006677+0000","last_fullsized":"2024-03-24T12:11:47.006677+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.784866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.784866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.520122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":826,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442866+0000","last_change":"2024-03-24T12:10:37.907768+0000","last_active":"2024-03-24T12:11:45.442866+0000","last_peered":"2024-03-24T12:11:45.442866+0000","last_clean":"2024-03-24T12:11:45.442866+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:11:45.442866+0000","last_undegraded":"2024-03-24T12:11:45.442866+0000","last_fullsized":"2024-03-24T12:11:45.442866+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:37.907635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:37.907635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:47.259474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":824,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442533+0000","last_change":"2024-03-24T12:10:35.026578+0000","last_active":"2024-03-24T12:11:45.442533+0000","last_peered":"2024-03-24T12:11:45.442533+0000","last_clean":"2024-03-24T12:11:45.442533+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:11:45.442533+0000","last_undegraded":"2024-03-24T12:11:45.442533+0000","last_fullsized":"2024-03-24T12:11:45.442533+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:35.026455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.026455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":842,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991835+0000","last_change":"2024-03-24T12:10:22.696939+0000","last_active":"2024-03-24T12:11:46.991835+0000","last_peered":"2024-03-24T12:11:46.991835+0000","last_clean":"2024-03-24T12:11:46.991835+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:11:46.991835+0000","last_undegraded":"2024-03-24T12:11:46.991835+0000","last_fullsized":"2024-03-24T12:11:46.991835+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:10:22.696749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:22.696749+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":833,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440359+0000","last_change":"2024-03-24T12:09:38.737527+0000","last_active":"2024-03-24T12:11:45.440359+0000","last_peered":"2024-03-24T12:11:45.440359+0000","last_clean":"2024-03-24T12:11:45.440359+0000","last_became_active":"2024-03-24T12:08:36.157290+0000","last_became_peered":"2024-03-24T12:08:36.157290+0000","last_unstale":"2024-03-24T12:11:45.440359+0000","last_undegraded":"2024-03-24T12:11:45.440359+0000","last_fullsized":"2024-03-24T12:11:45.440359+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.737417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":836,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442007+0000","last_change":"2024-03-24T12:11:45.406146+0000","last_active":"2024-03-24T12:11:45.442007+0000","last_peered":"2024-03-24T12:11:45.442007+0000","last_clean":"2024-03-24T12:11:45.442007+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:11:45.442007+0000","last_undegraded":"2024-03-24T12:11:45.442007+0000","last_fullsized":"2024-03-24T12:11:45.442007+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:45.406037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.406037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.099159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":803,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991058+0000","last_change":"2024-03-24T12:10:13.863177+0000","last_active":"2024-03-24T12:11:46.991058+0000","last_peered":"2024-03-24T12:11:46.991058+0000","last_clean":"2024-03-24T12:11:46.991058+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:11:46.991058+0000","last_undegraded":"2024-03-24T12:11:46.991058+0000","last_fullsized":"2024-03-24T12:11:46.991058+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.862457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.862457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"241'25","reported_seq":861,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992498+0000","last_change":"2024-03-24T12:10:35.061280+0000","last_active":"2024-03-24T12:11:46.992498+0000","last_peered":"2024-03-24T12:11:46.992498+0000","last_clean":"2024-03-24T12:11:46.992498+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:11:46.992498+0000","last_undegraded":"2024-03-24T12:11:46.992498+0000","last_fullsized":"2024-03-24T12:11:46.992498+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:35.061151+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.061151+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.815059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":673,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.005405+0000","last_change":"2024-03-24T12:10:24.690361+0000","last_active":"2024-03-24T12:11:47.005405+0000","last_peered":"2024-03-24T12:11:47.005405+0000","last_clean":"2024-03-24T12:11:47.005405+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:11:47.005405+0000","last_undegraded":"2024-03-24T12:11:47.005405+0000","last_fullsized":"2024-03-24T12:11:47.005405+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.690274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.690274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.520065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":1074,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440571+0000","last_change":"2024-03-24T12:11:30.060710+0000","last_active":"2024-03-24T12:11:45.440571+0000","last_peered":"2024-03-24T12:11:45.440571+0000","last_clean":"2024-03-24T12:11:45.440571+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:11:45.440571+0000","last_undegraded":"2024-03-24T12:11:45.440571+0000","last_fullsized":"2024-03-24T12:11:45.440571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:11:30.060478+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:11:30.060478+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.661103+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":842,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.443452+0000","last_change":"2024-03-24T12:10:27.009973+0000","last_active":"2024-03-24T12:11:45.443452+0000","last_peered":"2024-03-24T12:11:45.443452+0000","last_clean":"2024-03-24T12:11:45.443452+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:11:45.443452+0000","last_undegraded":"2024-03-24T12:11:45.443452+0000","last_fullsized":"2024-03-24T12:11:45.443452+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:10:27.009877+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:27.009877+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:48.532733+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":871,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440674+0000","last_change":"2024-03-24T12:10:15.799447+0000","last_active":"2024-03-24T12:11:45.440674+0000","last_peered":"2024-03-24T12:11:45.440674+0000","last_clean":"2024-03-24T12:11:45.440674+0000","last_became_active":"2024-03-24T12:08:36.156563+0000","last_became_peered":"2024-03-24T12:08:36.156563+0000","last_unstale":"2024-03-24T12:11:45.440674+0000","last_undegraded":"2024-03-24T12:11:45.440674+0000","last_fullsized":"2024-03-24T12:11:45.440674+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:10:15.799348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:15.799348+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":816,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991595+0000","last_change":"2024-03-24T12:10:23.652133+0000","last_active":"2024-03-24T12:11:46.991595+0000","last_peered":"2024-03-24T12:11:46.991595+0000","last_clean":"2024-03-24T12:11:46.991595+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:11:46.991595+0000","last_undegraded":"2024-03-24T12:11:46.991595+0000","last_fullsized":"2024-03-24T12:11:46.991595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:10:23.652045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.652045+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":828,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992598+0000","last_change":"2024-03-24T12:10:17.070842+0000","last_active":"2024-03-24T12:11:46.992598+0000","last_peered":"2024-03-24T12:11:46.992598+0000","last_clean":"2024-03-24T12:11:46.992598+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:11:46.992598+0000","last_undegraded":"2024-03-24T12:11:46.992598+0000","last_fullsized":"2024-03-24T12:11:46.992598+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:17.070732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:10:17.070732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:50.445360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":421,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.007377+0000","last_change":"2024-03-24T12:11:15.598297+0000","last_active":"2024-03-24T12:11:47.007377+0000","last_peered":"2024-03-24T12:11:47.007377+0000","last_clean":"2024-03-24T12:11:47.007377+0000","last_became_active":"2024-03-24T12:08:36.166973+0000","last_became_peered":"2024-03-24T12:08:36.166973+0000","last_unstale":"2024-03-24T12:11:47.007377+0000","last_undegraded":"2024-03-24T12:11:47.007377+0000","last_fullsized":"2024-03-24T12:11:47.007377+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:15.598041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:11:15.598041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.713677+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"241'34","reported_seq":773,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991020+0000","last_change":"2024-03-24T12:11:35.124433+0000","last_active":"2024-03-24T12:11:46.991020+0000","last_peered":"2024-03-24T12:11:46.991020+0000","last_clean":"2024-03-24T12:11:46.991020+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:11:46.991020+0000","last_undegraded":"2024-03-24T12:11:46.991020+0000","last_fullsized":"2024-03-24T12:11:46.991020+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:11:35.124291+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.124291+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.181527+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":833,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.440450+0000","last_change":"2024-03-24T12:11:15.939612+0000","last_active":"2024-03-24T12:11:45.440450+0000","last_peered":"2024-03-24T12:11:45.440450+0000","last_clean":"2024-03-24T12:11:45.440450+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:11:45.440450+0000","last_undegraded":"2024-03-24T12:11:45.440450+0000","last_fullsized":"2024-03-24T12:11:45.440450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:15.939499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:11:15.939499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:35.965808+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":570,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446747+0000","last_change":"2024-03-24T12:07:53.354298+0000","last_active":"2024-03-24T12:11:45.446747+0000","last_peered":"2024-03-24T12:11:45.446747+0000","last_clean":"2024-03-24T12:11:45.446747+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:11:45.446747+0000","last_undegraded":"2024-03-24T12:11:45.446747+0000","last_fullsized":"2024-03-24T12:11:45.446747+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:07:53.354213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":819,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446851+0000","last_change":"2024-03-24T12:10:13.573691+0000","last_active":"2024-03-24T12:11:45.446851+0000","last_peered":"2024-03-24T12:11:45.446851+0000","last_clean":"2024-03-24T12:11:45.446851+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:11:45.446851+0000","last_undegraded":"2024-03-24T12:11:45.446851+0000","last_fullsized":"2024-03-24T12:11:45.446851+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.573504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.573504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":873,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991527+0000","last_change":"2024-03-24T12:11:41.236799+0000","last_active":"2024-03-24T12:11:46.991527+0000","last_peered":"2024-03-24T12:11:46.991527+0000","last_clean":"2024-03-24T12:11:46.991527+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:11:46.991527+0000","last_undegraded":"2024-03-24T12:11:46.991527+0000","last_fullsized":"2024-03-24T12:11:46.991527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:11:41.236628+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:11:41.236628+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:56.929372+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":828,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990760+0000","last_change":"2024-03-24T12:11:20.004402+0000","last_active":"2024-03-24T12:11:46.990760+0000","last_peered":"2024-03-24T12:11:46.990760+0000","last_clean":"2024-03-24T12:11:46.990760+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:11:46.990760+0000","last_undegraded":"2024-03-24T12:11:46.990760+0000","last_fullsized":"2024-03-24T12:11:46.990760+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:11:20.004266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.004266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:34.289579+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":821,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446339+0000","last_change":"2024-03-24T12:11:23.602693+0000","last_active":"2024-03-24T12:11:45.446339+0000","last_peered":"2024-03-24T12:11:45.446339+0000","last_clean":"2024-03-24T12:11:45.446339+0000","last_became_active":"2024-03-24T12:11:16.022628+0000","last_became_peered":"2024-03-24T12:11:16.022628+0000","last_unstale":"2024-03-24T12:11:45.446339+0000","last_undegraded":"2024-03-24T12:11:45.446339+0000","last_fullsized":"2024-03-24T12:11:45.446339+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:23.602435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.602435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:35.820551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":813,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991793+0000","last_change":"2024-03-24T12:08:39.416538+0000","last_active":"2024-03-24T12:11:46.991793+0000","last_peered":"2024-03-24T12:11:46.991793+0000","last_clean":"2024-03-24T12:11:46.991793+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:11:46.991793+0000","last_undegraded":"2024-03-24T12:11:46.991793+0000","last_fullsized":"2024-03-24T12:11:46.991793+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:39.416426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:39.416426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"241'15","reported_seq":838,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446074+0000","last_change":"2024-03-24T12:07:40.191735+0000","last_active":"2024-03-24T12:11:45.446074+0000","last_peered":"2024-03-24T12:11:45.446074+0000","last_clean":"2024-03-24T12:11:45.446074+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:11:45.446074+0000","last_undegraded":"2024-03-24T12:11:45.446074+0000","last_fullsized":"2024-03-24T12:11:45.446074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:07:40.191513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:07:40.191513+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":855,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442856+0000","last_change":"2024-03-24T12:11:41.354046+0000","last_active":"2024-03-24T12:11:45.442856+0000","last_peered":"2024-03-24T12:11:45.442856+0000","last_clean":"2024-03-24T12:11:45.442856+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:11:45.442856+0000","last_undegraded":"2024-03-24T12:11:45.442856+0000","last_fullsized":"2024-03-24T12:11:45.442856+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:41.353830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:11:41.353830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:55.943062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":734,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.006049+0000","last_change":"2024-03-24T12:10:23.722476+0000","last_active":"2024-03-24T12:11:47.006049+0000","last_peered":"2024-03-24T12:11:47.006049+0000","last_clean":"2024-03-24T12:11:47.006049+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:11:47.006049+0000","last_undegraded":"2024-03-24T12:11:47.006049+0000","last_fullsized":"2024-03-24T12:11:47.006049+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:23.722321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.722321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:48.576677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":809,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991479+0000","last_change":"2024-03-24T12:10:20.093443+0000","last_active":"2024-03-24T12:11:46.991479+0000","last_peered":"2024-03-24T12:11:46.991479+0000","last_clean":"2024-03-24T12:11:46.991479+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:11:46.991479+0000","last_undegraded":"2024-03-24T12:11:46.991479+0000","last_fullsized":"2024-03-24T12:11:46.991479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:20.093338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:10:20.093338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.491397+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"241'9","reported_seq":836,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.446084+0000","last_change":"2024-03-24T12:11:36.552488+0000","last_active":"2024-03-24T12:11:45.446084+0000","last_peered":"2024-03-24T12:11:45.446084+0000","last_clean":"2024-03-24T12:11:45.446084+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:11:45.446084+0000","last_undegraded":"2024-03-24T12:11:45.446084+0000","last_fullsized":"2024-03-24T12:11:45.446084+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:11:36.552333+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:11:36.552333+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:01.517180+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":851,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442956+0000","last_change":"2024-03-24T12:11:13.237903+0000","last_active":"2024-03-24T12:11:45.442956+0000","last_peered":"2024-03-24T12:11:45.442956+0000","last_clean":"2024-03-24T12:11:45.442956+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:11:45.442956+0000","last_undegraded":"2024-03-24T12:11:45.442956+0000","last_fullsized":"2024-03-24T12:11:45.442956+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:13.237829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.237829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.385120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":843,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442299+0000","last_change":"2024-03-24T12:11:23.134759+0000","last_active":"2024-03-24T12:11:45.442299+0000","last_peered":"2024-03-24T12:11:45.442299+0000","last_clean":"2024-03-24T12:11:45.442299+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:11:45.442299+0000","last_undegraded":"2024-03-24T12:11:45.442299+0000","last_fullsized":"2024-03-24T12:11:45.442299+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:23.134643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.011992+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.134643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:29.103157+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":740,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:47.005060+0000","last_change":"2024-03-24T12:10:13.888112+0000","last_active":"2024-03-24T12:11:47.005060+0000","last_peered":"2024-03-24T12:11:47.005060+0000","last_clean":"2024-03-24T12:11:47.005060+0000","last_became_active":"2024-03-24T12:08:36.164166+0000","last_became_peered":"2024-03-24T12:08:36.164166+0000","last_unstale":"2024-03-24T12:11:47.005060+0000","last_undegraded":"2024-03-24T12:11:47.005060+0000","last_fullsized":"2024-03-24T12:11:47.005060+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.888002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.888002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:51.519879+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"241'19","reported_seq":851,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.445679+0000","last_change":"2024-03-24T12:11:44.652906+0000","last_active":"2024-03-24T12:11:45.445679+0000","last_peered":"2024-03-24T12:11:45.445679+0000","last_clean":"2024-03-24T12:11:45.445679+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:11:45.445679+0000","last_undegraded":"2024-03-24T12:11:45.445679+0000","last_fullsized":"2024-03-24T12:11:45.445679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:44.652731+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:11:44.652731+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:13.964162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":825,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.992009+0000","last_change":"2024-03-24T12:08:40.183479+0000","last_active":"2024-03-24T12:11:46.992009+0000","last_peered":"2024-03-24T12:11:46.992009+0000","last_clean":"2024-03-24T12:11:46.992009+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:11:46.992009+0000","last_undegraded":"2024-03-24T12:11:46.992009+0000","last_fullsized":"2024-03-24T12:11:46.992009+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.183311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.183311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":300,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.442633+0000","last_change":"2024-03-24T12:10:24.707449+0000","last_active":"2024-03-24T12:11:45.442633+0000","last_peered":"2024-03-24T12:11:45.442633+0000","last_clean":"2024-03-24T12:11:45.442633+0000","last_became_active":"2024-03-24T12:08:36.152717+0000","last_became_peered":"2024-03-24T12:08:36.152717+0000","last_unstale":"2024-03-24T12:11:45.442633+0000","last_undegraded":"2024-03-24T12:11:45.442633+0000","last_fullsized":"2024-03-24T12:11:45.442633+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.707367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.707367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":814,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-24T12:11:45.445978+0000","last_change":"2024-03-24T12:10:12.547927+0000","last_active":"2024-03-24T12:11:45.445978+0000","last_peered":"2024-03-24T12:11:45.445978+0000","last_clean":"2024-03-24T12:11:45.445978+0000","last_became_active":"2024-03-24T12:08:36.165334+0000","last_became_peered":"2024-03-24T12:08:36.165334+0000","last_unstale":"2024-03-24T12:11:45.445978+0000","last_undegraded":"2024-03-24T12:11:45.445978+0000","last_fullsized":"2024-03-24T12:11:45.445978+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.547803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.547803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"242'34","reported_seq":892,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991948+0000","last_change":"2024-03-24T12:08:47.101220+0000","last_active":"2024-03-24T12:11:46.991948+0000","last_peered":"2024-03-24T12:11:46.991948+0000","last_clean":"2024-03-24T12:11:46.991948+0000","last_became_active":"2024-03-24T12:08:36.154117+0000","last_became_peered":"2024-03-24T12:08:36.154117+0000","last_unstale":"2024-03-24T12:11:46.991948+0000","last_undegraded":"2024-03-24T12:11:46.991948+0000","last_fullsized":"2024-03-24T12:11:46.991948+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:47.101082+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:08:47.101082+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":822,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.991813+0000","last_change":"2024-03-24T12:10:24.957781+0000","last_active":"2024-03-24T12:11:46.991813+0000","last_peered":"2024-03-24T12:11:46.991813+0000","last_clean":"2024-03-24T12:11:46.991813+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:11:46.991813+0000","last_undegraded":"2024-03-24T12:11:46.991813+0000","last_fullsized":"2024-03-24T12:11:46.991813+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.957699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.957699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:50.445308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":285,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990908+0000","last_change":"2024-03-24T12:08:33.256620+0000","last_active":"2024-03-24T12:11:46.990908+0000","last_peered":"2024-03-24T12:11:46.990908+0000","last_clean":"2024-03-24T12:11:46.990908+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:11:46.990908+0000","last_undegraded":"2024-03-24T12:11:46.990908+0000","last_fullsized":"2024-03-24T12:11:46.990908+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:33.255322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:08:33.255322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":810,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-24T12:11:46.990336+0000","last_change":"2024-03-24T12:11:44.456204+0000","last_active":"2024-03-24T12:11:46.990336+0000","last_peered":"2024-03-24T12:11:46.990336+0000","last_clean":"2024-03-24T12:11:46.990336+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:11:46.990336+0000","last_undegraded":"2024-03-24T12:11:46.990336+0000","last_fullsized":"2024-03-24T12:11:46.990336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:44.455779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:44.455779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:53.752100+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6273302528,"num_objects":95723,"num_object_clones":0,"num_object_copies":287169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95723,"num_write_kb":6126272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18627624960,"data_stored":18627624960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74623,"ondisk_log_size":74623,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":390,"ondisk_log_size":390,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":368,"num_read_kb":363,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":792,"ondisk_log_size":792,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005294,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2962544,"kb_used_data":2255268,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":90782608,"statfs":{"total":95995035648,"available":92961390592,"internally_reserved":0,"allocated":2309394432,"data_stored":2308512599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":724236746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332341,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3673408,"kb_used_data":2966132,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":90071744,"statfs":{"total":95995035648,"available":92233465856,"internally_reserved":0,"allocated":3037319168,"data_stored":3036421237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":724236742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575117,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4554932,"kb_used_data":3747692,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":89190220,"statfs":{"total":95995035648,"available":91330785280,"internally_reserved":0,"allocated":3837636608,"data_stored":3836745980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738643,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3148992,"kb_used_data":2287608,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":90596160,"statfs":{"total":95995035648,"available":92770467840,"internally_reserved":0,"allocated":2342510592,"data_stored":2341614708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411593,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346340,"kb_used_data":747348,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92398812,"statfs":{"total":95995035648,"available":94616383488,"internally_reserved":0,"allocated":765284352,"data_stored":764425237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117252,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3610388,"kb_used_data":2758732,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":90134764,"statfs":{"total":95995035648,"available":92297998336,"internally_reserved":0,"allocated":2824941568,"data_stored":2824026629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":872085959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921022,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2554684,"kb_used_data":1841840,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91190468,"statfs":{"total":95995035648,"available":93379039232,"internally_reserved":0,"allocated":1886044160,"data_stored":1885149471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455528,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1845640,"kb_used_data":1520256,"kb_used_omap":7,"kb_used_meta":325368,"kb_avail":91899512,"statfs":{"total":95995035648,"available":94105100288,"internally_reserved":0,"allocated":1556742144,"data_stored":1555838913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":333176944},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2349858816,"data_stored":2349858816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3845390336,"data_stored":3845390336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":764674048,"data_stored":764674048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2831876096,"data_stored":2831876096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1884422144,"data_stored":1884422144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1569652736,"data_stored":1569652736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3064725504,"data_stored":3064725504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2317025280,"data_stored":2317025280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:11:55.886 INFO:tasks.thrashosds.thrasher:Setting 6.b to [2, 0, 4, 5, 1, 7] 2024-03-24T12:11:55.886 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.b', '2', '0', '4', '5', '1', '7'] 2024-03-24T12:11:55.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 6.b 2 0 4 5 1 7 2024-03-24T12:11:55.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:55.931 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:55.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:56.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:56.070 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:56.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:56.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:56.239 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:56.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:56.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:56.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 66 16 104569 104553 98.9856 73.6875 0.00842346 0.0100981 2024-03-24T12:11:56.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:56.378 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:56.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:56.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:56.513 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:56.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:56.615 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:56.654 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:56.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:56.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:56.794 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:56.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:56.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:56.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:57.032 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:57.068 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:57.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:57.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:57.077 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:11:57.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:57.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:57.207 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:57.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:57.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:57.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 67 16 105747 105731 98.6071 73.625 0.0129077 0.0101363 2024-03-24T12:11:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:57 smithi193 ceph-mon[33593]: pgmap v933: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-24T12:11:57.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:57 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2304766546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:57.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:57 smithi064 ceph-mon[30223]: pgmap v933: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-24T12:11:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:57 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2304766546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:57.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:57 smithi064 ceph-mon[42243]: pgmap v933: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-24T12:11:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:57 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2304766546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:11:57.447 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:57.481 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:57.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:57.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:57.619 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:57.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:57.721 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:57.759 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:57.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:57.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:11:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:11:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:57.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:57.890 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:57.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:57.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:58.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:58.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:11:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:58.170 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:58.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:11:58.271 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:58.297 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:58.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:58.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 68 16 106957 106941 98.269 75.625 0.00781578 0.0101714 2024-03-24T12:11:58.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:11:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:11:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:58 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:58 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2304766546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:11:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:58 smithi193 ceph-mon[33593]: osdmap e442: 8 total, 8 up, 8 in 2024-03-24T12:11:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:58.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:58.434 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:58.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2304766546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[30223]: osdmap e442: 8 total, 8 up, 8 in 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2304766546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[42243]: osdmap e442: 8 total, 8 up, 8 in 2024-03-24T12:11:58.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:11:58.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:58.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:58.575 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:58.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:58.677 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:11:58.709 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:58.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:11:58.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:58.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:58.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:11:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:58.996 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:58.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:11:59.097 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:59.127 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:59.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:59.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:11:59.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:59.265 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:59.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:11:59.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 69 16 108094 108078 97.8747 71.0625 0.0064973 0.0102119 2024-03-24T12:11:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:59 smithi193 ceph-mon[33593]: pgmap v935: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:11:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:11:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:59.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:11:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:59.394 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:59.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:11:59.495 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:59.534 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:59.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:59.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:59 smithi064 ceph-mon[30223]: pgmap v935: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:11:59.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:11:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:59.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:59 smithi064 ceph-mon[42243]: pgmap v935: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:11:59.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:11:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:59.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:11:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:11:59.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:11:59.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:11:59.815 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:59.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:11:59.917 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:11:59.952 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:11:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:59.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:00.054 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:00.091 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:00.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:00.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:00.237 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:00.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:00.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 70 16 109207 109191 97.4702 69.5625 0.00632984 0.0102549 2024-03-24T12:12:00.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:00.368 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:00.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:00.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:00.504 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:00.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:00.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:00.605 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:00.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:12:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:00.639 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:00.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:00.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:00.773 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:00.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:00.874 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:00.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:01.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:01.043 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:01.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:01.145 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:01.177 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:01.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:01.278 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:01.313 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:01.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:01.332 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 71 16 110369 110353 97.1202 72.625 0.0181354 0.0102919 2024-03-24T12:12:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:01 smithi193 ceph-mon[33593]: pgmap v936: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T12:12:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:01 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:12:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:01 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:12:01.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:01 smithi193 ceph-mon[33593]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T12:12:01.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:01.460 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:01.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:01 smithi064 ceph-mon[30223]: pgmap v936: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T12:12:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:01 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:12:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:01 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:12:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:01 smithi064 ceph-mon[30223]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T12:12:01.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:01 smithi064 ceph-mon[42243]: pgmap v936: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T12:12:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:01 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:12:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:01 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:12:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:01 smithi064 ceph-mon[42243]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T12:12:01.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:01.603 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:01.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:01.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:01.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:01.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:01.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:01.864 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:01.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:01.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:02.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:02.034 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:02.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:02.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:02.166 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:02.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:02.168 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:02.182 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:02.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:02.318 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:02.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:02.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 72 16 111484 111468 96.7391 69.6875 0.00820454 0.0103325 2024-03-24T12:12:02.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:02.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:02.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:02.613 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:02.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:02.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:02.764 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:02.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:02.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:02.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:02.952 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:12:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:12:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:12:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:12:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216862 9.2 9:4d09947b:::benchmark_data_smithi064_148274_object91049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.21426027699999, 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0907798720000002, 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216862 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735224+0000", 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735226+0000", 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.272e-06 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735233+0000", 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.494e-06 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735246+0000", 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.277e-05 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.735253+0000", 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7889999999999995e-06 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.744476+0000", 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092227190000000007 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.744486+0000", 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5070000000000006e-06 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722078+0000", 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97759208500000006 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722094+0000", 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6337999999999998e-05 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818237+0000", 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096142892 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818340+0000", 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010314099999999999 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.818710+0000", 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036944499999999998 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821730+0000", 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00302075 2024-03-24T12:12:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825432+0000", 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037018089999999999 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825705+0000", 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027328399999999999 2024-03-24T12:12:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825805+0000", 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9975000000000004e-05 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825875+0000", 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9665999999999993e-05 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826004+0000", 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012893599999999999 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216877 9.2 9:41202982:::benchmark_data_smithi064_148274_object91064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.19244611900001, 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0695391550000002, 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216877 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757038+0000", 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757041+0000", 2024-03-24T12:12:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4339999999999999e-06 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757048+0000", 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6369999999999999e-06 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757062+0000", 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.359e-05 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757072+0000", 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0125999999999999e-05 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757145+0000", 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2819999999999997e-05 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.757158+0000", 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2721999999999999e-05 2024-03-24T12:12:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722171+0000", 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96501314900000001 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722181+0000", 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0164e-05 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821770+0000", 2024-03-24T12:12:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0995892380000001 2024-03-24T12:12:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821855+0000", 2024-03-24T12:12:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4513999999999998e-05 2024-03-24T12:12:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.821947+0000", 2024-03-24T12:12:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2659999999999997e-05 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825725+0000", 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037774430000000001 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.825754+0000", 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9048e-05 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826079+0000", 2024-03-24T12:12:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000325015 2024-03-24T12:12:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826391+0000", 2024-03-24T12:12:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031238199999999999 2024-03-24T12:12:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826454+0000", 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2303000000000003e-05 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826577+0000", 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012391 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216879 9.2 9:4db289b8:::benchmark_data_smithi064_148274_object91066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.19091830500003, 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0685427669999998, 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216879 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758566+0000", 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758568+0000", 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9309999999999998e-06 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758574+0000", 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000002e-06 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758586+0000", 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2075999999999999e-05 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758595+0000", 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5930000000000006e-06 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758682+0000", 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7328000000000001e-05 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.758694+0000", 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2271e-05 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722222+0000", 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.963528159 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722232+0000", 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2650000000000002e-06 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824044+0000", 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1018119829999999 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824123+0000", 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9806000000000001e-05 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.824206+0000", 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2086e-05 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826021+0000", 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001815126 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826703+0000", 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068264900000000002 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826730+0000", 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6710999999999999e-05 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826787+0000", 2024-03-24T12:12:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7014e-05 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.826892+0000", 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105351 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827109+0000", 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021650799999999999 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216888 9.2 9:4c303ed8:::benchmark_data_smithi064_148274_object91075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.18867853099999, 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.079817389, 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216888 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760806+0000", 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760808+0000", 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9139999999999998e-06 2024-03-24T12:12:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760813+0000", 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6699999999999999e-06 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760826+0000", 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2533e-05 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760835+0000", 2024-03-24T12:12:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3319999999999998e-06 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760923+0000", 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7826999999999998e-05 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.760940+0000", 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6396999999999999e-05 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722255+0000", 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96131583300000001 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722263+0000", 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0669999999999992e-06 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827147+0000", 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.104883244 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827325+0000", 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001784 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.827519+0000", 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019378400000000001 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835233+0000", 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077136690000000003 2024-03-24T12:12:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835278+0000", 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5664000000000002e-05 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835321+0000", 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2817e-05 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840424+0000", 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051024970000000001 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840504+0000", 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0844e-05 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840623+0000", 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118897 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216890 9.2 9:4575dfdc:::benchmark_data_smithi064_148274_object91077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.18538422099999, 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0768427539999998, 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216890 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764100+0000", 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764103+0000", 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4870000000000001e-06 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764110+0000", 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2609999999999996e-06 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764123+0000", 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3369e-05 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764133+0000", 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8179999999999999e-06 2024-03-24T12:12:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764206+0000", 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2471999999999993e-05 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.764244+0000", 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8655000000000002e-05 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722287+0000", 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95804261999999996 2024-03-24T12:12:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722295+0000", 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6129999999999998e-06 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830293+0000", 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079985539999999 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830416+0000", 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012251699999999999 2024-03-24T12:12:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.830550+0000", 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013488000000000001 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835368+0000", 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048179049999999999 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840637+0000", 2024-03-24T12:12:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005268928 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840707+0000", 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9367000000000002e-05 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840755+0000", 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8789000000000003e-05 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840813+0000", 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7757999999999998e-05 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840943+0000", 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012976099999999999 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216938 9.2 9:41fc8627:::benchmark_data_smithi064_148274_object91125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:12:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.141513734, 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.033607655, 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216938 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807971+0000", 2024-03-24T12:12:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807972+0000", 2024-03-24T12:12:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-03-24T12:12:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807977+0000", 2024-03-24T12:12:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8450000000000002e-06 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807990+0000", 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3091999999999999e-05 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.807998+0000", 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6769999999999995e-06 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.808072+0000", 2024-03-24T12:12:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3719000000000003e-05 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.808082+0000", 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0648999999999999e-05 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722352+0000", 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.9142692 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722361+0000", 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7030000000000001e-06 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832521+0000", 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.110159882 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832692+0000", 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171242 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.832843+0000", 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015096200000000001 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840408+0000", 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075646669999999997 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840644+0000", 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023627500000000001 2024-03-24T12:12:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840991+0000", 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034632899999999998 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841121+0000", 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130445 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841180+0000", 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8777999999999999e-05 2024-03-24T12:12:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841578+0000", 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039849699999999999 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216940 9.2 9:46d14649:::benchmark_data_smithi064_148274_object91127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.13995012499998, 2024-03-24T12:12:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.032468546, 2024-03-24T12:12:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216940 2024-03-24T12:12:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809534+0000", 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809536+0000", 2024-03-24T12:12:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.57e-06 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809541+0000", 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0470000000000001e-06 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809555+0000", 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.417e-05 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809563+0000", 2024-03-24T12:12:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.622e-06 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809619+0000", 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.643e-05 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.809631+0000", 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1477e-05 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722387+0000", 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.91275614100000002 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722395+0000", 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4279999999999996e-06 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835404+0000", 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1130083449999999 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835526+0000", 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122069 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.835639+0000", 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113329 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.840651+0000", 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050119439999999999 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841055+0000", 2024-03-24T12:12:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000404127 2024-03-24T12:12:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841634+0000", 2024-03-24T12:12:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057911699999999998 2024-03-24T12:12:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841735+0000", 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100603 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841807+0000", 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1997999999999997e-05 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842003+0000", 2024-03-24T12:12:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019612899999999999 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216945 9.2 9:46734ef5:::benchmark_data_smithi064_148274_object91132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.13439773099998, 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.027242598, 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216945 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815087+0000", 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815089+0000", 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5110000000000002e-06 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815098+0000", 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815115+0000", 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833999999999998e-05 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815125+0000", 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7019999999999996e-06 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815205+0000", 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0593999999999994e-05 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.815214+0000", 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7630000000000006e-06 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722420+0000", 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.90720568000000001 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722428+0000", 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3610000000000001e-06 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837728+0000", 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.115299724 2024-03-24T12:12:03.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837849+0000", 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121298 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.837993+0000", 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014408300000000001 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841007+0000", 2024-03-24T12:12:03.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030141429999999999 2024-03-24T12:12:03.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.841682+0000", 2024-03-24T12:12:03.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067442900000000004 2024-03-24T12:12:03.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842071+0000", 2024-03-24T12:12:03.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038890700000000002 2024-03-24T12:12:03.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842145+0000", 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4214999999999993e-05 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842218+0000", 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3461999999999996e-05 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842329+0000", 2024-03-24T12:12:03.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110983 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216966 9.2 9:44cdb63b:::benchmark_data_smithi064_148274_object91153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.09686041399999, 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0008592350000001, 2024-03-24T12:12:03.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216966 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852624+0000", 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852626+0000", 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852629+0000", 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3390000000000001e-06 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852638+0000", 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5720000000000003e-06 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852643+0000", 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5369999999999997e-06 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852719+0000", 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6086000000000003e-05 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.852728+0000", 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3000000000000002e-06 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722453+0000", 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86972569 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722461+0000", 2024-03-24T12:12:03.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2830000000000001e-06 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842378+0000", 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1199161879999999 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842492+0000", 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114769 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.842616+0000", 2024-03-24T12:12:03.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012353200000000001 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851293+0000", 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086773370000000002 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851656+0000", 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036241099999999998 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851701+0000", 2024-03-24T12:12:03.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5457999999999997e-05 2024-03-24T12:12:03.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853267+0000", 2024-03-24T12:12:03.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001566095 2024-03-24T12:12:03.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853330+0000", 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2955999999999998e-05 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853483+0000", 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153022 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.035 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.35306.0:216969 9.2 9:4ad9ca10:::benchmark_data_smithi064_148274_object91156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.088085412, 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9923648649999999, 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.35306", 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/1591329942", 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 216969 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:12:03.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861399+0000", 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861400+0000", 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.438e-06 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861404+0000", 2024-03-24T12:12:03.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9469999999999996e-06 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861415+0000", 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0396e-05 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861421+0000", 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2600000000000002e-06 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861479+0000", 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7921000000000003e-05 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:43.861486+0000", 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6359999999999997e-06 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722487+0000", 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86100120400000002 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:44.722495+0000", 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2300000000000008e-06 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.844900+0000", 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1224051230000001 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.845026+0000", 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126062 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,2,7", 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.845364+0000", 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000337809 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851301+0000", 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059373220000000001 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851745+0000", 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044327599999999999 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.851788+0000", 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3272999999999997e-05 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853579+0000", 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001791555 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853644+0000", 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4944999999999996e-05 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:05:45.853764+0000", 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119468 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:116402 10.3 10:c1f1e178:::benchmark_data_smithi064_174908_object116401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.133056+0000", 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.81642816999999, 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14694664800000001, 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116402 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133056+0000", 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133056+0000", 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133059+0000", 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3460000000000001e-06 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133198+0000", 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013925499999999999 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133321+0000", 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012289000000000001 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133330+0000", 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0310000000000004e-06 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133387+0000", 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7463000000000002e-05 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133541+0000", 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015386999999999999 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.133695+0000", 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015424500000000001 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.143037+0000", 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093416969999999995 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.279825+0000", 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.136788142 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.279901+0000", 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5309999999999996e-05 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.280003+0000", 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010239900000000001 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:116409 10.3 10:c7c388dc:::benchmark_data_smithi064_174908_object116408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:07:47.135330+0000", 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.81415394800001, 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14482766799999999, 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116409 2024-03-24T12:12:03.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135330+0000", 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135330+0000", 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135332+0000", 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135383+0000", 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0902999999999997e-05 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135394+0000", 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0686000000000001e-05 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.135401+0000", 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4479999999999997e-06 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.143198+0000", 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077970599999999998 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.143540+0000", 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034143800000000001 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-03-24T12:12:03.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.144004+0000", 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046368700000000001 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.150341+0000", 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063375949999999997 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.280014+0000", 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.129673288 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.280072+0000", 2024-03-24T12:12:03.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7744999999999997e-05 2024-03-24T12:12:03.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:07:47.280158+0000", 2024-03-24T12:12:03.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5958000000000006e-05 2024-03-24T12:12:03.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:176786 10.3 10:c36c4db1:::benchmark_data_smithi064_174908_object32369:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:36.141359+0000", 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 206.80812586600001, 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.021666908, 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176786 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.055 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141359+0000", 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141359+0000", 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141361+0000", 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3269999999999999e-06 2024-03-24T12:12:03.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141368+0000", 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6109999999999999e-06 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141383+0000", 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5525000000000001e-05 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141395+0000", 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2238e-05 2024-03-24T12:12:03.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141640+0000", 2024-03-24T12:12:03.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244532 2024-03-24T12:12:03.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141651+0000", 2024-03-24T12:12:03.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1308000000000001e-05 2024-03-24T12:12:03.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.148627+0000", 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0069758929999999 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.148720+0000", 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3406999999999995e-05 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5", 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.148817+0000", 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6154000000000001e-05 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.152066+0000", 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032490890000000001 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.154076+0000", 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020105990000000001 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.162885+0000", 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088089559999999997 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.162927+0000", 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2169999999999998e-05 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.163025+0000", 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8098999999999997e-05 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.062 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:176800 10.3 10:c90e6f21:::benchmark_data_smithi064_174908_object32383:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e354)", 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:36.141870+0000", 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 206.807614089, 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.021365817, 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176800 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141870+0000", 2024-03-24T12:12:03.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141870+0000", 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141872+0000", 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.485e-06 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141877+0000", 2024-03-24T12:12:03.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0320000000000003e-06 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141888+0000", 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1093e-05 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141896+0000", 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2709999999999992e-06 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141949+0000", 2024-03-24T12:12:03.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3224999999999999e-05 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:36.141960+0000", 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543999999999999e-05 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.150568+0000", 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.008608478 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.150705+0000", 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013608899999999999 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,5", 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.150849+0000", 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014446000000000001 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.152226+0000", 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001376927 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.154185+0000", 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019586059999999999 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.163081+0000", 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088962069999999997 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.163140+0000", 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8943999999999997e-05 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:37.163236+0000", 2024-03-24T12:12:03.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6455999999999999e-05 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:231434 10.5 e369/320 10:a86799ca:::benchmark_data_smithi064_174908_object87017:head v 369'14825, mlcod=369'14823)", 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:09:27.268055+0000", 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.681429847, 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14237074399999999, 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.268055+0000", 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.268055+0000", 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.268057+0000", 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6110000000000001e-06 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.268067+0000", 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.389394+0000", 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121326562 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.389405+0000", 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1371e-05 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.389490+0000", 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4950999999999994e-05 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.389515+0000", 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4666000000000001e-05 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410385+0000", 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020870003000000002 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:09:27.410425+0000", 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0404000000000001e-05 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.16023.0:69830 11.9 11:96955522:::benchmark_data_smithi064_206665_object69829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:11:27.923171+0000", 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.026313651999999, 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18219073799999999, 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.16023", 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 69830 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.923171+0000", 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.923171+0000", 2024-03-24T12:12:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.923172+0000", 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0599999999999999e-07 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.923195+0000", 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3218e-05 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.923202+0000", 2024-03-24T12:12:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.311e-06 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.923209+0000", 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-06 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.928472+0000", 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052636829999999999 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.928606+0000", 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013382100000000001 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,6", 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.928737+0000", 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013130800000000001 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.933579+0000", 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048413609999999998 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.936136+0000", 2024-03-24T12:12:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002556815 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105254+0000", 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16911812800000001 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105317+0000", 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2917999999999996e-05 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105362+0000", 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4959000000000001e-05 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.16023.0:69838 11.9 11:959da8a2:::benchmark_data_smithi064_206665_object69837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:11:27.927184+0000", 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.022300291999997, 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.178341521, 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.16023", 2024-03-24T12:12:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 69838 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.927184+0000", 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.927184+0000", 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.927186+0000", 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9549999999999999e-06 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.927209+0000", 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3096e-05 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.927220+0000", 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1113e-05 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.927228+0000", 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4240000000000004e-06 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931361+0000", 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041333430000000003 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931514+0000", 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015253200000000001 2024-03-24T12:12:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,6", 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931657+0000", 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014315700000000001 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.936140+0000", 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004482766 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.937349+0000", 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001209644 2024-03-24T12:12:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105401+0000", 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16805213799999999 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105489+0000", 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7843000000000006e-05 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105526+0000", 2024-03-24T12:12:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6510000000000001e-05 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.16023.0:69841 11.b 11:d3d49313:::benchmark_data_smithi064_206665_object69840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:11:27.930602+0000", 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.018882611000002, 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17040153199999999, 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.16023", 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 69841 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.930602+0000", 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.930602+0000", 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.930604+0000", 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9929999999999998e-06 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.930626+0000", 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2189e-05 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.930637+0000", 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1389000000000001e-05 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.930645+0000", 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9389999999999997e-06 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.930718+0000", 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2571999999999996e-05 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.930907+0000", 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018931500000000001 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931064+0000", 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000157124 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.933700+0000", 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026357479999999998 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.933932+0000", 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023189099999999999 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.100755+0000", 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.166822567 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.100947+0000", 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192766 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.101003+0000", 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6039000000000003e-05 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.16023.0:69844 11.9 11:9f47f578:::benchmark_data_smithi064_206665_object69843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:11:27.931359+0000", 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.018125394999998, 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174323752, 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.16023", 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 69844 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931359+0000", 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931359+0000", 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931360+0000", 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4309999999999999e-06 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931435+0000", 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4122999999999995e-05 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931453+0000", 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8706999999999999e-05 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.931462+0000", 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3380000000000005e-06 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.933708+0000", 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002245903 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.933843+0000", 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013537299999999999 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,6", 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.933983+0000", 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000139887 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.936208+0000", 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022254050000000002 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.937402+0000", 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011934770000000001 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105581+0000", 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.1681793 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105643+0000", 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2430999999999999e-05 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105683+0000", 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9376999999999999e-05 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.16023.0:69856 11.9 11:9dc30d50:::benchmark_data_smithi064_206665_object69855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:03.096 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:11:27.942146+0000", 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.007338556000001, 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16366518099999999, 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.16023", 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 69856 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942146+0000", 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942146+0000", 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942147+0000", 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2309999999999999e-06 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942168+0000", 2024-03-24T12:12:03.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1202000000000001e-05 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942180+0000", 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1204e-05 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942187+0000", 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3370000000000002e-06 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942242+0000", 2024-03-24T12:12:03.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5345e-05 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942372+0000", 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013007599999999999 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,6", 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.942520+0000", 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014764800000000001 2024-03-24T12:12:03.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.944695+0000", 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021753860000000001 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:27.944970+0000", 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000274472 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105731+0000", 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.160761131 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105777+0000", 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6366999999999998e-05 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:11:28.105811+0000", 2024-03-24T12:12:03.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3782000000000003e-05 2024-03-24T12:12:03.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.103 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:12:03.103 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:12:03.103 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:12:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:03.105 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:03.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:03.106 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:03 smithi064 ceph-mon[42243]: pgmap v938: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T12:12:03.106 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:03 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:12:03.106 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:03 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:12:03.106 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:03 smithi064 ceph-mon[42243]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T12:12:03.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:03.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:03.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 73 16 112660 112644 96.4207 73.5 0.00852089 0.0103662 2024-03-24T12:12:03.345 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:03.383 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:03.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:03.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:03 smithi064 ceph-mon[30223]: pgmap v938: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T12:12:03.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:03 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:12:03.475 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:03 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:12:03.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:03 smithi064 ceph-mon[30223]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T12:12:03.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:03.514 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:03.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:03 smithi193 ceph-mon[33593]: pgmap v938: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T12:12:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:03 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:12:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:03 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:12:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:03 smithi193 ceph-mon[33593]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T12:12:03.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:03.655 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:03.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:03.758 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:03.797 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:03.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:03.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:03.933 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:03.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:04.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:04.077 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:04.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:04.179 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:04.218 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:04.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:04.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:04.334 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 74 16 113917 113901 96.1794 78.5625 0.0135399 0.0103931 2024-03-24T12:12:04.357 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:04.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:04.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:04.498 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:04.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:04.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:04.626 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:04.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:04.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:04.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:04.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:04.915 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:04.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:05.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:05.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:05.055 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:05.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:05.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:05.197 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:05.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:05.298 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:05.332 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:05.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 75 16 115183 115167 95.9519 79.125 0.00625864 0.0104169 2024-03-24T12:12:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:05.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:05.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:05 smithi064 ceph-mon[30223]: pgmap v940: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 791 B/s rd, 60 MiB/s wr, 962 op/s 2024-03-24T12:12:05.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:05 smithi064 ceph-mon[42243]: pgmap v940: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 791 B/s rd, 60 MiB/s wr, 962 op/s 2024-03-24T12:12:05.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:05.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:05.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:05.579 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:05 smithi193 ceph-mon[33593]: pgmap v940: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 791 B/s rd, 60 MiB/s wr, 962 op/s 2024-03-24T12:12:05.609 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:05.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:05.711 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:05.748 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:05.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:05.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:05.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:05.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:05.995 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:06.024 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:06.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:06.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:06.161 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:06.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:06.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:06.301 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:06.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:06.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 76 16 116470 116454 95.7477 80.4375 0.00451563 0.0104396 2024-03-24T12:12:06.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:06.436 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:06.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:06.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:06.583 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:06.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:06.685 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:06.716 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:06.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:06.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:06.856 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:06.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:06.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:07.003 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:07.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:07.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:07.139 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:07.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:07.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:07.278 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:07.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-mon[30223]: pgmap v941: 158 pgs: 158 active+clean; 6.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-24T12:12:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:12:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:12:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-mon[30223]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T12:12:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-mon[42243]: pgmap v941: 158 pgs: 158 active+clean; 6.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-24T12:12:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:12:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:12:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-mon[42243]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T12:12:07.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 77 16 117730 117714 95.5269 78.75 0.0107567 0.0104632 2024-03-24T12:12:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:07 smithi193 ceph-mon[33593]: pgmap v941: 158 pgs: 158 active+clean; 6.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-24T12:12:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:07 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:12:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:07 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:12:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:07 smithi193 ceph-mon[33593]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T12:12:07.381 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:07.417 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:07.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:07.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:07.551 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:07.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:07.654 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:07.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:07.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:07.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:07.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:12:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:12:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:07.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:07.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:07.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:07.958 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:07.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:08.060 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:08.086 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:12:08.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:08.093 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:08.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:08.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:08.248 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:08.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:08 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:12:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:08 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:12:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:08 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:12:08.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:08 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:12:08.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 78 16 118866 118850 95.2125 71 0.00976559 0.010498 2024-03-24T12:12:08.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:12:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:12:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:08 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:12:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:08 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:12:08.349 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:08.397 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:08.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:08.486 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:08.497 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:08.531 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:08.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:08.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:08.671 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:08.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:08.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:08.816 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:08.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:08.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:08.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:09.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:09.101 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:09.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:09.147 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:12:09.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:09.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:09.249 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:09 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:12:09.249 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:09 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:12:09.249 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:09 smithi193 ceph-mon[33593]: pgmap v943: 158 pgs: 158 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-24T12:12:09.249 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:09.249 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:09 smithi193 ceph-mon[33593]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T12:12:09.249 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:09 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2344886464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:09.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 79 16 120052 120036 94.9455 74.125 0.00639528 0.0105278 2024-03-24T12:12:09.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:09.367 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:12:09.367 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:12:09.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[42243]: pgmap v943: 158 pgs: 158 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-24T12:12:09.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:09.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[42243]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T12:12:09.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2344886464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:09.368 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:12:09.368 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:12:09.369 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[30223]: pgmap v943: 158 pgs: 158 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-24T12:12:09.369 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:09.369 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[30223]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T12:12:09.369 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:09 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2344886464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:09.381 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:09.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:09.482 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:09.521 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:09.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:09.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:09.660 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:09.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:09.762 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:09.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:09.904 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:09.939 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:09.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:10.041 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:10.071 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:10.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:10.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:10.225 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:10.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:10.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:10.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:2024-03-24T12:12:10.335431+0000 min lat: 0.00124462 max lat: 0.182416 avg lat: 0.0105524 2024-03-24T12:12:10.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:12:10.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 80 16 121286 121270 94.7227 77.125 0.0123483 0.0105524 2024-03-24T12:12:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:10.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:10.387 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:10 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:10.387 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2344886464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:12:10.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:10 smithi193 ceph-mon[33593]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T12:12:10.403 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:10 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:10.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2344886464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:12:10.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:10 smithi064 ceph-mon[30223]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T12:12:10.484 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:10 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:10.484 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2344886464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:12:10.484 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:10 smithi064 ceph-mon[42243]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T12:12:10.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:10.528 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:10.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:10.629 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:10.661 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:10.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:10.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:10.797 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:10.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:10.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:10.933 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:10.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:11.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:11.069 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:11.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:11.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:11.220 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:11.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:11.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[30223]: pgmap v946: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-24T12:12:11.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[30223]: 11.f scrub starts 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[42243]: pgmap v946: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:12:11.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:11 smithi064 ceph-mon[42243]: 11.f scrub starts 2024-03-24T12:12:11.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:11.334 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 81 16 122355 122339 94.3781 66.8125 0.0151022 0.0105909 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:122350 11.e 11:7528238c:::benchmark_data_smithi064_206665_object122349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:11.332121+0000", 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0054963069999999998, 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055356219999999996, 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:11.337 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 122350 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332121+0000", 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332121+0000", 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:11.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332122+0000", 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.426e-06 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332173+0000", 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0463000000000001e-05 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332183+0000", 2024-03-24T12:12:11.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.91e-06 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332191+0000", 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8760000000000005e-06 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332237+0000", 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6081999999999997e-05 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332328+0000", 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1346000000000003e-05 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,7", 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.332415+0000", 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7020000000000004e-05 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.335420+0000", 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030049450000000002 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:11.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:122352 11.d 11:bc66e990:::benchmark_data_smithi064_206665_object122351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:11.335908+0000", 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.001709005, 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018397839999999999, 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 122352 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:11.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.335908+0000", 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.335908+0000", 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:11.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.335911+0000", 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6989999999999999e-06 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.335955+0000", 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3955000000000002e-05 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.335972+0000", 2024-03-24T12:12:11.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7453999999999999e-05 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.335986+0000", 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3451e-05 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.336179+0000", 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019351500000000001 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.345 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.336322+0000", 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014258399999999999 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:11.336474+0000", 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000151586 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 2 2024-03-24T12:12:11.346 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:12:11.358 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:11.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:11.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:11.504 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:11.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:11 smithi193 ceph-mon[33593]: pgmap v946: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-24T12:12:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:11 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T12:12:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:11 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T12:12:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:11 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:12:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:11 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:12:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:11 smithi193 ceph-mon[33593]: 11.f scrub starts 2024-03-24T12:12:11.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:11.645 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:11.645 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:12:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:11.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:11.691 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:12:11.730 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:12:11.747 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:11.775 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:12:11.783 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:11.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:11.815 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:12:11.855 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:12:11.885 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:11.894 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:12:11.915 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:11.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:11.936 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:12:11.946 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:12:11.946 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:12:11.946 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:12:11.947 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:12:11.947 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:12:11.947 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:12:11.947 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:12:11.947 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:12:12.016 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:12.048 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:12.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:12.097 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:12:12.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:12:12.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:12.164 INFO:teuthology.orchestra.run.smithi064.stderr:set 6.b pg_upmap_items mapping to [2->0,4->5,1->7] 2024-03-24T12:12:12.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:12 smithi064 ceph-mon[42243]: 11.b scrub starts 2024-03-24T12:12:12.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:12 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1035449229' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [2, 0, 4, 5, 1, 7]}]: dispatch 2024-03-24T12:12:12.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:12 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3929399053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:12.197 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:12 smithi064 ceph-mon[30223]: 11.b scrub starts 2024-03-24T12:12:12.197 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:12 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1035449229' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [2, 0, 4, 5, 1, 7]}]: dispatch 2024-03-24T12:12:12.197 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:12 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3929399053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:12.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:12.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:12.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:12.333 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 82 16 123342 123326 93.9794 61.6875 0.0158258 0.0106357 2024-03-24T12:12:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:12 smithi193 ceph-mon[33593]: 11.b scrub starts 2024-03-24T12:12:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:12 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1035449229' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [2, 0, 4, 5, 1, 7]}]: dispatch 2024-03-24T12:12:12.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:12 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3929399053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:12.363 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:12.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:12.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:12.475 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:12.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:12.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:12.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:12.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:12.790 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:12.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:12.892 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:12.932 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:12.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:13.034 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:13.079 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:13.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:13.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:13.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:13.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:13.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[30223]: pgmap v947: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T12:12:13.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1035449229' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [2, 0, 4, 5, 1, 7]}]': finished 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[30223]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[30223]: 11.5 scrub starts 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[30223]: 11.f scrub ok 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[42243]: pgmap v947: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1035449229' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [2, 0, 4, 5, 1, 7]}]': finished 2024-03-24T12:12:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[42243]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T12:12:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[42243]: 11.5 scrub starts 2024-03-24T12:12:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:13.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:13 smithi064 ceph-mon[42243]: 11.f scrub ok 2024-03-24T12:12:13.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:13.334 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 83 15 124437 124422 93.6723 68.5 0.0179559 0.0106705 2024-03-24T12:12:13.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:13 smithi193 ceph-mon[33593]: pgmap v947: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T12:12:13.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:13 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:12:13.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:13 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:12:13.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:13 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1035449229' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [2, 0, 4, 5, 1, 7]}]': finished 2024-03-24T12:12:13.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:13 smithi193 ceph-mon[33593]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T12:12:13.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:13 smithi193 ceph-mon[33593]: 11.5 scrub starts 2024-03-24T12:12:13.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:13 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:13.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:13 smithi193 ceph-mon[33593]: 11.f scrub ok 2024-03-24T12:12:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:13.352 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:13.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:13.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:13.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:13.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:13.586 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:13.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:13.627 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:13.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:13.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:13.764 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:13.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:13.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:13.904 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:13.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:14.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:14.049 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:14.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:14.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:14.197 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:14.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:14.298 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:14.334 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 84 16 125594 125578 93.4173 72.25 0.00502612 0.0106992 2024-03-24T12:12:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:14 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T12:12:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:14 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T12:12:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:14 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T12:12:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:14 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T12:12:14.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:14 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:12:14.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:14 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:12:14.346 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:14.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:14.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:14.488 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:14.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T12:12:14.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T12:12:14.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T12:12:14.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T12:12:14.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:12:14.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:12:14.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T12:12:14.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T12:12:14.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T12:12:14.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T12:12:14.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:12:14.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:14 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:12:14.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:14.629 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:14.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:14.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:14.767 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:14.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:14.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:14.850 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:12:14.851 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:12:14.868 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:14.911 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:14.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:15.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:15.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:15.169 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:15.204 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:15.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:15.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:15 smithi064 ceph-mon[30223]: pgmap v949: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-24T12:12:15.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:15 smithi064 ceph-mon[30223]: 11.b scrub ok 2024-03-24T12:12:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:15 smithi064 ceph-mon[30223]: 11.5 scrub ok 2024-03-24T12:12:15.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:15 smithi064 ceph-mon[30223]: 11.4 scrub starts 2024-03-24T12:12:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:15 smithi064 ceph-mon[42243]: pgmap v949: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-24T12:12:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:15 smithi064 ceph-mon[42243]: 11.b scrub ok 2024-03-24T12:12:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:15 smithi064 ceph-mon[42243]: 11.5 scrub ok 2024-03-24T12:12:15.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:15 smithi064 ceph-mon[42243]: 11.4 scrub starts 2024-03-24T12:12:15.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:15.334 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 85 16 126706 126690 93.1359 69.5 0.00999076 0.010732 2024-03-24T12:12:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:15.345 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:15.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:15.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:15.482 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:15.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:15.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:15 smithi193 ceph-mon[33593]: pgmap v949: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-24T12:12:15.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:15 smithi193 ceph-mon[33593]: 11.b scrub ok 2024-03-24T12:12:15.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:15 smithi193 ceph-mon[33593]: 11.5 scrub ok 2024-03-24T12:12:15.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:15 smithi193 ceph-mon[33593]: 11.4 scrub starts 2024-03-24T12:12:15.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:15.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:12:15.620 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:12:15.625 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:15.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:15.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:15.773 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:15.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:15.873 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:15.911 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:15.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:15.988 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:16.013 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:16.066 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:16.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:16.168 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:16.197 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:16.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:16.298 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:16.334 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 86 16 127737 127721 92.8021 64.4375 0.0173301 0.0107706 2024-03-24T12:12:16.343 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:16.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:16.445 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:16.479 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:16.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:16.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:16 smithi193 ceph-mon[33593]: 6.b scrub starts 2024-03-24T12:12:16.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:16 smithi193 ceph-mon[33593]: 6.b scrub ok 2024-03-24T12:12:16.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:16 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T12:12:16.482 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:16 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T12:12:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:16 smithi064 ceph-mon[30223]: 6.b scrub starts 2024-03-24T12:12:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:16 smithi064 ceph-mon[30223]: 6.b scrub ok 2024-03-24T12:12:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:16 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T12:12:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:16 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T12:12:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:16 smithi064 ceph-mon[42243]: 6.b scrub starts 2024-03-24T12:12:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:16 smithi064 ceph-mon[42243]: 6.b scrub ok 2024-03-24T12:12:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:16 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T12:12:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:16 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T12:12:16.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:16.625 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:16.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:16.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:16.767 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:16.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:16.868 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:16.910 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:16.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:17.012 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:17.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:17.045 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:17.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:17.146 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:17.183 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:17.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:17.230 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:12:17.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:17.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:17.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:17.334 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 87 16 128813 128797 92.5084 67.25 0.0115025 0.0108052 2024-03-24T12:12:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:17 smithi193 ceph-mon[33593]: pgmap v950: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 689 B/s rd, 55 MiB/s wr, 880 op/s 2024-03-24T12:12:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:17 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T12:12:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:17 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T12:12:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:17 smithi193 ceph-mon[33593]: 11.c scrub starts 2024-03-24T12:12:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:17 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3783706170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:17 smithi193 ceph-mon[33593]: from='client.25918 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:17.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:17 smithi193 ceph-mon[33593]: 11.4 scrub ok 2024-03-24T12:12:17.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:17.448 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:17.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:17.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[30223]: pgmap v950: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 689 B/s rd, 55 MiB/s wr, 880 op/s 2024-03-24T12:12:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T12:12:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T12:12:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[30223]: 11.c scrub starts 2024-03-24T12:12:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3783706170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[30223]: from='client.25918 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:17.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[30223]: 11.4 scrub ok 2024-03-24T12:12:17.554 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:12:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[42243]: pgmap v950: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 689 B/s rd, 55 MiB/s wr, 880 op/s 2024-03-24T12:12:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T12:12:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T12:12:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[42243]: 11.c scrub starts 2024-03-24T12:12:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3783706170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[42243]: from='client.25918 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:17.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:17 smithi064 ceph-mon[42243]: 11.4 scrub ok 2024-03-24T12:12:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:17.600 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:17.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:17.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:17.745 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:17.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:17.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:17.884 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:17.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:17.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:18.027 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:18.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:18.032 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:12:18.032 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:12:18.033 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-24T12:12:18.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:12:18.061 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:12:18.061 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:12:18.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:18.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:18.209 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:12:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:12:18.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:18.327 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:18.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:18.334 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 88 16 129869 129853 92.2069 66 0.00719406 0.0108391 2024-03-24T12:12:18.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:18.473 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:18 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:12:18.473 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:18 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:12:18.473 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:18 smithi193 ceph-mon[33593]: from='client.25918 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:12:18.473 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:18 smithi193 ceph-mon[33593]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T12:12:18.473 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:18 smithi193 ceph-mon[33593]: 4.8 scrub starts 2024-03-24T12:12:18.473 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:18 smithi193 ceph-mon[33593]: 4.8 scrub ok 2024-03-24T12:12:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:18.481 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:18.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[30223]: from='client.25918 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[30223]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[30223]: 4.8 scrub starts 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[30223]: 4.8 scrub ok 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[42243]: from='client.25918 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[42243]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[42243]: 4.8 scrub starts 2024-03-24T12:12:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:18 smithi064 ceph-mon[42243]: 4.8 scrub ok 2024-03-24T12:12:18.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:18.626 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:18.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:18.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:18.768 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:18.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:18.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:12:18.870 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:18.918 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:18.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:19.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:19.073 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:19.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:19.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:19.209 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:19.219 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:19.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:19.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:19.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout: 89 16 130942 130926 91.924 67.0625 0.0221804 0.0108739 2024-03-24T12:12:19.364 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:19.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:19.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:19.498 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:19.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:19.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:19.510 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:19 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T12:12:19.511 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:19 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T12:12:19.511 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:19 smithi193 ceph-mon[33593]: pgmap v952: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T12:12:19.511 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:19.511 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:19 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T12:12:19.511 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:19 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T12:12:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[30223]: pgmap v952: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[42243]: pgmap v952: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T12:12:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:19 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T12:12:19.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:19.638 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:19.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:19.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:19.778 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:19.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:19.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:19.921 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:19.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:20.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:20.060 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:20.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:20.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:20.206 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:20.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:20.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total time run: 90.0077 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Total writes made: 132130 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Write size: 65536 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Object size: 65536 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Bandwidth (MB/sec): 91.7491 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Bandwidth: 31.5626 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max bandwidth (MB/sec): 205.625 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min bandwidth (MB/sec): 61.6875 2024-03-24T12:12:20.335 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average IOPS: 1467 2024-03-24T12:12:20.336 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev IOPS: 505.002 2024-03-24T12:12:20.336 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max IOPS: 3290 2024-03-24T12:12:20.336 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min IOPS: 987 2024-03-24T12:12:20.336 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Average Latency(s): 0.0108951 2024-03-24T12:12:20.336 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Stddev Latency(s): 0.00754634 2024-03-24T12:12:20.336 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Max latency(s): 0.182416 2024-03-24T12:12:20.336 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Min latency(s): 0.00124462 2024-03-24T12:12:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:20.337 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:20.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:20.338 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:12:20.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:20 smithi064 ceph-mon[30223]: 11.c scrub ok 2024-03-24T12:12:20.393 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:20 smithi064 ceph-mon[42243]: 11.c scrub ok 2024-03-24T12:12:20.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:20.483 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:20.496 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:20.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:20.520 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:12:20.558 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:12:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:20 smithi193 ceph-mon[33593]: 11.c scrub ok 2024-03-24T12:12:20.598 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:20.601 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:12:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:20.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:20.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:20.641 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:12:20.681 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:12:20.726 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:12:20.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:20.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:12:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:20.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:20.807 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:12:20.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:12:20.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:12:20.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:12:20.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:12:20.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:12:20.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:12:20.815 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:12:20.815 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:12:20.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:20.923 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:20.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:21.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:21.063 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:21.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:21.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:21.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:21.303 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:21.343 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:21.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:21.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:21.483 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:21.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:21.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: 5.1e scrub starts 2024-03-24T12:12:21.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: 5.1e scrub ok 2024-03-24T12:12:21.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: pgmap v953: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:12:21.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4197254202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: 5.1e scrub starts 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: 5.1e scrub ok 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: pgmap v953: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T12:12:21.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T12:12:21.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T12:12:21.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4197254202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:21.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: 5.1e scrub starts 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: 5.1e scrub ok 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: pgmap v953: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T12:12:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4197254202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:21.609 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:21.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:21.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:21.753 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:21.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:21.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:12:21.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:21.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:22.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:22.072 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:22.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:22.173 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:22.204 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:22.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:22.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:22.335 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:22.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:22.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:22.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:22.474 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:22.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[30223]: 4.d scrub starts 2024-03-24T12:12:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[30223]: 4.d scrub ok 2024-03-24T12:12:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[30223]: 6.1 deep-scrub starts 2024-03-24T12:12:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[30223]: 6.1 deep-scrub ok 2024-03-24T12:12:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[30223]: 5.7 deep-scrub starts 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[30223]: 5.7 deep-scrub ok 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[42243]: 4.d scrub starts 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[42243]: 4.d scrub ok 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[42243]: 6.1 deep-scrub starts 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[42243]: 6.1 deep-scrub ok 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[42243]: 5.7 deep-scrub starts 2024-03-24T12:12:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:22 smithi064 ceph-mon[42243]: 5.7 deep-scrub ok 2024-03-24T12:12:22.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:22 smithi193 ceph-mon[33593]: 4.d scrub starts 2024-03-24T12:12:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:22 smithi193 ceph-mon[33593]: 4.d scrub ok 2024-03-24T12:12:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:22 smithi193 ceph-mon[33593]: 6.1 deep-scrub starts 2024-03-24T12:12:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:22 smithi193 ceph-mon[33593]: 6.1 deep-scrub ok 2024-03-24T12:12:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:22 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T12:12:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:22 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T12:12:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:22 smithi193 ceph-mon[33593]: 5.7 deep-scrub starts 2024-03-24T12:12:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:22 smithi193 ceph-mon[33593]: 5.7 deep-scrub ok 2024-03-24T12:12:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:22.614 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:22.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:22.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:22.749 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:22.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:22.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:22.888 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:22.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:134370 11.3 11:c8c34685:::benchmark_data_smithi064_206665_object2237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:22.306338+0000", 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0052733550000000004, 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00530877, 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:12:22.974 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 134370 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306338+0000", 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306338+0000", 2024-03-24T12:12:22.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306340+0000", 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7579999999999999e-06 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306345+0000", 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5719999999999996e-06 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306355+0000", 2024-03-24T12:12:22.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9559999999999992e-06 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306363+0000", 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2509999999999994e-06 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306421+0000", 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7537000000000002e-05 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306546+0000", 2024-03-24T12:12:22.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000125618 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,7", 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.306617+0000", 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1279999999999995e-05 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.308986+0000", 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023682780000000001 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:22.309131+0000", 2024-03-24T12:12:22.978 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014517800000000001 2024-03-24T12:12:22.979 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:22.979 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:22.979 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:22.979 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:22.979 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T12:12:22.979 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 1 2024-03-24T12:12:22.979 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:12:22.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:23.024 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:23.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:23.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:23.164 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:23.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:23.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:23.291 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:23.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:23.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:23 smithi064 ceph-mon[30223]: pgmap v955: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 764 op/s 2024-03-24T12:12:23.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:23 smithi064 ceph-mon[42243]: pgmap v955: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 764 op/s 2024-03-24T12:12:23.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:23.429 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:23.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:23.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:23.567 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:23.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:23 smithi193 ceph-mon[33593]: pgmap v955: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 764 op/s 2024-03-24T12:12:23.669 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:23.703 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:23.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:23.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:23.853 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:23.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:23.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:23.989 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:23.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:24.091 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:24.132 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:24.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:24.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:24.270 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:24.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:24.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:12:24.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:24.401 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:24.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:24.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:24.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:24.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:24.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:24 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T12:12:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:24 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T12:12:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:24 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T12:12:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:24 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T12:12:24.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:24 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T12:12:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:24 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T12:12:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:24 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T12:12:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:24 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T12:12:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:24 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T12:12:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:24 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T12:12:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:24 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T12:12:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:24 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T12:12:24.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:24.697 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:24.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:24.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:24.845 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:24.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:24.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:24.982 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:24.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:25.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:25.112 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:25.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:25.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:25.250 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:25.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:25.351 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:25.388 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:25.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:25.462 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:25.489 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:25.522 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:25.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[30223]: pgmap v956: 158 pgs: 158 active+clean; 8.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.28k op/s 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[42243]: pgmap v956: 158 pgs: 158 active+clean; 8.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.28k op/s 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:12:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:12:25.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:25 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:12:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:25 smithi193 ceph-mon[33593]: pgmap v956: 158 pgs: 158 active+clean; 8.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.28k op/s 2024-03-24T12:12:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:25 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T12:12:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:25 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T12:12:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:25 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:12:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:25 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:12:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:25 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:12:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:25 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:12:25.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:25.624 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:25.677 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:25.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:25.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:12:25.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:25.892 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:25.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:25.994 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:26.048 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:26.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:26.100 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:26.150 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:26.192 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:26.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:26.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:26.317 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:26.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:26.418 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:26.445 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:26.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:26.546 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:26 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T12:12:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:26 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T12:12:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:26 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T12:12:26.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:26 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T12:12:26.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:26.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:26.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:26 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T12:12:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:26 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T12:12:26.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:26.723 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:26.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:26.824 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:26.881 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:26.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:26.983 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:27.008 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:27.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:27.110 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:27.149 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:27.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:27.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:27.298 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:27.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:27.383 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:27 smithi064 ceph-mon[30223]: pgmap v957: 158 pgs: 158 active+clean; 7.9 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 9.0 KiB/s rd, 35 MiB/s wr, 813 op/s 2024-03-24T12:12:27.384 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:27 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:12:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:27 smithi064 ceph-mon[42243]: pgmap v957: 158 pgs: 158 active+clean; 7.9 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 9.0 KiB/s rd, 35 MiB/s wr, 813 op/s 2024-03-24T12:12:27.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:27 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:12:27.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:27.447 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:27.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:27.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:27.549 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:27.580 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:27.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:27 smithi193 ceph-mon[33593]: pgmap v957: 158 pgs: 158 active+clean; 7.9 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 9.0 KiB/s rd, 35 MiB/s wr, 813 op/s 2024-03-24T12:12:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:27 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:12:27.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:27.726 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:27.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:27.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:12:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:12:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:27.827 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:27.859 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:27.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:27.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:28.007 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:28.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:28.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:12:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:12:28.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:28.147 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:28.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:28.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:28.283 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:28.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:28.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:28.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:28.417 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:28.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:28.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:12:28.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:12:28.537 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:28.562 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:28.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T12:12:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T12:12:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T12:12:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T12:12:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:12:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:12:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:28.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:28.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:28.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:28.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:28.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:28.996 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:28.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:29.098 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:29.130 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:29.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:29.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:29.275 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:29.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:29.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:29.416 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:29.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:29.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:29 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T12:12:29.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:29 smithi193 ceph-mon[33593]: pgmap v958: 158 pgs: 158 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 40 KiB/s rd, 27 MiB/s wr, 1.19k op/s 2024-03-24T12:12:29.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:29 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T12:12:29.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:29 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:29.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:29 smithi193 ceph-mon[33593]: 1.4 scrub starts 2024-03-24T12:12:29.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:29 smithi193 ceph-mon[33593]: 1.4 scrub ok 2024-03-24T12:12:29.458 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:12:29.461 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:12:29.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:29.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T12:12:29.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[30223]: pgmap v958: 158 pgs: 158 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 40 KiB/s rd, 27 MiB/s wr, 1.19k op/s 2024-03-24T12:12:29.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T12:12:29.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:29.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[30223]: 1.4 scrub starts 2024-03-24T12:12:29.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[30223]: 1.4 scrub ok 2024-03-24T12:12:29.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T12:12:29.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[42243]: pgmap v958: 158 pgs: 158 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 40 KiB/s rd, 27 MiB/s wr, 1.19k op/s 2024-03-24T12:12:29.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T12:12:29.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:29.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[42243]: 1.4 scrub starts 2024-03-24T12:12:29.531 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:29 smithi064 ceph-mon[42243]: 1.4 scrub ok 2024-03-24T12:12:29.558 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:29.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:29.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:29.697 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:29.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:29.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:29.841 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:29.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:29.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:29.974 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:29.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:30.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:30.119 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:30.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:30.221 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:30.249 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:30.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:30 smithi193 ceph-mon[33593]: 4.c scrub starts 2024-03-24T12:12:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:30 smithi193 ceph-mon[33593]: 4.c scrub ok 2024-03-24T12:12:30.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:30 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T12:12:30.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:30 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T12:12:30.351 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:30.387 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:30.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:30.489 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:30.522 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:30.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:30.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:30 smithi064 ceph-mon[30223]: 4.c scrub starts 2024-03-24T12:12:30.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:30 smithi064 ceph-mon[30223]: 4.c scrub ok 2024-03-24T12:12:30.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:30 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T12:12:30.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:30 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T12:12:30.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:30 smithi064 ceph-mon[42243]: 4.c scrub starts 2024-03-24T12:12:30.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:30 smithi064 ceph-mon[42243]: 4.c scrub ok 2024-03-24T12:12:30.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:30 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T12:12:30.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:30 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T12:12:30.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:30.660 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:30.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:30.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:30.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:30.800 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:30.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:30.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:30.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:30.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:31.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:31.091 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:31.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:31.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:31.231 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:31.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:31.298 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:12:31.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:31.378 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:31.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:31.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:31.515 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:31.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[30223]: from='client.35957 v1:172.21.15.64:0/1427412240' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:12:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[30223]: pgmap v959: 158 pgs: 158 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-03-24T12:12:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T12:12:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T12:12:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1826234200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:12:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T12:12:31.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T12:12:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[42243]: from='client.35957 v1:172.21.15.64:0/1427412240' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:12:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[42243]: pgmap v959: 158 pgs: 158 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-03-24T12:12:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T12:12:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T12:12:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1826234200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:12:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T12:12:31.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:31 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T12:12:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:31 smithi193 ceph-mon[33593]: from='client.35957 v1:172.21.15.64:0/1427412240' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:12:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:31 smithi193 ceph-mon[33593]: pgmap v959: 158 pgs: 158 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-03-24T12:12:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:31 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T12:12:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:31 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T12:12:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1826234200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:12:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:31 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T12:12:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:31 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T12:12:31.610 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:12:31.611 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:12:31.611 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T12:12:31.611 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T12:12:31.611 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:12:31.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:31.648 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:31.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:31.749 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:31.790 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:31.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:31.892 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:31.932 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:31.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:32.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:32.055 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":958,"stamp":"2024-03-24T12:12:27.556446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8207341942,"num_objects":125424,"num_object_clones":0,"num_object_copies":376272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125424,"num_whiteouts":0,"num_read":1935,"num_read_kb":1687,"num_write":140016,"num_write_kb":8456366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":329608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78868,"ondisk_log_size":78868,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":29899592,"kb_used_data":24162248,"kb_used_omap":18,"kb_used_meta":5737197,"kb_avail":720061624,"statfs":{"total":767960285184,"available":737343102976,"internally_reserved":0,"allocated":24742141952,"data_stored":24736475962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":5874889947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-165543912,"num_objects":-2525,"num_object_clones":0,"num_object_copies":-7575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2525,"num_whiteouts":0,"num_read":398,"num_read_kb":398,"num_write":11543,"num_write_kb":280257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003305"},"pg_stats":[{"pgid":"11.7","version":"450'8587","reported_seq":8663,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.269472+0000","last_change":"2024-03-24T12:10:42.552347+0000","last_active":"2024-03-24T12:12:24.269472+0000","last_peered":"2024-03-24T12:12:24.269472+0000","last_clean":"2024-03-24T12:12:24.269472+0000","last_became_active":"2024-03-24T12:10:42.552023+0000","last_became_peered":"2024-03-24T12:10:42.552023+0000","last_unstale":"2024-03-24T12:12:24.269472+0000","last_undegraded":"2024-03-24T12:12:24.269472+0000","last_fullsized":"2024-03-24T12:12:24.269472+0000","mapping_epoch":417,"log_start":"437'3700","ondisk_log_start":"437'3700","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521469952,"num_objects":7957,"num_object_clones":0,"num_object_copies":23871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8587,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":1117,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.293867+0000","last_change":"2024-03-24T12:12:23.293867+0000","last_active":"2024-03-24T12:12:23.293867+0000","last_peered":"2024-03-24T12:12:23.293867+0000","last_clean":"2024-03-24T12:12:23.293867+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:12:23.293867+0000","last_undegraded":"2024-03-24T12:12:23.293867+0000","last_fullsized":"2024-03-24T12:12:23.293867+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:12:23.293737+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.293737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:48.062463+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"450'37","reported_seq":784,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.102683+0000","last_change":"2024-03-24T12:12:17.532582+0000","last_active":"2024-03-24T12:12:24.102683+0000","last_peered":"2024-03-24T12:12:24.102683+0000","last_clean":"2024-03-24T12:12:24.102683+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:12:24.102683+0000","last_undegraded":"2024-03-24T12:12:24.102683+0000","last_fullsized":"2024-03-24T12:12:24.102683+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:12:17.532295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:17.532295+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:24.750874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":38,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":329,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:21.258403+0000","last_change":"2024-03-24T12:12:21.258445+0000","last_active":"2024-03-24T12:12:21.258403+0000","last_peered":"2024-03-24T12:12:21.258403+0000","last_clean":"2024-03-24T12:12:21.258403+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:12:21.258403+0000","last_undegraded":"2024-03-24T12:12:21.258403+0000","last_fullsized":"2024-03-24T12:12:21.258403+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:21.258323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:21.258323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:32.254341+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":753,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246374+0000","last_change":"2024-03-24T12:10:19.713100+0000","last_active":"2024-03-24T12:12:20.246374+0000","last_peered":"2024-03-24T12:12:20.246374+0000","last_clean":"2024-03-24T12:12:20.246374+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:12:20.246374+0000","last_undegraded":"2024-03-24T12:12:20.246374+0000","last_fullsized":"2024-03-24T12:12:20.246374+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.713016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.713016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"450'8852","reported_seq":9016,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.730037+0000","last_change":"2024-03-24T12:12:16.501705+0000","last_active":"2024-03-24T12:12:26.730037+0000","last_peered":"2024-03-24T12:12:26.730037+0000","last_clean":"2024-03-24T12:12:26.730037+0000","last_became_active":"2024-03-24T12:10:42.558408+0000","last_became_peered":"2024-03-24T12:10:42.558408+0000","last_unstale":"2024-03-24T12:12:26.730037+0000","last_undegraded":"2024-03-24T12:12:26.730037+0000","last_fullsized":"2024-03-24T12:12:26.730037+0000","mapping_epoch":417,"log_start":"439'4500","ondisk_log_start":"439'4500","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'8080","last_scrub_stamp":"2024-03-24T12:12:16.501400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:16.501400+0000","objects_scrubbed":8006,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:35.863020+0000","scrub_duration":2228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512884736,"num_objects":7826,"num_object_clones":0,"num_object_copies":23478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"450'19","reported_seq":874,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.761129+0000","last_change":"2024-03-24T12:11:18.368554+0000","last_active":"2024-03-24T12:12:23.761129+0000","last_peered":"2024-03-24T12:12:23.761129+0000","last_clean":"2024-03-24T12:12:23.761129+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:12:23.761129+0000","last_undegraded":"2024-03-24T12:12:23.761129+0000","last_fullsized":"2024-03-24T12:12:23.761129+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:11:18.368416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.368416+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:29.701377+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":26,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":880,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241159+0000","last_change":"2024-03-24T12:11:33.850001+0000","last_active":"2024-03-24T12:12:20.241159+0000","last_peered":"2024-03-24T12:12:20.241159+0000","last_clean":"2024-03-24T12:12:20.241159+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:12:20.241159+0000","last_undegraded":"2024-03-24T12:12:20.241159+0000","last_fullsized":"2024-03-24T12:12:20.241159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:33.849353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:11:33.849353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:51.989463+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":328,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240870+0000","last_change":"2024-03-24T12:11:37.592094+0000","last_active":"2024-03-24T12:12:20.240870+0000","last_peered":"2024-03-24T12:12:20.240870+0000","last_clean":"2024-03-24T12:12:20.240870+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:12:20.240870+0000","last_undegraded":"2024-03-24T12:12:20.240870+0000","last_fullsized":"2024-03-24T12:12:20.240870+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:37.591971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.591971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:53.358009+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":834,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.247371+0000","last_change":"2024-03-24T12:10:26.001526+0000","last_active":"2024-03-24T12:12:20.247371+0000","last_peered":"2024-03-24T12:12:20.247371+0000","last_clean":"2024-03-24T12:12:20.247371+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:12:20.247371+0000","last_undegraded":"2024-03-24T12:12:20.247371+0000","last_fullsized":"2024-03-24T12:12:20.247371+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:26.001400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:26.001400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"450'8688","reported_seq":8850,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.716509+0000","last_change":"2024-03-24T12:12:14.190930+0000","last_active":"2024-03-24T12:12:26.716509+0000","last_peered":"2024-03-24T12:12:26.716509+0000","last_clean":"2024-03-24T12:12:26.716509+0000","last_became_active":"2024-03-24T12:10:42.558497+0000","last_became_peered":"2024-03-24T12:10:42.558497+0000","last_unstale":"2024-03-24T12:12:26.716509+0000","last_undegraded":"2024-03-24T12:12:26.716509+0000","last_fullsized":"2024-03-24T12:12:26.716509+0000","mapping_epoch":417,"log_start":"439'4300","ondisk_log_start":"439'4300","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7845","last_scrub_stamp":"2024-03-24T12:12:14.190779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.190779+0000","objects_scrubbed":7762,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:34.139978+0000","scrub_duration":2001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510918656,"num_objects":7796,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8688,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"450'60","reported_seq":939,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.851924+0000","last_change":"2024-03-24T12:11:20.907053+0000","last_active":"2024-03-24T12:12:24.851924+0000","last_peered":"2024-03-24T12:12:24.851924+0000","last_clean":"2024-03-24T12:12:24.851924+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:12:24.851924+0000","last_undegraded":"2024-03-24T12:12:24.851924+0000","last_fullsized":"2024-03-24T12:12:24.851924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:11:20.906884+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.906884+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:39.544494+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":223,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246095+0000","last_change":"2024-03-24T12:11:27.439531+0000","last_active":"2024-03-24T12:12:20.246095+0000","last_peered":"2024-03-24T12:12:20.246095+0000","last_clean":"2024-03-24T12:12:20.246095+0000","last_became_active":"2024-03-24T12:08:36.164337+0000","last_became_peered":"2024-03-24T12:08:36.164337+0000","last_unstale":"2024-03-24T12:12:20.246095+0000","last_undegraded":"2024-03-24T12:12:20.246095+0000","last_fullsized":"2024-03-24T12:12:20.246095+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:27.439409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.439409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:27.467179+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":336,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.249589+0000","last_change":"2024-03-24T12:11:22.405559+0000","last_active":"2024-03-24T12:12:20.249589+0000","last_peered":"2024-03-24T12:12:20.249589+0000","last_clean":"2024-03-24T12:12:20.249589+0000","last_became_active":"2024-03-24T12:08:36.155105+0000","last_became_peered":"2024-03-24T12:08:36.155105+0000","last_unstale":"2024-03-24T12:12:20.249589+0000","last_undegraded":"2024-03-24T12:12:20.249589+0000","last_fullsized":"2024-03-24T12:12:20.249589+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:22.404955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:22.404955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:31.732903+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":859,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240470+0000","last_change":"2024-03-24T12:11:32.175174+0000","last_active":"2024-03-24T12:12:20.240470+0000","last_peered":"2024-03-24T12:12:20.240470+0000","last_clean":"2024-03-24T12:12:20.240470+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:12:20.240470+0000","last_undegraded":"2024-03-24T12:12:20.240470+0000","last_fullsized":"2024-03-24T12:12:20.240470+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.175048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.175048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:00.349651+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"450'8709","reported_seq":8785,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.585423+0000","last_change":"2024-03-24T12:10:42.555659+0000","last_active":"2024-03-24T12:12:26.585423+0000","last_peered":"2024-03-24T12:12:26.585423+0000","last_clean":"2024-03-24T12:12:26.585423+0000","last_became_active":"2024-03-24T12:10:42.555322+0000","last_became_peered":"2024-03-24T12:10:42.555322+0000","last_unstale":"2024-03-24T12:12:26.585423+0000","last_undegraded":"2024-03-24T12:12:26.585423+0000","last_fullsized":"2024-03-24T12:12:26.585423+0000","mapping_epoch":417,"log_start":"439'4300","ondisk_log_start":"439'4300","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510853120,"num_objects":7795,"num_object_clones":0,"num_object_copies":23385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8709,"num_write_kb":528128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"450'53","reported_seq":944,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.710804+0000","last_change":"2024-03-24T12:12:20.673896+0000","last_active":"2024-03-24T12:12:24.710804+0000","last_peered":"2024-03-24T12:12:24.710804+0000","last_clean":"2024-03-24T12:12:24.710804+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:12:24.710804+0000","last_undegraded":"2024-03-24T12:12:24.710804+0000","last_fullsized":"2024-03-24T12:12:24.710804+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:12:20.673798+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:12:20.673798+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.976568+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":628,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242340+0000","last_change":"2024-03-24T12:09:04.230431+0000","last_active":"2024-03-24T12:12:20.242340+0000","last_peered":"2024-03-24T12:12:20.242340+0000","last_clean":"2024-03-24T12:12:20.242340+0000","last_became_active":"2024-03-24T12:09:04.230111+0000","last_became_peered":"2024-03-24T12:09:04.230111+0000","last_unstale":"2024-03-24T12:12:20.242340+0000","last_undegraded":"2024-03-24T12:12:20.242340+0000","last_fullsized":"2024-03-24T12:12:20.242340+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:08:42.842178+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.842178+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":591,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242715+0000","last_change":"2024-03-24T12:11:43.684235+0000","last_active":"2024-03-24T12:12:20.242715+0000","last_peered":"2024-03-24T12:12:20.242715+0000","last_clean":"2024-03-24T12:12:20.242715+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:12:20.242715+0000","last_undegraded":"2024-03-24T12:12:20.242715+0000","last_fullsized":"2024-03-24T12:12:20.242715+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.684070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.684070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:05.544597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":320,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.239024+0000","last_change":"2024-03-24T12:08:34.445597+0000","last_active":"2024-03-24T12:12:20.239024+0000","last_peered":"2024-03-24T12:12:20.239024+0000","last_clean":"2024-03-24T12:12:20.239024+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:12:20.239024+0000","last_undegraded":"2024-03-24T12:12:20.239024+0000","last_fullsized":"2024-03-24T12:12:20.239024+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:34.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.445477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":783,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246777+0000","last_change":"2024-03-24T12:10:09.762381+0000","last_active":"2024-03-24T12:12:20.246777+0000","last_peered":"2024-03-24T12:12:20.246777+0000","last_clean":"2024-03-24T12:12:20.246777+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:12:20.246777+0000","last_undegraded":"2024-03-24T12:12:20.246777+0000","last_fullsized":"2024-03-24T12:12:20.246777+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:09.762247+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.762247+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"450'8902","reported_seq":8978,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.824142+0000","last_change":"2024-03-24T12:10:42.557591+0000","last_active":"2024-03-24T12:12:26.824142+0000","last_peered":"2024-03-24T12:12:26.824142+0000","last_clean":"2024-03-24T12:12:26.824142+0000","last_became_active":"2024-03-24T12:10:42.557222+0000","last_became_peered":"2024-03-24T12:10:42.557222+0000","last_unstale":"2024-03-24T12:12:26.824142+0000","last_undegraded":"2024-03-24T12:12:26.824142+0000","last_fullsized":"2024-03-24T12:12:26.824142+0000","mapping_epoch":417,"log_start":"437'4200","ondisk_log_start":"437'4200","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520749056,"num_objects":7946,"num_object_clones":0,"num_object_copies":23838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8902,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"450'35","reported_seq":884,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.496166+0000","last_change":"2024-03-24T12:07:56.328634+0000","last_active":"2024-03-24T12:12:24.496166+0000","last_peered":"2024-03-24T12:12:24.496166+0000","last_clean":"2024-03-24T12:12:24.496166+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:12:24.496166+0000","last_undegraded":"2024-03-24T12:12:24.496166+0000","last_fullsized":"2024-03-24T12:12:24.496166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:07:56.328393+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:07:56.328393+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":592,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.236932+0000","last_change":"2024-03-24T12:10:31.694887+0000","last_active":"2024-03-24T12:12:20.236932+0000","last_peered":"2024-03-24T12:12:20.236932+0000","last_clean":"2024-03-24T12:12:20.236932+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:12:20.236932+0000","last_undegraded":"2024-03-24T12:12:20.236932+0000","last_fullsized":"2024-03-24T12:12:20.236932+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:31.694756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:10:31.694756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":1117,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.439698+0000","last_change":"2024-03-24T12:11:45.062251+0000","last_active":"2024-03-24T12:12:20.439698+0000","last_peered":"2024-03-24T12:12:20.439698+0000","last_clean":"2024-03-24T12:12:20.439698+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:12:20.439698+0000","last_undegraded":"2024-03-24T12:12:20.439698+0000","last_fullsized":"2024-03-24T12:12:20.439698+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:11:45.062114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.062114+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.519407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":846,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.243233+0000","last_change":"2024-03-24T12:10:09.602488+0000","last_active":"2024-03-24T12:12:20.243233+0000","last_peered":"2024-03-24T12:12:20.243233+0000","last_clean":"2024-03-24T12:12:20.243233+0000","last_became_active":"2024-03-24T12:08:36.161088+0000","last_became_peered":"2024-03-24T12:08:36.161088+0000","last_unstale":"2024-03-24T12:12:20.243233+0000","last_undegraded":"2024-03-24T12:12:20.243233+0000","last_fullsized":"2024-03-24T12:12:20.243233+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:10:09.602371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.602371+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":1179,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240701+0000","last_change":"2024-03-24T12:11:42.368078+0000","last_active":"2024-03-24T12:12:20.240701+0000","last_peered":"2024-03-24T12:12:20.240701+0000","last_clean":"2024-03-24T12:12:20.240701+0000","last_became_active":"2024-03-24T12:08:36.161668+0000","last_became_peered":"2024-03-24T12:08:36.161668+0000","last_unstale":"2024-03-24T12:12:20.240701+0000","last_undegraded":"2024-03-24T12:12:20.240701+0000","last_fullsized":"2024-03-24T12:12:20.240701+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:42.367917+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:11:42.367917+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.913748+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"450'8666","reported_seq":8742,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.557771+0000","last_change":"2024-03-24T12:10:42.559881+0000","last_active":"2024-03-24T12:12:26.557771+0000","last_peered":"2024-03-24T12:12:26.557771+0000","last_clean":"2024-03-24T12:12:26.557771+0000","last_became_active":"2024-03-24T12:10:42.559566+0000","last_became_peered":"2024-03-24T12:10:42.559566+0000","last_unstale":"2024-03-24T12:12:26.557771+0000","last_undegraded":"2024-03-24T12:12:26.557771+0000","last_fullsized":"2024-03-24T12:12:26.557771+0000","mapping_epoch":417,"log_start":"438'4300","ondisk_log_start":"438'4300","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511049728,"num_objects":7798,"num_object_clones":0,"num_object_copies":23394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8666,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"450'46","reported_seq":936,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.865559+0000","last_change":"2024-03-24T12:12:13.128854+0000","last_active":"2024-03-24T12:12:24.865559+0000","last_peered":"2024-03-24T12:12:24.865559+0000","last_clean":"2024-03-24T12:12:24.865559+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:12:24.865559+0000","last_undegraded":"2024-03-24T12:12:24.865559+0000","last_fullsized":"2024-03-24T12:12:24.865559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:12:13.128715+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:12:13.128715+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:25.266832+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":873,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241434+0000","last_change":"2024-03-24T12:11:43.926041+0000","last_active":"2024-03-24T12:12:20.241434+0000","last_peered":"2024-03-24T12:12:20.241434+0000","last_clean":"2024-03-24T12:12:20.241434+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:12:20.241434+0000","last_undegraded":"2024-03-24T12:12:20.241434+0000","last_fullsized":"2024-03-24T12:12:20.241434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.925905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.925905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:02.893804+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":1095,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.440672+0000","last_change":"2024-03-24T12:10:14.509080+0000","last_active":"2024-03-24T12:12:20.440672+0000","last_peered":"2024-03-24T12:12:20.440672+0000","last_clean":"2024-03-24T12:12:20.440672+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:12:20.440672+0000","last_undegraded":"2024-03-24T12:12:20.440672+0000","last_fullsized":"2024-03-24T12:12:20.440672+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:14.508959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.508959+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":761,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.247166+0000","last_change":"2024-03-24T12:11:13.744756+0000","last_active":"2024-03-24T12:12:20.247166+0000","last_peered":"2024-03-24T12:12:20.247166+0000","last_clean":"2024-03-24T12:12:20.247166+0000","last_became_active":"2024-03-24T12:08:36.166897+0000","last_became_peered":"2024-03-24T12:08:36.166897+0000","last_unstale":"2024-03-24T12:12:20.247166+0000","last_undegraded":"2024-03-24T12:12:20.247166+0000","last_fullsized":"2024-03-24T12:12:20.247166+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:13.744672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.744672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:34.151849+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":783,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246786+0000","last_change":"2024-03-24T12:10:11.797368+0000","last_active":"2024-03-24T12:12:20.246786+0000","last_peered":"2024-03-24T12:12:20.246786+0000","last_clean":"2024-03-24T12:12:20.246786+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:12:20.246786+0000","last_undegraded":"2024-03-24T12:12:20.246786+0000","last_fullsized":"2024-03-24T12:12:20.246786+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:11.797287+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:11.797287+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"450'8674","reported_seq":8750,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.813322+0000","last_change":"2024-03-24T12:10:42.557448+0000","last_active":"2024-03-24T12:12:26.813322+0000","last_peered":"2024-03-24T12:12:26.813322+0000","last_clean":"2024-03-24T12:12:26.813322+0000","last_became_active":"2024-03-24T12:10:42.557136+0000","last_became_peered":"2024-03-24T12:10:42.557136+0000","last_unstale":"2024-03-24T12:12:26.813322+0000","last_undegraded":"2024-03-24T12:12:26.813322+0000","last_fullsized":"2024-03-24T12:12:26.813322+0000","mapping_epoch":417,"log_start":"437'3900","ondisk_log_start":"437'3900","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512753664,"num_objects":7824,"num_object_clones":0,"num_object_copies":23472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8674,"num_write_kb":527936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"450'32","reported_seq":882,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.721991+0000","last_change":"2024-03-24T12:11:21.537427+0000","last_active":"2024-03-24T12:12:24.721991+0000","last_peered":"2024-03-24T12:12:24.721991+0000","last_clean":"2024-03-24T12:12:24.721991+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:12:24.721991+0000","last_undegraded":"2024-03-24T12:12:24.721991+0000","last_fullsized":"2024-03-24T12:12:24.721991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:11:21.537273+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:11:21.537273+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:37.951661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":865,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240728+0000","last_change":"2024-03-24T12:11:18.149890+0000","last_active":"2024-03-24T12:12:20.240728+0000","last_peered":"2024-03-24T12:12:20.240728+0000","last_clean":"2024-03-24T12:12:20.240728+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:12:20.240728+0000","last_undegraded":"2024-03-24T12:12:20.240728+0000","last_fullsized":"2024-03-24T12:12:20.240728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:18.149775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.149775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:31.562020+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":749,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.247298+0000","last_change":"2024-03-24T12:08:36.163502+0000","last_active":"2024-03-24T12:12:20.247298+0000","last_peered":"2024-03-24T12:12:20.247298+0000","last_clean":"2024-03-24T12:12:20.247298+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:12:20.247298+0000","last_undegraded":"2024-03-24T12:12:20.247298+0000","last_fullsized":"2024-03-24T12:12:20.247298+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:15.334211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:06:15.334211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":838,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.237729+0000","last_change":"2024-03-24T12:10:14.749596+0000","last_active":"2024-03-24T12:12:20.237729+0000","last_peered":"2024-03-24T12:12:20.237729+0000","last_clean":"2024-03-24T12:12:20.237729+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:12:20.237729+0000","last_undegraded":"2024-03-24T12:12:20.237729+0000","last_fullsized":"2024-03-24T12:12:20.237729+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:14.749468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.749468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"450'8641","reported_seq":8717,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.692335+0000","last_change":"2024-03-24T12:10:42.552380+0000","last_active":"2024-03-24T12:12:26.692335+0000","last_peered":"2024-03-24T12:12:26.692335+0000","last_clean":"2024-03-24T12:12:26.692335+0000","last_became_active":"2024-03-24T12:10:42.552054+0000","last_became_peered":"2024-03-24T12:10:42.552054+0000","last_unstale":"2024-03-24T12:12:26.692335+0000","last_undegraded":"2024-03-24T12:12:26.692335+0000","last_fullsized":"2024-03-24T12:12:26.692335+0000","mapping_epoch":417,"log_start":"437'3700","ondisk_log_start":"437'3700","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503644160,"num_objects":7685,"num_object_clones":0,"num_object_copies":23055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":522432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":1086,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:25.441061+0000","last_change":"2024-03-24T12:11:38.190269+0000","last_active":"2024-03-24T12:12:25.441061+0000","last_peered":"2024-03-24T12:12:25.441061+0000","last_clean":"2024-03-24T12:12:25.441061+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:12:25.441061+0000","last_undegraded":"2024-03-24T12:12:25.441061+0000","last_fullsized":"2024-03-24T12:12:25.441061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:11:38.190121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:38.190121+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.913586+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"450'49","reported_seq":914,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.742747+0000","last_change":"2024-03-24T12:10:19.407534+0000","last_active":"2024-03-24T12:12:24.742747+0000","last_peered":"2024-03-24T12:12:24.742747+0000","last_clean":"2024-03-24T12:12:24.742747+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:12:24.742747+0000","last_undegraded":"2024-03-24T12:12:24.742747+0000","last_fullsized":"2024-03-24T12:12:24.742747+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:10:19.407433+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.407433+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":855,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248394+0000","last_change":"2024-03-24T12:11:16.896924+0000","last_active":"2024-03-24T12:12:20.248394+0000","last_peered":"2024-03-24T12:12:20.248394+0000","last_clean":"2024-03-24T12:12:20.248394+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:12:20.248394+0000","last_undegraded":"2024-03-24T12:12:20.248394+0000","last_fullsized":"2024-03-24T12:12:20.248394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:16.896801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:11:16.896801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:23.741850+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":867,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.532132+0000","last_change":"2024-03-24T12:12:24.532173+0000","last_active":"2024-03-24T12:12:24.532132+0000","last_peered":"2024-03-24T12:12:24.532132+0000","last_clean":"2024-03-24T12:12:24.532132+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:12:24.532132+0000","last_undegraded":"2024-03-24T12:12:24.532132+0000","last_fullsized":"2024-03-24T12:12:24.532132+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:24.531916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:12:24.531916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:30.506866+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":917,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.237917+0000","last_change":"2024-03-24T12:10:47.972288+0000","last_active":"2024-03-24T12:12:20.237917+0000","last_peered":"2024-03-24T12:12:20.237917+0000","last_clean":"2024-03-24T12:12:20.237917+0000","last_became_active":"2024-03-24T12:10:47.971906+0000","last_became_peered":"2024-03-24T12:10:47.971906+0000","last_unstale":"2024-03-24T12:12:20.237917+0000","last_undegraded":"2024-03-24T12:12:20.237917+0000","last_fullsized":"2024-03-24T12:12:20.237917+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:08:36.201565+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:08:36.201565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"450'8878","reported_seq":9041,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.725613+0000","last_change":"2024-03-24T12:12:12.434955+0000","last_active":"2024-03-24T12:12:26.725613+0000","last_peered":"2024-03-24T12:12:26.725613+0000","last_clean":"2024-03-24T12:12:26.725613+0000","last_became_active":"2024-03-24T12:10:42.562110+0000","last_became_peered":"2024-03-24T12:10:42.562110+0000","last_unstale":"2024-03-24T12:12:26.725613+0000","last_undegraded":"2024-03-24T12:12:26.725613+0000","last_fullsized":"2024-03-24T12:12:26.725613+0000","mapping_epoch":417,"log_start":"439'4500","ondisk_log_start":"439'4500","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7838","last_scrub_stamp":"2024-03-24T12:12:12.434659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:12.434659+0000","objects_scrubbed":7761,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:20.803078+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522190848,"num_objects":7968,"num_object_clones":0,"num_object_copies":23904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8878,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":831,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240364+0000","last_change":"2024-03-24T12:11:32.519689+0000","last_active":"2024-03-24T12:12:20.240364+0000","last_peered":"2024-03-24T12:12:20.240364+0000","last_clean":"2024-03-24T12:12:20.240364+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:12:20.240364+0000","last_undegraded":"2024-03-24T12:12:20.240364+0000","last_fullsized":"2024-03-24T12:12:20.240364+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.519559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.519559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:57.052977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"450'50","reported_seq":915,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.596944+0000","last_change":"2024-03-24T12:10:29.733172+0000","last_active":"2024-03-24T12:12:24.596944+0000","last_peered":"2024-03-24T12:12:24.596944+0000","last_clean":"2024-03-24T12:12:24.596944+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:12:24.596944+0000","last_undegraded":"2024-03-24T12:12:24.596944+0000","last_fullsized":"2024-03-24T12:12:24.596944+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_deep_scrub":"241'38","last_deep_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.733073+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":608,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248239+0000","last_change":"2024-03-24T12:11:27.010877+0000","last_active":"2024-03-24T12:12:20.248239+0000","last_peered":"2024-03-24T12:12:20.248239+0000","last_clean":"2024-03-24T12:12:20.248239+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:12:20.248239+0000","last_undegraded":"2024-03-24T12:12:20.248239+0000","last_fullsized":"2024-03-24T12:12:20.248239+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:11:27.010737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.010737+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:34.851382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":849,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240949+0000","last_change":"2024-03-24T12:10:38.401132+0000","last_active":"2024-03-24T12:12:20.240949+0000","last_peered":"2024-03-24T12:12:20.240949+0000","last_clean":"2024-03-24T12:12:20.240949+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:12:20.240949+0000","last_undegraded":"2024-03-24T12:12:20.240949+0000","last_fullsized":"2024-03-24T12:12:20.240949+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:38.401028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:10:38.401028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":924,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.243561+0000","last_change":"2024-03-24T12:10:45.574511+0000","last_active":"2024-03-24T12:12:20.243561+0000","last_peered":"2024-03-24T12:12:20.243561+0000","last_clean":"2024-03-24T12:12:20.243561+0000","last_became_active":"2024-03-24T12:10:45.573960+0000","last_became_peered":"2024-03-24T12:10:45.573960+0000","last_unstale":"2024-03-24T12:12:20.243561+0000","last_undegraded":"2024-03-24T12:12:20.243561+0000","last_fullsized":"2024-03-24T12:12:20.243561+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:25.359883+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.359883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"450'8525","reported_seq":8689,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:27.259541+0000","last_change":"2024-03-24T12:12:19.183366+0000","last_active":"2024-03-24T12:12:27.259541+0000","last_peered":"2024-03-24T12:12:27.259541+0000","last_clean":"2024-03-24T12:12:27.259541+0000","last_became_active":"2024-03-24T12:10:42.552346+0000","last_became_peered":"2024-03-24T12:10:42.552346+0000","last_unstale":"2024-03-24T12:12:27.259541+0000","last_undegraded":"2024-03-24T12:12:27.259541+0000","last_fullsized":"2024-03-24T12:12:27.259541+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7988","last_scrub_stamp":"2024-03-24T12:12:19.183036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.183036+0000","objects_scrubbed":7889,"log_size":8525,"log_dups_size":0,"ondisk_log_size":8525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:25.262334+0000","scrub_duration":3159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499449856,"num_objects":7621,"num_object_clones":0,"num_object_copies":22863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8525,"num_write_kb":516672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":848,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.247876+0000","last_change":"2024-03-24T12:10:14.821427+0000","last_active":"2024-03-24T12:12:20.247876+0000","last_peered":"2024-03-24T12:12:20.247876+0000","last_clean":"2024-03-24T12:12:20.247876+0000","last_became_active":"2024-03-24T12:08:36.161658+0000","last_became_peered":"2024-03-24T12:08:36.161658+0000","last_unstale":"2024-03-24T12:12:20.247876+0000","last_undegraded":"2024-03-24T12:12:20.247876+0000","last_fullsized":"2024-03-24T12:12:20.247876+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:14.821157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.821157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"450'41","reported_seq":910,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.340270+0000","last_change":"2024-03-24T12:11:37.405546+0000","last_active":"2024-03-24T12:12:23.340270+0000","last_peered":"2024-03-24T12:12:23.340270+0000","last_clean":"2024-03-24T12:12:23.340270+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:12:23.340270+0000","last_undegraded":"2024-03-24T12:12:23.340270+0000","last_fullsized":"2024-03-24T12:12:23.340270+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'37","last_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_deep_scrub":"369'37","last_deep_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.405400+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:04.594252+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":46,"num_write":36,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":873,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242113+0000","last_change":"2024-03-24T12:11:17.183566+0000","last_active":"2024-03-24T12:12:20.242113+0000","last_peered":"2024-03-24T12:12:20.242113+0000","last_clean":"2024-03-24T12:12:20.242113+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:12:20.242113+0000","last_undegraded":"2024-03-24T12:12:20.242113+0000","last_fullsized":"2024-03-24T12:12:20.242113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:17.183126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:11:17.183126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.643290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":606,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.922062+0000","last_change":"2024-03-24T12:12:20.922086+0000","last_active":"2024-03-24T12:12:20.922062+0000","last_peered":"2024-03-24T12:12:20.922062+0000","last_clean":"2024-03-24T12:12:20.922062+0000","last_became_active":"2024-03-24T12:05:44.738926+0000","last_became_peered":"2024-03-24T12:05:44.738926+0000","last_unstale":"2024-03-24T12:12:20.922062+0000","last_undegraded":"2024-03-24T12:12:20.922062+0000","last_fullsized":"2024-03-24T12:12:20.922062+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:20.921985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:20.921985+0000","last_clean_scrub_stamp":"2024-03-24T12:12:20.921985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:22.281652+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":937,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248322+0000","last_change":"2024-03-24T12:11:45.081667+0000","last_active":"2024-03-24T12:12:20.248322+0000","last_peered":"2024-03-24T12:12:20.248322+0000","last_clean":"2024-03-24T12:12:20.248322+0000","last_became_active":"2024-03-24T12:10:41.554328+0000","last_became_peered":"2024-03-24T12:10:41.554328+0000","last_unstale":"2024-03-24T12:12:20.248322+0000","last_undegraded":"2024-03-24T12:12:20.248322+0000","last_fullsized":"2024-03-24T12:12:20.248322+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.081517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.235760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"450'8658","reported_seq":8734,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.546141+0000","last_change":"2024-03-24T12:10:42.558775+0000","last_active":"2024-03-24T12:12:26.546141+0000","last_peered":"2024-03-24T12:12:26.546141+0000","last_clean":"2024-03-24T12:12:26.546141+0000","last_became_active":"2024-03-24T12:10:42.558462+0000","last_became_peered":"2024-03-24T12:10:42.558462+0000","last_unstale":"2024-03-24T12:12:26.546141+0000","last_undegraded":"2024-03-24T12:12:26.546141+0000","last_fullsized":"2024-03-24T12:12:26.546141+0000","mapping_epoch":417,"log_start":"439'4300","ondisk_log_start":"439'4300","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510656512,"num_objects":7792,"num_object_clones":0,"num_object_copies":23376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8658,"num_write_kb":526400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":723,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:21.577833+0000","last_change":"2024-03-24T12:12:21.577856+0000","last_active":"2024-03-24T12:12:21.577833+0000","last_peered":"2024-03-24T12:12:21.577833+0000","last_clean":"2024-03-24T12:12:21.577833+0000","last_became_active":"2024-03-24T12:08:36.165265+0000","last_became_peered":"2024-03-24T12:08:36.165265+0000","last_unstale":"2024-03-24T12:12:21.577833+0000","last_undegraded":"2024-03-24T12:12:21.577833+0000","last_fullsized":"2024-03-24T12:12:21.577833+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:21.577764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:21.577764+0000","last_clean_scrub_stamp":"2024-03-24T12:12:21.577764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:45.211618+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"450'34","reported_seq":912,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.300117+0000","last_change":"2024-03-24T12:11:23.383853+0000","last_active":"2024-03-24T12:12:23.300117+0000","last_peered":"2024-03-24T12:12:23.300117+0000","last_clean":"2024-03-24T12:12:23.300117+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:12:23.300117+0000","last_undegraded":"2024-03-24T12:12:23.300117+0000","last_fullsized":"2024-03-24T12:12:23.300117+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:11:23.383616+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.383616+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:33.198353+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":684,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248675+0000","last_change":"2024-03-24T12:10:16.036972+0000","last_active":"2024-03-24T12:12:20.248675+0000","last_peered":"2024-03-24T12:12:20.248675+0000","last_clean":"2024-03-24T12:12:20.248675+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:12:20.248675+0000","last_undegraded":"2024-03-24T12:12:20.248675+0000","last_fullsized":"2024-03-24T12:12:20.248675+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:16.036850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:16.036850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":858,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.245878+0000","last_change":"2024-03-24T12:11:46.990122+0000","last_active":"2024-03-24T12:12:20.245878+0000","last_peered":"2024-03-24T12:12:20.245878+0000","last_clean":"2024-03-24T12:12:20.245878+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:12:20.245878+0000","last_undegraded":"2024-03-24T12:12:20.245878+0000","last_fullsized":"2024-03-24T12:12:20.245878+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:29.995266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.995266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":862,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.247573+0000","last_change":"2024-03-24T12:11:39.903255+0000","last_active":"2024-03-24T12:12:20.247573+0000","last_peered":"2024-03-24T12:12:20.247573+0000","last_clean":"2024-03-24T12:12:20.247573+0000","last_became_active":"2024-03-24T12:08:36.173357+0000","last_became_peered":"2024-03-24T12:08:36.173357+0000","last_unstale":"2024-03-24T12:12:20.247573+0000","last_undegraded":"2024-03-24T12:12:20.247573+0000","last_fullsized":"2024-03-24T12:12:20.247573+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_clean_scrub_stamp":"2024-03-24T12:11:39.903133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:04.706316+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":948,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.891723+0000","last_change":"2024-03-24T12:12:23.891768+0000","last_active":"2024-03-24T12:12:23.891723+0000","last_peered":"2024-03-24T12:12:23.891723+0000","last_clean":"2024-03-24T12:12:23.891723+0000","last_became_active":"2024-03-24T12:10:50.998877+0000","last_became_peered":"2024-03-24T12:10:50.998877+0000","last_unstale":"2024-03-24T12:12:23.891723+0000","last_undegraded":"2024-03-24T12:12:23.891723+0000","last_fullsized":"2024-03-24T12:12:23.891723+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:12:23.891623+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.891623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.276820+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"450'8603","reported_seq":8679,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.553711+0000","last_change":"2024-03-24T12:10:42.558754+0000","last_active":"2024-03-24T12:12:26.553711+0000","last_peered":"2024-03-24T12:12:26.553711+0000","last_clean":"2024-03-24T12:12:26.553711+0000","last_became_active":"2024-03-24T12:10:42.558225+0000","last_became_peered":"2024-03-24T12:10:42.558225+0000","last_unstale":"2024-03-24T12:12:26.553711+0000","last_undegraded":"2024-03-24T12:12:26.553711+0000","last_fullsized":"2024-03-24T12:12:26.553711+0000","mapping_epoch":417,"log_start":"438'4200","ondisk_log_start":"438'4200","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509935616,"num_objects":7781,"num_object_clones":0,"num_object_copies":23343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":858,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242489+0000","last_change":"2024-03-24T12:10:37.011600+0000","last_active":"2024-03-24T12:12:20.242489+0000","last_peered":"2024-03-24T12:12:20.242489+0000","last_clean":"2024-03-24T12:12:20.242489+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:12:20.242489+0000","last_undegraded":"2024-03-24T12:12:20.242489+0000","last_fullsized":"2024-03-24T12:12:20.242489+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:37.011493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:10:37.011493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"450'40","reported_seq":1168,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.709165+0000","last_change":"2024-03-24T12:11:32.027906+0000","last_active":"2024-03-24T12:12:24.709165+0000","last_peered":"2024-03-24T12:12:24.709165+0000","last_clean":"2024-03-24T12:12:24.709165+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:12:24.709165+0000","last_undegraded":"2024-03-24T12:12:24.709165+0000","last_fullsized":"2024-03-24T12:12:24.709165+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:11:32.027804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.027804+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:44.058819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":1178,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240629+0000","last_change":"2024-03-24T12:11:35.238528+0000","last_active":"2024-03-24T12:12:20.240629+0000","last_peered":"2024-03-24T12:12:20.240629+0000","last_clean":"2024-03-24T12:12:20.240629+0000","last_became_active":"2024-03-24T12:10:33.461798+0000","last_became_peered":"2024-03-24T12:10:33.461798+0000","last_unstale":"2024-03-24T12:12:20.240629+0000","last_undegraded":"2024-03-24T12:12:20.240629+0000","last_fullsized":"2024-03-24T12:12:20.240629+0000","mapping_epoch":64,"log_start":"83'130","ondisk_log_start":"83'130","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:35.238422+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:10:25.984109+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.238422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.501130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":755,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246373+0000","last_change":"2024-03-24T12:08:36.169985+0000","last_active":"2024-03-24T12:12:20.246373+0000","last_peered":"2024-03-24T12:12:20.246373+0000","last_clean":"2024-03-24T12:12:20.246373+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:12:20.246373+0000","last_undegraded":"2024-03-24T12:12:20.246373+0000","last_fullsized":"2024-03-24T12:12:20.246373+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:06:11.356759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:06:11.356759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":852,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.236792+0000","last_change":"2024-03-24T12:08:43.113674+0000","last_active":"2024-03-24T12:12:20.236792+0000","last_peered":"2024-03-24T12:12:20.236792+0000","last_clean":"2024-03-24T12:12:20.236792+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:12:20.236792+0000","last_undegraded":"2024-03-24T12:12:20.236792+0000","last_fullsized":"2024-03-24T12:12:20.236792+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:08:43.113510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.113510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":718,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.247102+0000","last_change":"2024-03-24T12:11:35.488009+0000","last_active":"2024-03-24T12:12:20.247102+0000","last_peered":"2024-03-24T12:12:20.247102+0000","last_clean":"2024-03-24T12:12:20.247102+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:12:20.247102+0000","last_undegraded":"2024-03-24T12:12:20.247102+0000","last_fullsized":"2024-03-24T12:12:20.247102+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:35.487872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.487872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:03.379847+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"450'8739","reported_seq":8813,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.569630+0000","last_change":"2024-03-24T12:10:42.558943+0000","last_active":"2024-03-24T12:12:26.569630+0000","last_peered":"2024-03-24T12:12:26.569630+0000","last_clean":"2024-03-24T12:12:26.569630+0000","last_became_active":"2024-03-24T12:10:42.558601+0000","last_became_peered":"2024-03-24T12:10:42.558601+0000","last_unstale":"2024-03-24T12:12:26.569630+0000","last_undegraded":"2024-03-24T12:12:26.569630+0000","last_fullsized":"2024-03-24T12:12:26.569630+0000","mapping_epoch":417,"log_start":"439'4300","ondisk_log_start":"439'4300","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511377408,"num_objects":7803,"num_object_clones":0,"num_object_copies":23409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8739,"num_write_kb":529344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":245,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246272+0000","last_change":"2024-03-24T12:12:18.565246+0000","last_active":"2024-03-24T12:12:20.246272+0000","last_peered":"2024-03-24T12:12:20.246272+0000","last_clean":"2024-03-24T12:12:20.246272+0000","last_became_active":"2024-03-24T12:08:36.162188+0000","last_became_peered":"2024-03-24T12:08:36.162188+0000","last_unstale":"2024-03-24T12:12:20.246272+0000","last_undegraded":"2024-03-24T12:12:20.246272+0000","last_fullsized":"2024-03-24T12:12:20.246272+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:18.565128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:12:18.565128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.801563+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"450'21","reported_seq":909,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.358914+0000","last_change":"2024-03-24T12:11:27.012324+0000","last_active":"2024-03-24T12:12:24.358914+0000","last_peered":"2024-03-24T12:12:24.358914+0000","last_clean":"2024-03-24T12:12:24.358914+0000","last_became_active":"2024-03-24T12:08:36.172879+0000","last_became_peered":"2024-03-24T12:08:36.172879+0000","last_unstale":"2024-03-24T12:12:24.358914+0000","last_undegraded":"2024-03-24T12:12:24.358914+0000","last_fullsized":"2024-03-24T12:12:24.358914+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:11:27.012204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.012204+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.287575+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":322,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.237449+0000","last_change":"2024-03-24T12:10:12.685963+0000","last_active":"2024-03-24T12:12:20.237449+0000","last_peered":"2024-03-24T12:12:20.237449+0000","last_clean":"2024-03-24T12:12:20.237449+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:12:20.237449+0000","last_undegraded":"2024-03-24T12:12:20.237449+0000","last_fullsized":"2024-03-24T12:12:20.237449+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.685815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.685815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":776,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:25.500125+0000","last_change":"2024-03-24T12:11:13.936448+0000","last_active":"2024-03-24T12:12:25.500125+0000","last_peered":"2024-03-24T12:12:25.500125+0000","last_clean":"2024-03-24T12:12:25.500125+0000","last_became_active":"2024-03-24T12:05:44.736437+0000","last_became_peered":"2024-03-24T12:05:44.736437+0000","last_unstale":"2024-03-24T12:12:25.500125+0000","last_undegraded":"2024-03-24T12:12:25.500125+0000","last_fullsized":"2024-03-24T12:12:25.500125+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:11:13.936256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.936256+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:31.155826+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":408,"num_read_kb":403,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":853,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242029+0000","last_change":"2024-03-24T12:11:31.240744+0000","last_active":"2024-03-24T12:12:20.242029+0000","last_peered":"2024-03-24T12:12:20.242029+0000","last_clean":"2024-03-24T12:12:20.242029+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:12:20.242029+0000","last_undegraded":"2024-03-24T12:12:20.242029+0000","last_fullsized":"2024-03-24T12:12:20.242029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:31.240650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:31.240650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:46.619564+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":1177,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241221+0000","last_change":"2024-03-24T12:11:30.174297+0000","last_active":"2024-03-24T12:12:20.241221+0000","last_peered":"2024-03-24T12:12:20.241221+0000","last_clean":"2024-03-24T12:12:20.241221+0000","last_became_active":"2024-03-24T12:10:23.398177+0000","last_became_peered":"2024-03-24T12:10:23.398177+0000","last_unstale":"2024-03-24T12:12:20.241221+0000","last_undegraded":"2024-03-24T12:12:20.241221+0000","last_fullsized":"2024-03-24T12:12:20.241221+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:30.173569+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:11:30.173569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:40.507040+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"450'8790","reported_seq":8954,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.577091+0000","last_change":"2024-03-24T12:12:14.049841+0000","last_active":"2024-03-24T12:12:26.577091+0000","last_peered":"2024-03-24T12:12:26.577091+0000","last_clean":"2024-03-24T12:12:26.577091+0000","last_became_active":"2024-03-24T12:10:42.559548+0000","last_became_peered":"2024-03-24T12:10:42.559548+0000","last_unstale":"2024-03-24T12:12:26.577091+0000","last_undegraded":"2024-03-24T12:12:26.577091+0000","last_fullsized":"2024-03-24T12:12:26.577091+0000","mapping_epoch":417,"log_start":"439'4400","ondisk_log_start":"439'4400","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7951","last_scrub_stamp":"2024-03-24T12:12:14.049510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.049510+0000","objects_scrubbed":7849,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:39.018031+0000","scrub_duration":2948,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519700480,"num_objects":7930,"num_object_clones":0,"num_object_copies":23790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8790,"num_write_kb":535040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":840,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246128+0000","last_change":"2024-03-24T12:11:38.437189+0000","last_active":"2024-03-24T12:12:20.246128+0000","last_peered":"2024-03-24T12:12:20.246128+0000","last_clean":"2024-03-24T12:12:20.246128+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:12:20.246128+0000","last_undegraded":"2024-03-24T12:12:20.246128+0000","last_fullsized":"2024-03-24T12:12:20.246128+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:38.437087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:11:38.437087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:46.086372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"450'54","reported_seq":1000,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.203041+0000","last_change":"2024-03-24T12:10:33.726470+0000","last_active":"2024-03-24T12:12:24.203041+0000","last_peered":"2024-03-24T12:12:24.203041+0000","last_clean":"2024-03-24T12:12:24.203041+0000","last_became_active":"2024-03-24T12:08:36.152866+0000","last_became_peered":"2024-03-24T12:08:36.152866+0000","last_unstale":"2024-03-24T12:12:24.203041+0000","last_undegraded":"2024-03-24T12:12:24.203041+0000","last_fullsized":"2024-03-24T12:12:24.203041+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'44","last_scrub_stamp":"2024-03-24T12:10:33.726357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:33.726357+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":727,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.245507+0000","last_change":"2024-03-24T12:11:19.470678+0000","last_active":"2024-03-24T12:12:20.245507+0000","last_peered":"2024-03-24T12:12:20.245507+0000","last_clean":"2024-03-24T12:12:20.245507+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:12:20.245507+0000","last_undegraded":"2024-03-24T12:12:20.245507+0000","last_fullsized":"2024-03-24T12:12:20.245507+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:19.470485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.470485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:38.111645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":830,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.238410+0000","last_change":"2024-03-24T12:11:28.156543+0000","last_active":"2024-03-24T12:12:20.238410+0000","last_peered":"2024-03-24T12:12:20.238410+0000","last_clean":"2024-03-24T12:12:20.238410+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:12:20.238410+0000","last_undegraded":"2024-03-24T12:12:20.238410+0000","last_fullsized":"2024-03-24T12:12:20.238410+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:28.156411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.156411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:30.950628+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":802,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.245633+0000","last_change":"2024-03-24T12:10:26.959642+0000","last_active":"2024-03-24T12:12:20.245633+0000","last_peered":"2024-03-24T12:12:20.245633+0000","last_clean":"2024-03-24T12:12:20.245633+0000","last_became_active":"2024-03-24T12:10:26.959434+0000","last_became_peered":"2024-03-24T12:10:26.959434+0000","last_unstale":"2024-03-24T12:12:20.245633+0000","last_undegraded":"2024-03-24T12:12:20.245633+0000","last_fullsized":"2024-03-24T12:12:20.245633+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:10.760367+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.760367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"450'8681","reported_seq":8757,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:26.709426+0000","last_change":"2024-03-24T12:10:42.558548+0000","last_active":"2024-03-24T12:12:26.709426+0000","last_peered":"2024-03-24T12:12:26.709426+0000","last_clean":"2024-03-24T12:12:26.709426+0000","last_became_active":"2024-03-24T12:10:42.558230+0000","last_became_peered":"2024-03-24T12:10:42.558230+0000","last_unstale":"2024-03-24T12:12:26.709426+0000","last_undegraded":"2024-03-24T12:12:26.709426+0000","last_fullsized":"2024-03-24T12:12:26.709426+0000","mapping_epoch":417,"log_start":"437'3700","ondisk_log_start":"437'3700","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508559384,"num_objects":7761,"num_object_clones":0,"num_object_copies":23283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7761,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8681,"num_write_kb":526081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":1101,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.249904+0000","last_change":"2024-03-24T12:11:19.363311+0000","last_active":"2024-03-24T12:12:20.249904+0000","last_peered":"2024-03-24T12:12:20.249904+0000","last_clean":"2024-03-24T12:12:20.249904+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:12:20.249904+0000","last_undegraded":"2024-03-24T12:12:20.249904+0000","last_fullsized":"2024-03-24T12:12:20.249904+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:11:19.362922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.362922+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:43.620435+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"450'35","reported_seq":920,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.693055+0000","last_change":"2024-03-24T12:11:24.029766+0000","last_active":"2024-03-24T12:12:24.693055+0000","last_peered":"2024-03-24T12:12:24.693055+0000","last_clean":"2024-03-24T12:12:24.693055+0000","last_became_active":"2024-03-24T12:08:36.174663+0000","last_became_peered":"2024-03-24T12:08:36.174663+0000","last_unstale":"2024-03-24T12:12:24.693055+0000","last_undegraded":"2024-03-24T12:12:24.693055+0000","last_fullsized":"2024-03-24T12:12:24.693055+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:11:24.029563+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:11:24.029563+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.448106+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":599,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246239+0000","last_change":"2024-03-24T12:12:17.361772+0000","last_active":"2024-03-24T12:12:20.246239+0000","last_peered":"2024-03-24T12:12:20.246239+0000","last_clean":"2024-03-24T12:12:20.246239+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:12:20.246239+0000","last_undegraded":"2024-03-24T12:12:20.246239+0000","last_fullsized":"2024-03-24T12:12:20.246239+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:17.361648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:12:17.361648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:23.972127+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":843,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248285+0000","last_change":"2024-03-24T12:11:26.507311+0000","last_active":"2024-03-24T12:12:20.248285+0000","last_peered":"2024-03-24T12:12:20.248285+0000","last_clean":"2024-03-24T12:12:20.248285+0000","last_became_active":"2024-03-24T12:08:36.145936+0000","last_became_peered":"2024-03-24T12:08:36.145936+0000","last_unstale":"2024-03-24T12:12:20.248285+0000","last_undegraded":"2024-03-24T12:12:20.248285+0000","last_fullsized":"2024-03-24T12:12:20.248285+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:26.507184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:11:26.507184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.897133+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":929,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248204+0000","last_change":"2024-03-24T12:11:18.911425+0000","last_active":"2024-03-24T12:12:20.248204+0000","last_peered":"2024-03-24T12:12:20.248204+0000","last_clean":"2024-03-24T12:12:20.248204+0000","last_became_active":"2024-03-24T12:10:36.965307+0000","last_became_peered":"2024-03-24T12:10:36.965307+0000","last_unstale":"2024-03-24T12:12:20.248204+0000","last_undegraded":"2024-03-24T12:12:20.248204+0000","last_fullsized":"2024-03-24T12:12:20.248204+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:18.911296+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.911296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:22.192298+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"450'8443","reported_seq":8519,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.834809+0000","last_change":"2024-03-24T12:10:42.558716+0000","last_active":"2024-03-24T12:12:23.834809+0000","last_peered":"2024-03-24T12:12:23.834809+0000","last_clean":"2024-03-24T12:12:23.834809+0000","last_became_active":"2024-03-24T12:10:42.558369+0000","last_became_peered":"2024-03-24T12:10:42.558369+0000","last_unstale":"2024-03-24T12:12:23.834809+0000","last_undegraded":"2024-03-24T12:12:23.834809+0000","last_fullsized":"2024-03-24T12:12:23.834809+0000","mapping_epoch":417,"log_start":"437'3400","ondisk_log_start":"437'3400","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520552448,"num_objects":7943,"num_object_clones":0,"num_object_copies":23829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8443,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":864,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242074+0000","last_change":"2024-03-24T12:11:19.790253+0000","last_active":"2024-03-24T12:12:20.242074+0000","last_peered":"2024-03-24T12:12:20.242074+0000","last_clean":"2024-03-24T12:12:20.242074+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:12:20.242074+0000","last_undegraded":"2024-03-24T12:12:20.242074+0000","last_fullsized":"2024-03-24T12:12:20.242074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:19.790086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.790086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.399995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"450'24","reported_seq":902,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.679104+0000","last_change":"2024-03-24T12:08:34.327181+0000","last_active":"2024-03-24T12:12:23.679104+0000","last_peered":"2024-03-24T12:12:23.679104+0000","last_clean":"2024-03-24T12:12:23.679104+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:12:23.679104+0000","last_undegraded":"2024-03-24T12:12:23.679104+0000","last_fullsized":"2024-03-24T12:12:23.679104+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:08:34.327037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.327037+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":858,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.704426+0000","last_change":"2024-03-24T12:12:23.704478+0000","last_active":"2024-03-24T12:12:23.704426+0000","last_peered":"2024-03-24T12:12:23.704426+0000","last_clean":"2024-03-24T12:12:23.704426+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:12:23.704426+0000","last_undegraded":"2024-03-24T12:12:23.704426+0000","last_fullsized":"2024-03-24T12:12:23.704426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:12:23.704204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.704204+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:27.358814+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":829,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240731+0000","last_change":"2024-03-24T12:12:14.425125+0000","last_active":"2024-03-24T12:12:20.240731+0000","last_peered":"2024-03-24T12:12:20.240731+0000","last_clean":"2024-03-24T12:12:20.240731+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:12:20.240731+0000","last_undegraded":"2024-03-24T12:12:20.240731+0000","last_fullsized":"2024-03-24T12:12:20.240731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:14.424990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.424990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:36.533158+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"450'25","reported_seq":889,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.740292+0000","last_change":"2024-03-24T12:11:40.464348+0000","last_active":"2024-03-24T12:12:23.740292+0000","last_peered":"2024-03-24T12:12:23.740292+0000","last_clean":"2024-03-24T12:12:23.740292+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:12:23.740292+0000","last_undegraded":"2024-03-24T12:12:23.740292+0000","last_fullsized":"2024-03-24T12:12:23.740292+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:40.464206+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:11:40.464206+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:55.591744+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":336,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248512+0000","last_change":"2024-03-24T12:11:20.390445+0000","last_active":"2024-03-24T12:12:20.248512+0000","last_peered":"2024-03-24T12:12:20.248512+0000","last_clean":"2024-03-24T12:12:20.248512+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:12:20.248512+0000","last_undegraded":"2024-03-24T12:12:20.248512+0000","last_fullsized":"2024-03-24T12:12:20.248512+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:20.390312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.390312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:44.725416+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":825,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.238887+0000","last_change":"2024-03-24T12:10:30.721092+0000","last_active":"2024-03-24T12:12:20.238887+0000","last_peered":"2024-03-24T12:12:20.238887+0000","last_clean":"2024-03-24T12:12:20.238887+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:12:20.238887+0000","last_undegraded":"2024-03-24T12:12:20.238887+0000","last_fullsized":"2024-03-24T12:12:20.238887+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:30.721003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:30.721003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":319,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242413+0000","last_change":"2024-03-24T12:07:50.278024+0000","last_active":"2024-03-24T12:12:20.242413+0000","last_peered":"2024-03-24T12:12:20.242413+0000","last_clean":"2024-03-24T12:12:20.242413+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:12:20.242413+0000","last_undegraded":"2024-03-24T12:12:20.242413+0000","last_fullsized":"2024-03-24T12:12:20.242413+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:50.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:07:50.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"450'18","reported_seq":864,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.554773+0000","last_change":"2024-03-24T12:10:25.685130+0000","last_active":"2024-03-24T12:12:24.554773+0000","last_peered":"2024-03-24T12:12:24.554773+0000","last_clean":"2024-03-24T12:12:24.554773+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:12:24.554773+0000","last_undegraded":"2024-03-24T12:12:24.554773+0000","last_fullsized":"2024-03-24T12:12:24.554773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:10:25.685009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.685009+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":862,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.245112+0000","last_change":"2024-03-24T12:11:43.032559+0000","last_active":"2024-03-24T12:12:20.245112+0000","last_peered":"2024-03-24T12:12:20.245112+0000","last_clean":"2024-03-24T12:12:20.245112+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:12:20.245112+0000","last_undegraded":"2024-03-24T12:12:20.245112+0000","last_fullsized":"2024-03-24T12:12:20.245112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.032403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.032403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:12.416512+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":675,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246380+0000","last_change":"2024-03-24T12:10:25.718633+0000","last_active":"2024-03-24T12:12:20.246380+0000","last_peered":"2024-03-24T12:12:20.246380+0000","last_clean":"2024-03-24T12:12:20.246380+0000","last_became_active":"2024-03-24T12:08:36.165715+0000","last_became_peered":"2024-03-24T12:08:36.165715+0000","last_unstale":"2024-03-24T12:12:20.246380+0000","last_undegraded":"2024-03-24T12:12:20.246380+0000","last_fullsized":"2024-03-24T12:12:20.246380+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:25.718555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.718555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":847,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240951+0000","last_change":"2024-03-24T12:10:19.030271+0000","last_active":"2024-03-24T12:12:20.240951+0000","last_peered":"2024-03-24T12:12:20.240951+0000","last_clean":"2024-03-24T12:12:20.240951+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:12:20.240951+0000","last_undegraded":"2024-03-24T12:12:20.240951+0000","last_fullsized":"2024-03-24T12:12:20.240951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.030172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.030172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"450'35","reported_seq":887,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.665912+0000","last_change":"2024-03-24T12:10:28.696002+0000","last_active":"2024-03-24T12:12:24.665912+0000","last_peered":"2024-03-24T12:12:24.665912+0000","last_clean":"2024-03-24T12:12:24.665912+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:12:24.665912+0000","last_undegraded":"2024-03-24T12:12:24.665912+0000","last_fullsized":"2024-03-24T12:12:24.665912+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:28.695890+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:10:28.695890+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":726,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246610+0000","last_change":"2024-03-24T12:11:32.439386+0000","last_active":"2024-03-24T12:12:20.246610+0000","last_peered":"2024-03-24T12:12:20.246610+0000","last_clean":"2024-03-24T12:12:20.246610+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:12:20.246610+0000","last_undegraded":"2024-03-24T12:12:20.246610+0000","last_fullsized":"2024-03-24T12:12:20.246610+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.439301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.439301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:32.556201+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":984,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246983+0000","last_change":"2024-03-24T12:10:15.754439+0000","last_active":"2024-03-24T12:12:20.246983+0000","last_peered":"2024-03-24T12:12:20.246983+0000","last_clean":"2024-03-24T12:12:20.246983+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:12:20.246983+0000","last_undegraded":"2024-03-24T12:12:20.246983+0000","last_fullsized":"2024-03-24T12:12:20.246983+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:15.754352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:15.754352+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":835,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.249806+0000","last_change":"2024-03-24T12:11:17.508810+0000","last_active":"2024-03-24T12:12:20.249806+0000","last_peered":"2024-03-24T12:12:20.249806+0000","last_clean":"2024-03-24T12:12:20.249806+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:12:20.249806+0000","last_undegraded":"2024-03-24T12:12:20.249806+0000","last_fullsized":"2024-03-24T12:12:20.249806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:17.508697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:17.508697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:21.358451+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"450'19","reported_seq":749,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.783025+0000","last_change":"2024-03-24T12:08:36.154109+0000","last_active":"2024-03-24T12:12:23.783025+0000","last_peered":"2024-03-24T12:12:23.783025+0000","last_clean":"2024-03-24T12:12:23.783025+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:12:23.783025+0000","last_undegraded":"2024-03-24T12:12:23.783025+0000","last_fullsized":"2024-03-24T12:12:23.783025+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:05:16.245204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:05:16.245204+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":721,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.249435+0000","last_change":"2024-03-24T12:10:34.022259+0000","last_active":"2024-03-24T12:12:20.249435+0000","last_peered":"2024-03-24T12:12:20.249435+0000","last_clean":"2024-03-24T12:12:20.249435+0000","last_became_active":"2024-03-24T12:10:30.443191+0000","last_became_peered":"2024-03-24T12:10:30.443191+0000","last_unstale":"2024-03-24T12:12:20.249435+0000","last_undegraded":"2024-03-24T12:12:20.249435+0000","last_fullsized":"2024-03-24T12:12:20.249435+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:34.022182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:34.022182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":1094,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.438647+0000","last_change":"2024-03-24T12:10:19.664090+0000","last_active":"2024-03-24T12:12:20.438647+0000","last_peered":"2024-03-24T12:12:20.438647+0000","last_clean":"2024-03-24T12:12:20.438647+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:12:20.438647+0000","last_undegraded":"2024-03-24T12:12:20.438647+0000","last_fullsized":"2024-03-24T12:12:20.438647+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:10:19.664009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.664009+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":835,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.243921+0000","last_change":"2024-03-24T12:10:17.566669+0000","last_active":"2024-03-24T12:12:20.243921+0000","last_peered":"2024-03-24T12:12:20.243921+0000","last_clean":"2024-03-24T12:12:20.243921+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:12:20.243921+0000","last_undegraded":"2024-03-24T12:12:20.243921+0000","last_fullsized":"2024-03-24T12:12:20.243921+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:17.566562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:10:17.566562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"450'21","reported_seq":652,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.184821+0000","last_change":"2024-03-24T12:11:28.054589+0000","last_active":"2024-03-24T12:12:24.184821+0000","last_peered":"2024-03-24T12:12:24.184821+0000","last_clean":"2024-03-24T12:12:24.184821+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:12:24.184821+0000","last_undegraded":"2024-03-24T12:12:24.184821+0000","last_fullsized":"2024-03-24T12:12:24.184821+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:11:28.054480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.054480+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.923586+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":863,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248753+0000","last_change":"2024-03-24T12:12:15.943971+0000","last_active":"2024-03-24T12:12:20.248753+0000","last_peered":"2024-03-24T12:12:20.248753+0000","last_clean":"2024-03-24T12:12:20.248753+0000","last_became_active":"2024-03-24T12:08:36.152805+0000","last_became_peered":"2024-03-24T12:08:36.152805+0000","last_unstale":"2024-03-24T12:12:20.248753+0000","last_undegraded":"2024-03-24T12:12:20.248753+0000","last_fullsized":"2024-03-24T12:12:20.248753+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:15.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:12:15.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:16.265713+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":839,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240651+0000","last_change":"2024-03-24T12:11:37.185727+0000","last_active":"2024-03-24T12:12:20.240651+0000","last_peered":"2024-03-24T12:12:20.240651+0000","last_clean":"2024-03-24T12:12:20.240651+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:12:20.240651+0000","last_undegraded":"2024-03-24T12:12:20.240651+0000","last_fullsized":"2024-03-24T12:12:20.240651+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:37.185629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.185629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:37.902893+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":844,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248180+0000","last_change":"2024-03-24T12:11:31.446015+0000","last_active":"2024-03-24T12:12:20.248180+0000","last_peered":"2024-03-24T12:12:20.248180+0000","last_clean":"2024-03-24T12:12:20.248180+0000","last_became_active":"2024-03-24T12:08:36.146995+0000","last_became_peered":"2024-03-24T12:08:36.146995+0000","last_unstale":"2024-03-24T12:12:20.248180+0000","last_undegraded":"2024-03-24T12:12:20.248180+0000","last_fullsized":"2024-03-24T12:12:20.248180+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:31.445925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:30.836662+0000","last_clean_scrub_stamp":"2024-03-24T12:11:31.445925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.589782+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"450'27","reported_seq":877,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.605044+0000","last_change":"2024-03-24T12:11:28.574992+0000","last_active":"2024-03-24T12:12:24.605044+0000","last_peered":"2024-03-24T12:12:24.605044+0000","last_clean":"2024-03-24T12:12:24.605044+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:12:24.605044+0000","last_undegraded":"2024-03-24T12:12:24.605044+0000","last_fullsized":"2024-03-24T12:12:24.605044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:28.574841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.574841+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:40.417840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":881,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241013+0000","last_change":"2024-03-24T12:12:10.078427+0000","last_active":"2024-03-24T12:12:20.241013+0000","last_peered":"2024-03-24T12:12:20.241013+0000","last_clean":"2024-03-24T12:12:20.241013+0000","last_became_active":"2024-03-24T12:08:36.174093+0000","last_became_peered":"2024-03-24T12:08:36.174093+0000","last_unstale":"2024-03-24T12:12:20.241013+0000","last_undegraded":"2024-03-24T12:12:20.241013+0000","last_fullsized":"2024-03-24T12:12:20.241013+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:10.078142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:10.078142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:39.723464+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":477,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.245333+0000","last_change":"2024-03-24T12:09:38.661845+0000","last_active":"2024-03-24T12:12:20.245333+0000","last_peered":"2024-03-24T12:12:20.245333+0000","last_clean":"2024-03-24T12:12:20.245333+0000","last_became_active":"2024-03-24T12:08:36.166281+0000","last_became_peered":"2024-03-24T12:08:36.166281+0000","last_unstale":"2024-03-24T12:12:20.245333+0000","last_undegraded":"2024-03-24T12:12:20.245333+0000","last_fullsized":"2024-03-24T12:12:20.245333+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.661755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.661755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":830,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.236938+0000","last_change":"2024-03-24T12:10:10.700383+0000","last_active":"2024-03-24T12:12:20.236938+0000","last_peered":"2024-03-24T12:12:20.236938+0000","last_clean":"2024-03-24T12:12:20.236938+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:12:20.236938+0000","last_undegraded":"2024-03-24T12:12:20.236938+0000","last_fullsized":"2024-03-24T12:12:20.236938+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:10.700237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.700237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"450'14","reported_seq":865,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.685625+0000","last_change":"2024-03-24T12:12:15.299189+0000","last_active":"2024-03-24T12:12:23.685625+0000","last_peered":"2024-03-24T12:12:23.685625+0000","last_clean":"2024-03-24T12:12:23.685625+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:12:23.685625+0000","last_undegraded":"2024-03-24T12:12:23.685625+0000","last_fullsized":"2024-03-24T12:12:23.685625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:12:15.299057+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:12:15.299057+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:15.663338+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":739,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246175+0000","last_change":"2024-03-24T12:10:12.784984+0000","last_active":"2024-03-24T12:12:20.246175+0000","last_peered":"2024-03-24T12:12:20.246175+0000","last_clean":"2024-03-24T12:12:20.246175+0000","last_became_active":"2024-03-24T12:08:36.166947+0000","last_became_peered":"2024-03-24T12:08:36.166947+0000","last_unstale":"2024-03-24T12:12:20.246175+0000","last_undegraded":"2024-03-24T12:12:20.246175+0000","last_fullsized":"2024-03-24T12:12:20.246175+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.784866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.784866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":846,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.245037+0000","last_change":"2024-03-24T12:10:37.907768+0000","last_active":"2024-03-24T12:12:20.245037+0000","last_peered":"2024-03-24T12:12:20.245037+0000","last_clean":"2024-03-24T12:12:20.245037+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:12:20.245037+0000","last_undegraded":"2024-03-24T12:12:20.245037+0000","last_fullsized":"2024-03-24T12:12:20.245037+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:37.907635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:37.907635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":844,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241276+0000","last_change":"2024-03-24T12:10:35.026578+0000","last_active":"2024-03-24T12:12:20.241276+0000","last_peered":"2024-03-24T12:12:20.241276+0000","last_clean":"2024-03-24T12:12:20.241276+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:12:20.241276+0000","last_undegraded":"2024-03-24T12:12:20.241276+0000","last_fullsized":"2024-03-24T12:12:20.241276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:35.026455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.026455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"450'27","reported_seq":872,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.754254+0000","last_change":"2024-03-24T12:10:22.696939+0000","last_active":"2024-03-24T12:12:24.754254+0000","last_peered":"2024-03-24T12:12:24.754254+0000","last_clean":"2024-03-24T12:12:24.754254+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:12:24.754254+0000","last_undegraded":"2024-03-24T12:12:24.754254+0000","last_fullsized":"2024-03-24T12:12:24.754254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:10:22.696749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:22.696749+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":861,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248304+0000","last_change":"2024-03-24T12:12:13.011602+0000","last_active":"2024-03-24T12:12:20.248304+0000","last_peered":"2024-03-24T12:12:20.248304+0000","last_clean":"2024-03-24T12:12:20.248304+0000","last_became_active":"2024-03-24T12:08:36.157290+0000","last_became_peered":"2024-03-24T12:08:36.157290+0000","last_unstale":"2024-03-24T12:12:20.248304+0000","last_undegraded":"2024-03-24T12:12:20.248304+0000","last_fullsized":"2024-03-24T12:12:20.248304+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:13.011469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_clean_scrub_stamp":"2024-03-24T12:12:13.011469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:32.523753+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":856,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242562+0000","last_change":"2024-03-24T12:11:45.406146+0000","last_active":"2024-03-24T12:12:20.242562+0000","last_peered":"2024-03-24T12:12:20.242562+0000","last_clean":"2024-03-24T12:12:20.242562+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:12:20.242562+0000","last_undegraded":"2024-03-24T12:12:20.242562+0000","last_fullsized":"2024-03-24T12:12:20.242562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:45.406037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.406037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.099159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":821,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240813+0000","last_change":"2024-03-24T12:10:13.863177+0000","last_active":"2024-03-24T12:12:20.240813+0000","last_peered":"2024-03-24T12:12:20.240813+0000","last_clean":"2024-03-24T12:12:20.240813+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:12:20.240813+0000","last_undegraded":"2024-03-24T12:12:20.240813+0000","last_fullsized":"2024-03-24T12:12:20.240813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.862457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.862457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"450'29","reported_seq":886,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.799328+0000","last_change":"2024-03-24T12:10:35.061280+0000","last_active":"2024-03-24T12:12:23.799328+0000","last_peered":"2024-03-24T12:12:23.799328+0000","last_clean":"2024-03-24T12:12:23.799328+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:12:23.799328+0000","last_undegraded":"2024-03-24T12:12:23.799328+0000","last_fullsized":"2024-03-24T12:12:23.799328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:35.061151+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.061151+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":31,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":691,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246754+0000","last_change":"2024-03-24T12:10:24.690361+0000","last_active":"2024-03-24T12:12:20.246754+0000","last_peered":"2024-03-24T12:12:20.246754+0000","last_clean":"2024-03-24T12:12:20.246754+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:12:20.246754+0000","last_undegraded":"2024-03-24T12:12:20.246754+0000","last_fullsized":"2024-03-24T12:12:20.246754+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.690274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.690274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":1102,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.439640+0000","last_change":"2024-03-24T12:11:30.060710+0000","last_active":"2024-03-24T12:12:20.439640+0000","last_peered":"2024-03-24T12:12:20.439640+0000","last_clean":"2024-03-24T12:12:20.439640+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:12:20.439640+0000","last_undegraded":"2024-03-24T12:12:20.439640+0000","last_fullsized":"2024-03-24T12:12:20.439640+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:11:30.060478+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:11:30.060478+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.661103+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":863,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242446+0000","last_change":"2024-03-24T12:10:27.009973+0000","last_active":"2024-03-24T12:12:20.242446+0000","last_peered":"2024-03-24T12:12:20.242446+0000","last_clean":"2024-03-24T12:12:20.242446+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:12:20.242446+0000","last_undegraded":"2024-03-24T12:12:20.242446+0000","last_fullsized":"2024-03-24T12:12:20.242446+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:10:27.009877+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:27.009877+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"450'27","reported_seq":911,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.772781+0000","last_change":"2024-03-24T12:12:19.829113+0000","last_active":"2024-03-24T12:12:24.772781+0000","last_peered":"2024-03-24T12:12:24.772781+0000","last_clean":"2024-03-24T12:12:24.772781+0000","last_became_active":"2024-03-24T12:08:36.156563+0000","last_became_peered":"2024-03-24T12:08:36.156563+0000","last_unstale":"2024-03-24T12:12:24.772781+0000","last_undegraded":"2024-03-24T12:12:24.772781+0000","last_fullsized":"2024-03-24T12:12:24.772781+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:12:19.829011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.829011+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:49.797083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":834,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.237151+0000","last_change":"2024-03-24T12:10:23.652133+0000","last_active":"2024-03-24T12:12:20.237151+0000","last_peered":"2024-03-24T12:12:20.237151+0000","last_clean":"2024-03-24T12:12:20.237151+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:12:20.237151+0000","last_undegraded":"2024-03-24T12:12:20.237151+0000","last_fullsized":"2024-03-24T12:12:20.237151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:10:23.652045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.652045+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":854,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.250272+0000","last_change":"2024-03-24T12:12:19.386952+0000","last_active":"2024-03-24T12:12:20.250272+0000","last_peered":"2024-03-24T12:12:20.250272+0000","last_clean":"2024-03-24T12:12:20.250272+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:12:20.250272+0000","last_undegraded":"2024-03-24T12:12:20.250272+0000","last_fullsized":"2024-03-24T12:12:20.250272+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:19.386800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.386800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:31.593985+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":439,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.247084+0000","last_change":"2024-03-24T12:11:15.598297+0000","last_active":"2024-03-24T12:12:20.247084+0000","last_peered":"2024-03-24T12:12:20.247084+0000","last_clean":"2024-03-24T12:12:20.247084+0000","last_became_active":"2024-03-24T12:08:36.166973+0000","last_became_peered":"2024-03-24T12:08:36.166973+0000","last_unstale":"2024-03-24T12:12:20.247084+0000","last_undegraded":"2024-03-24T12:12:20.247084+0000","last_fullsized":"2024-03-24T12:12:20.247084+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:15.598041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:11:15.598041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.713677+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"450'46","reported_seq":809,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.422818+0000","last_change":"2024-03-24T12:11:35.124433+0000","last_active":"2024-03-24T12:12:24.422818+0000","last_peered":"2024-03-24T12:12:24.422818+0000","last_clean":"2024-03-24T12:12:24.422818+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:12:24.422818+0000","last_undegraded":"2024-03-24T12:12:24.422818+0000","last_fullsized":"2024-03-24T12:12:24.422818+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:11:35.124291+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.124291+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.181527+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":854,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.247796+0000","last_change":"2024-03-24T12:11:15.939612+0000","last_active":"2024-03-24T12:12:20.247796+0000","last_peered":"2024-03-24T12:12:20.247796+0000","last_clean":"2024-03-24T12:12:20.247796+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:12:20.247796+0000","last_undegraded":"2024-03-24T12:12:20.247796+0000","last_fullsized":"2024-03-24T12:12:20.247796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:15.939499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:11:15.939499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:35.965808+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":591,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.243953+0000","last_change":"2024-03-24T12:07:53.354298+0000","last_active":"2024-03-24T12:12:20.243953+0000","last_peered":"2024-03-24T12:12:20.243953+0000","last_clean":"2024-03-24T12:12:20.243953+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:12:20.243953+0000","last_undegraded":"2024-03-24T12:12:20.243953+0000","last_fullsized":"2024-03-24T12:12:20.243953+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:07:53.354213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":840,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.243186+0000","last_change":"2024-03-24T12:10:13.573691+0000","last_active":"2024-03-24T12:12:20.243186+0000","last_peered":"2024-03-24T12:12:20.243186+0000","last_clean":"2024-03-24T12:12:20.243186+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:12:20.243186+0000","last_undegraded":"2024-03-24T12:12:20.243186+0000","last_fullsized":"2024-03-24T12:12:20.243186+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.573504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.573504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"450'29","reported_seq":901,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.824106+0000","last_change":"2024-03-24T12:11:41.236799+0000","last_active":"2024-03-24T12:12:24.824106+0000","last_peered":"2024-03-24T12:12:24.824106+0000","last_clean":"2024-03-24T12:12:24.824106+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:12:24.824106+0000","last_undegraded":"2024-03-24T12:12:24.824106+0000","last_fullsized":"2024-03-24T12:12:24.824106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:11:41.236628+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:11:41.236628+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:56.929372+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":847,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240551+0000","last_change":"2024-03-24T12:11:20.004402+0000","last_active":"2024-03-24T12:12:20.240551+0000","last_peered":"2024-03-24T12:12:20.240551+0000","last_clean":"2024-03-24T12:12:20.240551+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:12:20.240551+0000","last_undegraded":"2024-03-24T12:12:20.240551+0000","last_fullsized":"2024-03-24T12:12:20.240551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:11:20.004266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.004266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:34.289579+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":842,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.242552+0000","last_change":"2024-03-24T12:11:23.602693+0000","last_active":"2024-03-24T12:12:20.242552+0000","last_peered":"2024-03-24T12:12:20.242552+0000","last_clean":"2024-03-24T12:12:20.242552+0000","last_became_active":"2024-03-24T12:11:16.022628+0000","last_became_peered":"2024-03-24T12:11:16.022628+0000","last_unstale":"2024-03-24T12:12:20.242552+0000","last_undegraded":"2024-03-24T12:12:20.242552+0000","last_fullsized":"2024-03-24T12:12:20.242552+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:23.602435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.602435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:35.820551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":831,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.238691+0000","last_change":"2024-03-24T12:08:39.416538+0000","last_active":"2024-03-24T12:12:20.238691+0000","last_peered":"2024-03-24T12:12:20.238691+0000","last_clean":"2024-03-24T12:12:20.238691+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:12:20.238691+0000","last_undegraded":"2024-03-24T12:12:20.238691+0000","last_fullsized":"2024-03-24T12:12:20.238691+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:39.416426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:39.416426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"450'21","reported_seq":876,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.214883+0000","last_change":"2024-03-24T12:12:09.683485+0000","last_active":"2024-03-24T12:12:24.214883+0000","last_peered":"2024-03-24T12:12:24.214883+0000","last_clean":"2024-03-24T12:12:24.214883+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:12:24.214883+0000","last_undegraded":"2024-03-24T12:12:24.214883+0000","last_fullsized":"2024-03-24T12:12:24.214883+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:12:09.683324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:09.683324+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:20.470987+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":876,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241204+0000","last_change":"2024-03-24T12:11:41.354046+0000","last_active":"2024-03-24T12:12:20.241204+0000","last_peered":"2024-03-24T12:12:20.241204+0000","last_clean":"2024-03-24T12:12:20.241204+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:12:20.241204+0000","last_undegraded":"2024-03-24T12:12:20.241204+0000","last_fullsized":"2024-03-24T12:12:20.241204+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:41.353830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:11:41.353830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:55.943062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":752,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.246284+0000","last_change":"2024-03-24T12:10:23.722476+0000","last_active":"2024-03-24T12:12:20.246284+0000","last_peered":"2024-03-24T12:12:20.246284+0000","last_clean":"2024-03-24T12:12:20.246284+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:12:20.246284+0000","last_undegraded":"2024-03-24T12:12:20.246284+0000","last_fullsized":"2024-03-24T12:12:20.246284+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:23.722321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.722321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":827,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.248769+0000","last_change":"2024-03-24T12:10:20.093443+0000","last_active":"2024-03-24T12:12:20.248769+0000","last_peered":"2024-03-24T12:12:20.248769+0000","last_clean":"2024-03-24T12:12:20.248769+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:12:20.248769+0000","last_undegraded":"2024-03-24T12:12:20.248769+0000","last_fullsized":"2024-03-24T12:12:20.248769+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:20.093338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:10:20.093338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"450'11","reported_seq":860,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.072014+0000","last_change":"2024-03-24T12:11:36.552488+0000","last_active":"2024-03-24T12:12:24.072014+0000","last_peered":"2024-03-24T12:12:24.072014+0000","last_clean":"2024-03-24T12:12:24.072014+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:12:24.072014+0000","last_undegraded":"2024-03-24T12:12:24.072014+0000","last_fullsized":"2024-03-24T12:12:24.072014+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:11:36.552333+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:11:36.552333+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:01.517180+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":872,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241935+0000","last_change":"2024-03-24T12:11:13.237903+0000","last_active":"2024-03-24T12:12:20.241935+0000","last_peered":"2024-03-24T12:12:20.241935+0000","last_clean":"2024-03-24T12:12:20.241935+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:12:20.241935+0000","last_undegraded":"2024-03-24T12:12:20.241935+0000","last_fullsized":"2024-03-24T12:12:20.241935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:13.237829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.237829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.385120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":864,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241013+0000","last_change":"2024-03-24T12:11:23.134759+0000","last_active":"2024-03-24T12:12:20.241013+0000","last_peered":"2024-03-24T12:12:20.241013+0000","last_clean":"2024-03-24T12:12:20.241013+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:12:20.241013+0000","last_undegraded":"2024-03-24T12:12:20.241013+0000","last_fullsized":"2024-03-24T12:12:20.241013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:23.134643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.011992+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.134643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:29.103157+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":758,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.245413+0000","last_change":"2024-03-24T12:10:13.888112+0000","last_active":"2024-03-24T12:12:20.245413+0000","last_peered":"2024-03-24T12:12:20.245413+0000","last_clean":"2024-03-24T12:12:20.245413+0000","last_became_active":"2024-03-24T12:08:36.164166+0000","last_became_peered":"2024-03-24T12:08:36.164166+0000","last_unstale":"2024-03-24T12:12:20.245413+0000","last_undegraded":"2024-03-24T12:12:20.245413+0000","last_fullsized":"2024-03-24T12:12:20.245413+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.888002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.888002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"450'25","reported_seq":880,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:23.559806+0000","last_change":"2024-03-24T12:11:44.652906+0000","last_active":"2024-03-24T12:12:23.559806+0000","last_peered":"2024-03-24T12:12:23.559806+0000","last_clean":"2024-03-24T12:12:23.559806+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:12:23.559806+0000","last_undegraded":"2024-03-24T12:12:23.559806+0000","last_fullsized":"2024-03-24T12:12:23.559806+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:44.652731+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:11:44.652731+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:13.964162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":844,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.238360+0000","last_change":"2024-03-24T12:08:40.183479+0000","last_active":"2024-03-24T12:12:20.238360+0000","last_peered":"2024-03-24T12:12:20.238360+0000","last_clean":"2024-03-24T12:12:20.238360+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:12:20.238360+0000","last_undegraded":"2024-03-24T12:12:20.238360+0000","last_fullsized":"2024-03-24T12:12:20.238360+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.183311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.183311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":328,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:22.837622+0000","last_change":"2024-03-24T12:12:22.837708+0000","last_active":"2024-03-24T12:12:22.837622+0000","last_peered":"2024-03-24T12:12:22.837622+0000","last_clean":"2024-03-24T12:12:22.837622+0000","last_became_active":"2024-03-24T12:08:36.152717+0000","last_became_peered":"2024-03-24T12:08:36.152717+0000","last_unstale":"2024-03-24T12:12:22.837622+0000","last_undegraded":"2024-03-24T12:12:22.837622+0000","last_fullsized":"2024-03-24T12:12:22.837622+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:22.837495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:22.837495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.498764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":835,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.240589+0000","last_change":"2024-03-24T12:10:12.547927+0000","last_active":"2024-03-24T12:12:20.240589+0000","last_peered":"2024-03-24T12:12:20.240589+0000","last_clean":"2024-03-24T12:12:20.240589+0000","last_became_active":"2024-03-24T12:08:36.165334+0000","last_became_peered":"2024-03-24T12:08:36.165334+0000","last_unstale":"2024-03-24T12:12:20.240589+0000","last_undegraded":"2024-03-24T12:12:20.240589+0000","last_fullsized":"2024-03-24T12:12:20.240589+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.547803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.547803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"450'46","reported_seq":928,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:24.800860+0000","last_change":"2024-03-24T12:08:47.101220+0000","last_active":"2024-03-24T12:12:24.800860+0000","last_peered":"2024-03-24T12:12:24.800860+0000","last_clean":"2024-03-24T12:12:24.800860+0000","last_became_active":"2024-03-24T12:08:36.154117+0000","last_became_peered":"2024-03-24T12:08:36.154117+0000","last_unstale":"2024-03-24T12:12:24.800860+0000","last_undegraded":"2024-03-24T12:12:24.800860+0000","last_fullsized":"2024-03-24T12:12:24.800860+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:47.101082+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:08:47.101082+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":848,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.249415+0000","last_change":"2024-03-24T12:12:12.477784+0000","last_active":"2024-03-24T12:12:20.249415+0000","last_peered":"2024-03-24T12:12:20.249415+0000","last_clean":"2024-03-24T12:12:20.249415+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:12:20.249415+0000","last_undegraded":"2024-03-24T12:12:20.249415+0000","last_fullsized":"2024-03-24T12:12:20.249415+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:12.477654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:12:12.477654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:27.896776+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":303,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.241566+0000","last_change":"2024-03-24T12:08:33.256620+0000","last_active":"2024-03-24T12:12:20.241566+0000","last_peered":"2024-03-24T12:12:20.241566+0000","last_clean":"2024-03-24T12:12:20.241566+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:12:20.241566+0000","last_undegraded":"2024-03-24T12:12:20.241566+0000","last_fullsized":"2024-03-24T12:12:20.241566+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:33.255322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:08:33.255322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":829,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-24T12:12:20.236456+0000","last_change":"2024-03-24T12:11:44.456204+0000","last_active":"2024-03-24T12:12:20.236456+0000","last_peered":"2024-03-24T12:12:20.236456+0000","last_clean":"2024-03-24T12:12:20.236456+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:12:20.236456+0000","last_undegraded":"2024-03-24T12:12:20.236456+0000","last_fullsized":"2024-03-24T12:12:20.236456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:44.455779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:44.455779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:53.752100+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":8206745624,"num_objects":125226,"num_object_clones":0,"num_object_copies":375678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125226,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139036,"num_write_kb":8456321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24650002432,"data_stored":24649990216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77336,"ondisk_log_size":77336,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":390,"ondisk_log_size":390,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":408,"num_read_kb":403,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1509,"num_read_kb":1266,"num_write":961,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":1050,"ondisk_log_size":1050,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005302,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3816264,"kb_used_data":3022912,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":89928888,"statfs":{"total":95995035648,"available":92087181312,"internally_reserved":0,"allocated":3095461888,"data_stored":3094763002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":812382666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332349,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4730908,"kb_used_data":3999956,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":89014244,"statfs":{"total":95995035648,"available":91150585856,"internally_reserved":0,"allocated":4095954944,"data_stored":4095235888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748419526},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575125,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5828564,"kb_used_data":4985228,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":87916588,"statfs":{"total":95995035648,"available":90026586112,"internally_reserved":0,"allocated":5104873472,"data_stored":5104161719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738650,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3969680,"kb_used_data":3099976,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":89775472,"statfs":{"total":95995035648,"available":91930083328,"internally_reserved":0,"allocated":3174375424,"data_stored":3173667712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411601,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1577644,"kb_used_data":986980,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92167508,"statfs":{"total":95995035648,"available":94379528192,"internally_reserved":0,"allocated":1010667520,"data_stored":1009996577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117259,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4376164,"kb_used_data":3514780,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":89368988,"statfs":{"total":95995035648,"available":91513843712,"internally_reserved":0,"allocated":3599134720,"data_stored":3598407953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":882047431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921030,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3231576,"kb_used_data":2496528,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":90513576,"statfs":{"total":95995035648,"available":92685901824,"internally_reserved":0,"allocated":2556444672,"data_stored":2555728858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455535,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2368792,"kb_used_data":2055888,"kb_used_omap":7,"kb_used_meta":312888,"kb_avail":91376360,"statfs":{"total":95995035648,"available":93569392640,"internally_reserved":0,"allocated":2105229312,"data_stored":2104514253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":320397424},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3160735744,"data_stored":3160735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5101457408,"data_stored":5101453336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1007288320,"data_stored":1007288320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3587178496,"data_stored":3587178496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2531790848,"data_stored":2531786776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2087387136,"data_stored":2087387136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4093644800,"data_stored":4093640728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3080519680,"data_stored":3080519680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:12:32.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-24T12:12:32.077 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:32.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:32.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:32.227 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:32.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:32.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:32.363 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:32.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:32.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:32.477 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:12:32.504 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:32.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:32.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T12:12:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T12:12:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T12:12:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T12:12:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T12:12:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1826234200' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:12:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T12:12:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:32 smithi193 ceph-mon[33593]: pgmap v961: 158 pgs: 158 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-03-24T12:12:32.605 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:32.639 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:32.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:32.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:32.777 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:32.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T12:12:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T12:12:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1826234200' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[30223]: pgmap v961: 158 pgs: 158 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T12:12:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:32.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1826234200' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:12:32.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T12:12:32.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:32 smithi064 ceph-mon[42243]: pgmap v961: 158 pgs: 158 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-03-24T12:12:32.857 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:32.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:32.932 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:32.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:33.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:33.071 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:33.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:33.172 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:33.215 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:33.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:33.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:33.350 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:33.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:33.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:33.482 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:33.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:33.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:33.625 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:33.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:33.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:33.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:33.756 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:33.770 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:33.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:33.871 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:33.897 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:12:33.903 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:33.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:33.940 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:12:33.987 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:12:34.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:34.031 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:12:34.035 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:34.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:34.071 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:12:34.109 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:12:34.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:34.151 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:12:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:34.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:34.191 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:12:34.198 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:12:34.198 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:12:34.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:12:34.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:12:34.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:12:34.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:12:34.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:12:34.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:12:34.283 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:34.315 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:34.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:34.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:34.451 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:34.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:34.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:34.589 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:34.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:34.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:34.728 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:34.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:34.760 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:34 smithi193 ceph-mon[33593]: pgmap v962: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 44 KiB/s rd, 1.2 MiB/s wr, 1.22k op/s 2024-03-24T12:12:34.760 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:34 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/886087273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:34.816 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:12:34.816 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:12:34.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:34.866 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:34.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:34.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:12:34.934 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:34 smithi064 ceph-mon[42243]: pgmap v962: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 44 KiB/s rd, 1.2 MiB/s wr, 1.22k op/s 2024-03-24T12:12:34.934 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:34 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/886087273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:34.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:34 smithi064 ceph-mon[30223]: pgmap v962: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 44 KiB/s rd, 1.2 MiB/s wr, 1.22k op/s 2024-03-24T12:12:34.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:34 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/886087273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:34.967 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:35.002 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:35.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:35.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:35.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:35.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:35.291 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:35.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:35.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:35.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:35.442 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:35.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:35.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:35.575 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:35.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:35.586 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:12:35.676 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:35.717 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:35.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:35.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:35.852 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:35.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:35.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:35.991 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:35.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:36.038 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:36.065 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:36.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:36.140 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:36.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:36.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:36.285 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:36.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:36.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:36.423 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:36.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:36.526 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:36.557 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:36.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:36.660 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:36.690 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:36.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:36.793 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:36.830 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:36.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:36.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:36 smithi064 ceph-mon[30223]: pgmap v963: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:36.873 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:36 smithi064 ceph-mon[42243]: pgmap v963: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:36.899 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:36 smithi193 ceph-mon[33593]: pgmap v963: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:36.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:36.970 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:36.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:37.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:37.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:37.110 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:37.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:37.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:37.255 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:37.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:37.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:37.395 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:37.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:37.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:37.536 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:37.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:37.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:12:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:12:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:37.637 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:37.669 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:37.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:37.698 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 1 pgp_num to 11 2024-03-24T12:12:37.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:37 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3493743138' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:12:37.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:37 smithi193 ceph-mon[33593]: from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:12:37.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:37.808 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:37.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:37.909 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:37.943 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:37.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:37.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:12:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:12:38.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:12:38.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:37 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3493743138' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:12:38.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:37 smithi064 ceph-mon[30223]: from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:12:38.044 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:37 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3493743138' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:12:38.044 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:37 smithi064 ceph-mon[42243]: from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:12:38.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:38.089 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:38.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:38.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:38.238 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:38.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:38.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:38.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:38.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:38.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:38.524 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:38.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:38.544 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:12:38.545 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:12:38.626 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:38.660 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:38.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:38.762 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:38.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:38 smithi064 ceph-mon[30223]: pgmap v964: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:38.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:38 smithi064 ceph-mon[30223]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T12:12:38.796 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:38 smithi064 ceph-mon[30223]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T12:12:38.797 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:38.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:38 smithi064 ceph-mon[42243]: pgmap v964: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:38.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:38 smithi064 ceph-mon[42243]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T12:12:38.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:38 smithi064 ceph-mon[42243]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T12:12:38.797 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:38.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:38.798 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:38.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:38 smithi193 ceph-mon[33593]: pgmap v964: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:38 smithi193 ceph-mon[33593]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T12:12:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:38 smithi193 ceph-mon[33593]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T12:12:38.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:38.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:38.931 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:38.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:39.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:39.075 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:39.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:39.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:39.214 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:39.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:39.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:39.349 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:39.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:39.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:39.483 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:39.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:39.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:39.615 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:39.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:39.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:39.759 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:39.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:39.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:39 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:12:39.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:39 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:12:39.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:39 smithi193 ceph-mon[33593]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T12:12:39.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:39.897 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:39.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:39.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:39 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:12:39.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:39 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:12:39.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:39 smithi064 ceph-mon[30223]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T12:12:39.911 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:39 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:12:39.911 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:39 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:12:39.911 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:39 smithi064 ceph-mon[42243]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T12:12:39.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:12:40.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:40.015 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:40.047 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:40.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:40.149 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:40.197 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:40.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:40.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:40.341 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:40.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:40.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:40.493 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:40.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:40.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:40.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:40.630 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:40.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:40.674 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:12:40.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:40.767 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:40.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:40 smithi064 ceph-mon[30223]: pgmap v967: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:12:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:40 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1949068144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:40.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:40 smithi064 ceph-mon[42243]: pgmap v967: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:12:40.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:40 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1949068144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:40 smithi193 ceph-mon[33593]: pgmap v967: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-24T12:12:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:40 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1949068144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:12:40.869 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:40.905 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:40.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:41.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:41.046 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:41.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:41.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:41.187 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:41.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:41.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:41.329 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:41.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:41.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:41.464 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:41.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:41.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:41.620 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:41.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:41.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:41.762 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:41.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:41.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:41 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:41.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:41 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1949068144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:12:41.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:41 smithi064 ceph-mon[30223]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T12:12:41.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:41 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:41.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:41 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1949068144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:12:41.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:41 smithi064 ceph-mon[42243]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T12:12:41.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:41 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:41.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:41 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1949068144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:12:41.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:41 smithi193 ceph-mon[33593]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T12:12:41.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:41.911 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:41.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:42.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:42.051 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:42.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:42.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:42.185 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:42.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:42.286 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:42.324 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:42.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:42.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:42.453 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:42.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:42.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:42.592 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:42.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:42.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:42.729 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:42.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:42.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:42.872 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:42.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:42.974 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:43.022 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:43.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:43.090 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:43 smithi193 ceph-mon[33593]: pgmap v969: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:12:43.090 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:43.090 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:43.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:43.160 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:43.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:43.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:43.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:43 smithi064 ceph-mon[30223]: pgmap v969: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:12:43.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:43.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:43.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:43 smithi064 ceph-mon[42243]: pgmap v969: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:12:43.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:12:43.268 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:43.296 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:43.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:43.397 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:43.435 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:43.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:43.536 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:12:43.537 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:12:43.537 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-03-24T12:12:43.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:12:43.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:43.590 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:43.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:43.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:43.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:43.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:43.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:43.884 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:43.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:43.902 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:12:43.902 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:12:43.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:44.024 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:44.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:44.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:44.157 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:44.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:44.211 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:12:44.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:44.305 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:44.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:44.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:44.451 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:44.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:44.553 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:44.578 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:44.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:44.654 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:12:44.679 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:44.742 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:44.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:44.843 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:44.886 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:44.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:44.987 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:45.045 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:45.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:45.074 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:45.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:45.185 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:45.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:45.286 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:45.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:45 smithi064 ceph-mon[30223]: pgmap v970: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:12:45.289 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:45 smithi064 ceph-mon[42243]: pgmap v970: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:12:45.322 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:45.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:45 smithi193 ceph-mon[33593]: pgmap v970: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:12:45.424 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:45.425 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:12:45.425 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:12:45.425 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:12:45.425 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:12:45.426 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:12:45.455 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:12:45.457 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:45.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:45.503 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:45.503 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:12:45.543 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:12:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:12:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:12:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:12:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:12:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:12:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:12:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:12:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:12:45.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:45.589 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:45.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:45.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:45.733 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:45.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:45.835 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:45.874 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:45.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:45.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:46.011 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:46.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:46.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:46.155 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:46.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:46.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:46.299 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:46.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:46.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:46 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3079363497' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:46 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3079363497' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:46 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3079363497' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:46.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:46.437 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:46.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:46.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:46.577 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:46.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:46.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:46.709 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:46.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:46.731 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:46.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:46.847 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:46.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:46.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:46.984 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:46.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:47.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:47.092 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:155008 10.c 10:3511e9fc:::benchmark_data_smithi064_174908_object10591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:18.646437+0000", 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 268.44318128999998, 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.113593917, 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:12:47.093 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 155008 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646437+0000", 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646437+0000", 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646438+0000", 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.265e-06 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646442+0000", 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0040000000000001e-06 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.095 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646450+0000", 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5520000000000005e-06 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646457+0000", 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7469999999999999e-06 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646500+0000", 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2688000000000002e-05 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646593+0000", 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2856999999999995e-05 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.646715+0000", 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012233400000000001 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.649180+0000", 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024649260000000001 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.097 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.653296+0000", 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041161469999999997 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.759920+0000", 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.10662337700000001 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.759969+0000", 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9818999999999999e-05 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.098 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.760031+0000", 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1200999999999999e-05 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:155024 10.c 10:3eb07ef8:::benchmark_data_smithi064_174908_object10607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:08:18.656780+0000", 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 268.43283842199997, 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.103383609, 2024-03-24T12:12:47.099 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 155024 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.656780+0000", 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.656780+0000", 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.656782+0000", 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0959999999999999e-06 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.656792+0000", 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0637e-05 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.656805+0000", 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2937e-05 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.656813+0000", 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7079999999999997e-06 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.656859+0000", 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6108e-05 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.656944+0000", 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4955000000000002e-05 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.657013+0000", 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9492000000000005e-05 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.658923+0000", 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0019091469999999999 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.103 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.659074+0000", 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015111700000000001 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.760073+0000", 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.100998949 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.760129+0000", 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6069999999999997e-05 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:08:18.760163+0000", 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4393000000000001e-05 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:224487 10.c 10:31f562e1:::benchmark_data_smithi064_174908_object80070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:20.462155+0000", 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 206.62746271699999, 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.208835779, 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 224487 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462155+0000", 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462155+0000", 2024-03-24T12:12:47.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462157+0000", 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.883e-06 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462163+0000", 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4160000000000003e-06 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462173+0000", 2024-03-24T12:12:47.107 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0682999999999999e-05 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462181+0000", 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5170000000000002e-06 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462240+0000", 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8900000000000002e-05 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462351+0000", 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011094000000000001 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.462455+0000", 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000104753 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.464973+0000", 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025179030000000002 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.467652+0000", 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026789650000000002 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.670918+0000", 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.20326625400000001 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.670958+0000", 2024-03-24T12:12:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9263e-05 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.670991+0000", 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3302000000000001e-05 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:224497 10.c 10:32b383d1:::benchmark_data_smithi064_174908_object80080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:20.471935+0000", 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 206.61768315800001, 2024-03-24T12:12:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.19922583199999999, 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 224497 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.471935+0000", 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.471935+0000", 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.471938+0000", 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8789999999999999e-06 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.471945+0000", 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9099999999999999e-06 2024-03-24T12:12:47.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.471958+0000", 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3439e-05 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.471968+0000", 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7019999999999996e-06 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.472028+0000", 2024-03-24T12:12:47.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0748000000000002e-05 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.472153+0000", 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012491100000000001 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.472275+0000", 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012191199999999999 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.670796+0000", 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.19852110000000001 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.671040+0000", 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000243646 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.671088+0000", 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8374999999999997e-05 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.671130+0000", 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2018000000000003e-05 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.671161+0000", 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0192000000000002e-05 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.117 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.15714.0:224501 10.8 e367/320 10:14fd65e4:::benchmark_data_smithi064_174908_object80084:head v 367'13927, mlcod=367'13925)", 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:20.475161+0000", 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 206.61445709700001, 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.20074499500000001, 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.475161+0000", 2024-03-24T12:12:47.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.475161+0000", 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.475162+0000", 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4729999999999999e-06 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.475170+0000", 2024-03-24T12:12:47.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6920000000000002e-06 2024-03-24T12:12:47.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.670797+0000", 2024-03-24T12:12:47.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.19562682000000001 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.670818+0000", 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1259000000000001e-05 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.670880+0000", 2024-03-24T12:12:47.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1638999999999997e-05 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.670905+0000", 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4958999999999999e-05 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.675718+0000", 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048132959999999999 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:20.675906+0000", 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018785699999999999 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:231574 10.c 10:3a38cd64:::benchmark_data_smithi064_174908_object87157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:27.480175+0000", 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 199.60944299100001, 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.11909528799999999, 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:12:47.123 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 231574 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480175+0000", 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480175+0000", 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480177+0000", 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3740000000000001e-06 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480184+0000", 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.246e-06 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480197+0000", 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3791999999999999e-05 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480216+0000", 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8357000000000001e-05 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480375+0000", 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015978100000000001 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480650+0000", 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027464500000000001 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.480992+0000", 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00034162599999999998 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.484124+0000", 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031321309999999998 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.484964+0000", 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00084054799999999999 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.599190+0000", 2024-03-24T12:12:47.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.114225479 2024-03-24T12:12:47.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.599223+0000", 2024-03-24T12:12:47.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3538999999999999e-05 2024-03-24T12:12:47.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.599270+0000", 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6770000000000001e-05 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.15714.0:231578 10.c 10:3f8d147b:::benchmark_data_smithi064_174908_object87161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:09:27.488927+0000", 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 199.600690947, 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.112099054, 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.130 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.15714", 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 231578 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.488927+0000", 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.488927+0000", 2024-03-24T12:12:47.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.488930+0000", 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7609999999999999e-06 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.488942+0000", 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2425e-05 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.488958+0000", 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6013000000000001e-05 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.488968+0000", 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8709999999999992e-06 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.489080+0000", 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000112057 2024-03-24T12:12:47.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.489191+0000", 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011106400000000001 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.489262+0000", 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1025000000000005e-05 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.491489+0000", 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022264979999999999 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.492088+0000", 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00059979399999999996 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.600855+0000", 2024-03-24T12:12:47.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.108766221 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.600939+0000", 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4417000000000002e-05 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:09:27.601026+0000", 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6908000000000002e-05 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:69832 11.f 11:fb0cac3f:::benchmark_data_smithi064_206665_object69831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:11:27.923324+0000", 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 79.166293680999999, 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.172865079, 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 69832 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923324+0000", 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923324+0000", 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923326+0000", 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.401e-06 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923385+0000", 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9698999999999999e-05 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923397+0000", 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1802e-05 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923405+0000", 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7260000000000002e-06 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923471+0000", 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6179000000000005e-05 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923642+0000", 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000170753 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923823+0000", 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018125199999999999 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.929352+0000", 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055286190000000002 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.933459+0000", 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041070220000000001 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.096072+0000", 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16261356699999999 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.096144+0000", 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1228000000000003e-05 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.096189+0000", 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5831000000000002e-05 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:69835 11.f 11:f786d37a:::benchmark_data_smithi064_206665_object69834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:11:27.923620+0000", 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 79.165997856999994, 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17272685199999999, 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 69835 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923620+0000", 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923620+0000", 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923621+0000", 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2330000000000001e-06 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923762+0000", 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014091599999999999 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923771+0000", 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8230000000000001e-06 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.923778+0000", 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2370000000000003e-06 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.926455+0000", 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026767940000000001 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.926673+0000", 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021822500000000001 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.926831+0000", 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015771399999999999 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.929527+0000", 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026957420000000001 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.933502+0000", 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0039747530000000001 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.096279+0000", 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16277734299999999 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.096316+0000", 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6984999999999999e-05 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.096347+0000", 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1087000000000002e-05 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:69842 11.f 11:f47758de:::benchmark_data_smithi064_206665_object69841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:11:27.930627+0000", 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 79.158990523, 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17291589700000001, 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 69842 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.930627+0000", 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.930627+0000", 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.930636+0000", 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8119999999999997e-06 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.930676+0000", 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9320000000000003e-05 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.930690+0000", 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.486e-05 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.930698+0000", 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4749999999999996e-06 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.930752+0000", 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3811999999999999e-05 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.930911+0000", 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015969199999999999 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.931024+0000", 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000112658 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.933574+0000", 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025503230000000002 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934282+0000", 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00070812800000000003 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.103439+0000", 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.169156051 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.103492+0000", 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3946000000000002e-05 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.103543+0000", 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0819999999999998e-05 2024-03-24T12:12:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:69849 11.f 11:f8aa1af7:::benchmark_data_smithi064_206665_object69848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:11:27.934431+0000", 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 79.155187158000004, 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16928485099999999, 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 69849 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934431+0000", 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934431+0000", 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934432+0000", 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2300000000000001e-06 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934503+0000", 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0794e-05 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934511+0000", 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9559999999999997e-06 2024-03-24T12:12:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934518+0000", 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3830000000000003e-06 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934647+0000", 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000129014 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934781+0000", 2024-03-24T12:12:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013416900000000001 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.934944+0000", 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016224999999999999 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.937595+0000", 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026517229999999999 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.941802+0000", 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042071720000000003 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.103627+0000", 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16182444700000001 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.103675+0000", 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7964999999999999e-05 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.103716+0000", 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0747999999999997e-05 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:69854 11.f 11:fda7ec72:::benchmark_data_smithi064_206665_object69853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:11:27.938848+0000", 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 79.150770108000003, 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16968976099999999, 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 69854 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.938848+0000", 2024-03-24T12:12:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.938848+0000", 2024-03-24T12:12:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.938849+0000", 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.669e-06 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.938910+0000", 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0846e-05 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.938921+0000", 2024-03-24T12:12:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0885e-05 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.938929+0000", 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7239999999999992e-06 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.938980+0000", 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1177000000000001e-05 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.939114+0000", 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013390799999999999 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.939266+0000", 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000151676 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.941347+0000", 2024-03-24T12:12:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020815149999999999 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.941861+0000", 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00051352899999999998 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.108425+0000", 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.16656396600000001 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.108487+0000", 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2776999999999999e-05 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.108538+0000", 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0089000000000001e-05 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:69860 11.f 11:ffe23f6a:::benchmark_data_smithi064_206665_object69859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:12:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:11:27.947138+0000", 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 79.142479668999997, 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.161525262, 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 69860 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947138+0000", 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947138+0000", 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947140+0000", 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.246e-06 2024-03-24T12:12:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947197+0000", 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7961000000000003e-05 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947207+0000", 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7999999999999993e-06 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947215+0000", 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5689999999999999e-06 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947253+0000", 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7700999999999997e-05 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947341+0000", 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8906000000000006e-05 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.947740+0000", 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039863700000000003 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.950065+0000", 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002325227 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:27.950363+0000", 2024-03-24T12:12:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000297826 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.108578+0000", 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.15821506799999999 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.108627+0000", 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8591000000000003e-05 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:11:28.108664+0000", 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6730000000000002e-05 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:133590 11.f 11:f783be51:::benchmark_data_smithi064_206665_object1457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:21.530809+0000", 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 25.558808976999998, 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.145455217, 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 133590 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.530809+0000", 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.530809+0000", 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.530811+0000", 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2060000000000001e-06 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.530817+0000", 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5369999999999997e-06 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.178 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:47 smithi193 ceph-mon[33593]: pgmap v971: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 870 op/s 2024-03-24T12:12:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.530828+0000", 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.163e-05 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.530836+0000", 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9959999999999993e-06 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.530890+0000", 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3875000000000002e-05 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.530998+0000", 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107459 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.531100+0000", 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010269599999999999 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.533657+0000", 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002556848 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.544323+0000", 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010665399000000001 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676183+0000", 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.13186025300000001 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676227+0000", 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4119000000000002e-05 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676264+0000", 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7199000000000001e-05 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:133633 11.4 11:23cc277b:::benchmark_data_smithi064_206665_object1500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-24T12:12:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:21.566892+0000", 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 25.522726300999999, 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.10919891900000001, 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 133633 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.566892+0000", 2024-03-24T12:12:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.566892+0000", 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.566893+0000", 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8649999999999999e-06 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.566899+0000", 2024-03-24T12:12:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.152e-06 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.566909+0000", 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0312000000000001e-05 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.566916+0000", 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4460000000000004e-06 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.566958+0000", 2024-03-24T12:12:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1486999999999998e-05 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.567036+0000", 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8208999999999996e-05 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.567103+0000", 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7101999999999996e-05 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.571482+0000", 2024-03-24T12:12:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0043786210000000001 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573634+0000", 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021517369999999999 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.675999+0000", 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.102365789 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676045+0000", 2024-03-24T12:12:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5368000000000003e-05 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676091+0000", 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5831000000000002e-05 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:133637 11.4 11:291779e6:::benchmark_data_smithi064_206665_object1504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:21.568367+0000", 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 25.521250578, 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.107879748, 2024-03-24T12:12:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 133637 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.568367+0000", 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.568367+0000", 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.568369+0000", 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.376e-06 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.568376+0000", 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.695e-06 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.568387+0000", 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0421000000000001e-05 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.568394+0000", 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4220000000000003e-06 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.569055+0000", 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000660379 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.569132+0000", 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7471000000000004e-05 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.569239+0000", 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107253 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573592+0000", 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0043526190000000003 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573681+0000", 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9220000000000003e-05 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676155+0000", 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.10247342800000001 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676205+0000", 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0309000000000002e-05 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676247+0000", 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2154999999999999e-05 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:133639 11.4 11:228a21d4:::benchmark_data_smithi064_206665_object1506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:21.570843+0000", 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 25.518775151, 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.105517924, 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 133639 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.570843+0000", 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.570843+0000", 2024-03-24T12:12:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.570844+0000", 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5230000000000001e-06 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.570849+0000", 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6260000000000003e-06 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.570858+0000", 2024-03-24T12:12:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4399999999999994e-06 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.570865+0000", 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3690000000000003e-06 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.571506+0000", 2024-03-24T12:12:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00064124099999999997 2024-03-24T12:12:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.571612+0000", 2024-03-24T12:12:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000106191 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.571712+0000", 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9392e-05 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.675964+0000", 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.104252874 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676111+0000", 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014613399999999999 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676289+0000", 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000178555 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676332+0000", 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3259e-05 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676361+0000", 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.832e-05 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:133642 11.4 11:2e4c092f:::benchmark_data_smithi064_206665_object1509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:21.573011+0000", 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 25.516606749000001, 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.109193842, 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 133642 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573011+0000", 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573011+0000", 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573012+0000", 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2100000000000001e-06 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573016+0000", 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8850000000000001e-06 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573025+0000", 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6330000000000002e-06 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573030+0000", 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3419999999999998e-06 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573705+0000", 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067435800000000001 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573812+0000", 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107662 2024-03-24T12:12:47.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.573896+0000", 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4017999999999994e-05 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676403+0000", 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.102506604 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.679350+0000", 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00294744 2024-03-24T12:12:47.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.682146+0000", 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0027958739999999998 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.682177+0000", 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0898999999999997e-05 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.682205+0000", 2024-03-24T12:12:47.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7917000000000001e-05 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:133645 11.4 11:2b86e3cb:::benchmark_data_smithi064_206665_object1512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:21.575904+0000", 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 25.513714290999999, 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.108323476, 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:47.207 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 133645 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.575904+0000", 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.575904+0000", 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.575905+0000", 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5009999999999999e-06 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.575910+0000", 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8930000000000004e-06 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.672850+0000", 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.096939629999999999 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.672869+0000", 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9709999999999999e-05 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676430+0000", 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003560311 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676500+0000", 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0722999999999994e-05 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676570+0000", 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9233000000000006e-05 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.682120+0000", 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055503250000000001 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.683548+0000", 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014276619999999999 2024-03-24T12:12:47.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.684121+0000", 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00057360199999999999 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.684184+0000", 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2534999999999997e-05 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.684227+0000", 2024-03-24T12:12:47.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3350999999999999e-05 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop_reply(client.16023.0:133633 11.4 e450/417 ondisk, result = 0)", 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:21.576998+0000", 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 25.512619581999999, 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.099099682999999994, 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.576998+0000", 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.576998+0000", 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.576999+0000", 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1149999999999999e-06 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.577006+0000", 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2419999999999997e-06 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.674661+0000", 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.097655218000000002 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.674671+0000", 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0219e-05 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.675989+0000", 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013176920000000001 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.675994+0000", 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4650000000000002e-06 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:21.676098+0000", 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000103732 2024-03-24T12:12:47.216 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.217 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.217 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.217 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:47.217 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:47.217 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:12:47.218 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:47.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:47.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:47 smithi064 ceph-mon[42243]: pgmap v971: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 870 op/s 2024-03-24T12:12:47.272 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:47 smithi064 ceph-mon[30223]: pgmap v971: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 870 op/s 2024-03-24T12:12:47.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:47.351 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:47.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:47.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:47.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:47.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:47.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:12:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:12:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:47.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:47.637 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:47.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:47.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:47.774 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:47.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:47.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:47.913 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:47.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:48.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:48.052 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:48.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:48.154 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:48.186 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:48.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:48.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:48.330 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:48.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:48.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:12:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:12:48.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:48.481 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:48.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:48.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:48.626 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:48.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:48.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:48.768 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:48.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:48.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:48.908 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:48.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:49.010 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:49.042 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:49.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:49.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:49 smithi064 ceph-mon[30223]: pgmap v972: 158 pgs: 158 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T12:12:49.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:49.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:49 smithi064 ceph-mon[42243]: pgmap v972: 158 pgs: 158 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T12:12:49.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:49.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:49.187 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:49.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:49.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:49.322 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:49.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:49 smithi193 ceph-mon[33593]: pgmap v972: 158 pgs: 158 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T12:12:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:49.383 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:12:49.424 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:49.451 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:49.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:49.553 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:49.588 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:49.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:49.690 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:49.742 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:49.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:49.798 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:49.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:49.881 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:49.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:49.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:50.023 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:50.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:50.125 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:50.157 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:50.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:50.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:12:50.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:50.300 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:50.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:50.402 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:50.431 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:50.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:50.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:50.558 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:50.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:50.659 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:50.691 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:50.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:50.793 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:50.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:50.817 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:50.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:50.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:12:50.955 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:12:50.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:50.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:51.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:51.098 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:51.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:51.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:51.236 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:51.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:51 smithi064 ceph-mon[30223]: pgmap v973: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:51 smithi064 ceph-mon[30223]: from='client.16155 v1:172.21.15.64:0/4249911560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:12:51.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:51 smithi064 ceph-mon[42243]: pgmap v973: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:51.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:51 smithi064 ceph-mon[42243]: from='client.16155 v1:172.21.15.64:0/4249911560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:12:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:51 smithi193 ceph-mon[33593]: pgmap v973: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T12:12:51.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:51 smithi193 ceph-mon[33593]: from='client.16155 v1:172.21.15.64:0/4249911560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:12:51.338 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:51.370 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:51.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:51.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:51.504 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:51.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:51.606 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:51.647 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:51.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:51.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:51.750 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop_reply(client.16023.0:169495 11.7 e454/417 ondisk, result = 0)", 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:51.749304+0000", 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.000438322, 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00046626099999999999, 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749304+0000", 2024-03-24T12:12:51.751 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749304+0000", 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749305+0000", 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2999999999999999e-07 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:51.752 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749308+0000", 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7929999999999999e-06 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749315+0000", 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2500000000000003e-06 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749320+0000", 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5369999999999997e-06 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.753 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749379+0000", 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8523000000000002e-05 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749393+0000", 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3944999999999999e-05 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.16023.0:169502 11.a e454/417 11:5b11067c:::benchmark_data_smithi064_206665_object37369:head v 454'10569, mlcod=454'10567)", 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:51.749412+0000", 2024-03-24T12:12:51.754 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.00032973200000000001, 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00043300599999999999, 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749412+0000", 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749412+0000", 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.755 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749413+0000", 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6799999999999999e-07 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749419+0000", 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7540000000000003e-06 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749426+0000", 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9070000000000001e-06 2024-03-24T12:12:51.756 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749431+0000", 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2179999999999998e-06 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749487+0000", 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5599000000000001e-05 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749505+0000", 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7992e-05 2024-03-24T12:12:51.757 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:169481 11.7 11:efdef5c1:::benchmark_data_smithi064_206665_object37348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:51.732572+0000", 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.017170325, 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017337216999999999, 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:51.758 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 169481 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.732572+0000", 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.732572+0000", 2024-03-24T12:12:51.759 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.732574+0000", 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1890000000000001e-06 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.732579+0000", 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.13e-06 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.732594+0000", 2024-03-24T12:12:51.760 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5175e-05 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.732603+0000", 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3729999999999993e-06 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.734094+0000", 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014908720000000001 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.734546+0000", 2024-03-24T12:12:51.761 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000452452 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.735404+0000", 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00085827099999999997 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.744774+0000", 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0093698670000000005 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:51.762 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.745139+0000", 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00036510299999999998 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:169489 11.7 11:ebf95506:::benchmark_data_smithi064_206665_object37356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:51.738437+0000", 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.011305109000000001, 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011558465, 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:51.763 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 169489 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.738437+0000", 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.738437+0000", 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.764 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.738440+0000", 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7030000000000002e-06 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.738446+0000", 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7939999999999999e-06 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.738458+0000", 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2225000000000001e-05 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.765 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.738467+0000", 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9649999999999997e-06 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.740761+0000", 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00229452 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.741068+0000", 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00030647699999999999 2024-03-24T12:12:51.766 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.741465+0000", 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039703699999999999 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.745152+0000", 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036869479999999998 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749289+0000", 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041373979999999996 2024-03-24T12:12:51.767 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.16023.0:169495 11.7 11:e1196fe1:::benchmark_data_smithi064_206665_object37362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-03-24T12:12:51.742630+0000", 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 0.0071123970000000003, 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0074509320000000004, 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.16023", 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:12:51.768 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 169495 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.742630+0000", 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.742630+0000", 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-03-24T12:12:51.769 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.742632+0000", 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.17e-06 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.742638+0000", 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9569999999999999e-06 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.742651+0000", 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.344e-05 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.770 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.742660+0000", 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7790000000000001e-06 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.745181+0000", 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025204910000000001 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.745317+0000", 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013651599999999999 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.745446+0000", 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012846600000000001 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.748119+0000", 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026731329999999998 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-03-24T12:12:51.749398+0000", 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012797520000000001 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T12:12:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-24T12:12:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-03-24T12:12:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 5 2024-03-24T12:12:51.773 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:12:51.813 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:51.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:51.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:51.955 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:51.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:52.002 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":973,"stamp":"2024-03-24T12:12:49.565497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6577002870,"num_objects":100547,"num_object_clones":0,"num_object_copies":301641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100547,"num_whiteouts":0,"num_read":1997,"num_read_kb":1749,"num_write":164921,"num_write_kb":8456366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":329608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80573,"ondisk_log_size":80573,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25146440,"kb_used_data":19302152,"kb_used_omap":18,"kb_used_meta":5844077,"kb_avail":724814776,"statfs":{"total":767960285184,"available":742210330624,"internally_reserved":0,"allocated":19765403648,"data_stored":19762007362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":5984335067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-793509888,"num_objects":-12108,"num_object_clones":0,"num_object_copies":-36324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12108,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12108,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003575"},"pg_stats":[{"pgid":"11.7","version":"454'10429","reported_seq":10513,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:49.330180+0000","last_change":"2024-03-24T12:10:42.552347+0000","last_active":"2024-03-24T12:12:49.330180+0000","last_peered":"2024-03-24T12:12:49.330180+0000","last_clean":"2024-03-24T12:12:49.330180+0000","last_became_active":"2024-03-24T12:10:42.552023+0000","last_became_peered":"2024-03-24T12:10:42.552023+0000","last_unstale":"2024-03-24T12:12:49.330180+0000","last_undegraded":"2024-03-24T12:12:49.330180+0000","last_fullsized":"2024-03-24T12:12:49.330180+0000","mapping_epoch":417,"log_start":"439'5500","ondisk_log_start":"439'5500","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:52.660113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400752640,"num_objects":6115,"num_object_clones":0,"num_object_copies":18345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10429,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":1129,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.678771+0000","last_change":"2024-03-24T12:12:23.293867+0000","last_active":"2024-03-24T12:12:40.678771+0000","last_peered":"2024-03-24T12:12:40.678771+0000","last_clean":"2024-03-24T12:12:40.678771+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:12:40.678771+0000","last_undegraded":"2024-03-24T12:12:40.678771+0000","last_fullsized":"2024-03-24T12:12:40.678771+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:12:23.293737+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.293737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:48.062463+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"450'39","reported_seq":795,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676821+0000","last_change":"2024-03-24T12:12:17.532582+0000","last_active":"2024-03-24T12:12:40.676821+0000","last_peered":"2024-03-24T12:12:40.676821+0000","last_clean":"2024-03-24T12:12:40.676821+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:12:40.676821+0000","last_undegraded":"2024-03-24T12:12:40.676821+0000","last_fullsized":"2024-03-24T12:12:40.676821+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:12:17.532295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:17.532295+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:24.750874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":338,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.672476+0000","last_change":"2024-03-24T12:12:21.258445+0000","last_active":"2024-03-24T12:12:40.672476+0000","last_peered":"2024-03-24T12:12:40.672476+0000","last_clean":"2024-03-24T12:12:40.672476+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:12:40.672476+0000","last_undegraded":"2024-03-24T12:12:40.672476+0000","last_fullsized":"2024-03-24T12:12:40.672476+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:21.258323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:21.258323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:32.254341+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":761,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677406+0000","last_change":"2024-03-24T12:10:19.713100+0000","last_active":"2024-03-24T12:12:40.677406+0000","last_peered":"2024-03-24T12:12:40.677406+0000","last_clean":"2024-03-24T12:12:40.677406+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:12:40.677406+0000","last_undegraded":"2024-03-24T12:12:40.677406+0000","last_fullsized":"2024-03-24T12:12:40.677406+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.713016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.713016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564018+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"454'10366","reported_seq":10539,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.757801+0000","last_change":"2024-03-24T12:12:16.501705+0000","last_active":"2024-03-24T12:12:46.757801+0000","last_peered":"2024-03-24T12:12:46.757801+0000","last_clean":"2024-03-24T12:12:46.757801+0000","last_became_active":"2024-03-24T12:10:42.558408+0000","last_became_peered":"2024-03-24T12:10:42.558408+0000","last_unstale":"2024-03-24T12:12:46.757801+0000","last_undegraded":"2024-03-24T12:12:46.757801+0000","last_fullsized":"2024-03-24T12:12:46.757801+0000","mapping_epoch":417,"log_start":"441'6000","ondisk_log_start":"441'6000","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'8080","last_scrub_stamp":"2024-03-24T12:12:16.501400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:16.501400+0000","objects_scrubbed":8006,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:35.863020+0000","scrub_duration":2228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413663232,"num_objects":6312,"num_object_clones":0,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10366,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"450'21","reported_seq":886,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.683287+0000","last_change":"2024-03-24T12:11:18.368554+0000","last_active":"2024-03-24T12:12:40.683287+0000","last_peered":"2024-03-24T12:12:40.683287+0000","last_clean":"2024-03-24T12:12:40.683287+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:12:40.683287+0000","last_undegraded":"2024-03-24T12:12:40.683287+0000","last_fullsized":"2024-03-24T12:12:40.683287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:11:18.368416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.368416+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":889,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.681396+0000","last_change":"2024-03-24T12:11:33.850001+0000","last_active":"2024-03-24T12:12:40.681396+0000","last_peered":"2024-03-24T12:12:40.681396+0000","last_clean":"2024-03-24T12:12:40.681396+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:12:40.681396+0000","last_undegraded":"2024-03-24T12:12:40.681396+0000","last_fullsized":"2024-03-24T12:12:40.681396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:33.849353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:11:33.849353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:51.989463+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.680522+0000","last_change":"2024-03-24T12:11:37.592094+0000","last_active":"2024-03-24T12:12:40.680522+0000","last_peered":"2024-03-24T12:12:40.680522+0000","last_clean":"2024-03-24T12:12:40.680522+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:12:40.680522+0000","last_undegraded":"2024-03-24T12:12:40.680522+0000","last_fullsized":"2024-03-24T12:12:40.680522+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:37.591971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.591971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:53.358009+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":842,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.681857+0000","last_change":"2024-03-24T12:10:26.001526+0000","last_active":"2024-03-24T12:12:40.681857+0000","last_peered":"2024-03-24T12:12:40.681857+0000","last_clean":"2024-03-24T12:12:40.681857+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:12:40.681857+0000","last_undegraded":"2024-03-24T12:12:40.681857+0000","last_fullsized":"2024-03-24T12:12:40.681857+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:26.001400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:26.001400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"454'10215","reported_seq":10386,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.803586+0000","last_change":"2024-03-24T12:12:14.190930+0000","last_active":"2024-03-24T12:12:46.803586+0000","last_peered":"2024-03-24T12:12:46.803586+0000","last_clean":"2024-03-24T12:12:46.803586+0000","last_became_active":"2024-03-24T12:10:42.558497+0000","last_became_peered":"2024-03-24T12:10:42.558497+0000","last_unstale":"2024-03-24T12:12:46.803586+0000","last_undegraded":"2024-03-24T12:12:46.803586+0000","last_fullsized":"2024-03-24T12:12:46.803586+0000","mapping_epoch":417,"log_start":"440'5800","ondisk_log_start":"440'5800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7845","last_scrub_stamp":"2024-03-24T12:12:14.190779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.190779+0000","objects_scrubbed":7762,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:34.139978+0000","scrub_duration":2001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410845184,"num_objects":6269,"num_object_clones":0,"num_object_copies":18807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10215,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"450'60","reported_seq":947,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674261+0000","last_change":"2024-03-24T12:11:20.907053+0000","last_active":"2024-03-24T12:12:40.674261+0000","last_peered":"2024-03-24T12:12:40.674261+0000","last_clean":"2024-03-24T12:12:40.674261+0000","last_became_active":"2024-03-24T11:52:17.423797+0000","last_became_peered":"2024-03-24T11:52:17.423797+0000","last_unstale":"2024-03-24T12:12:40.674261+0000","last_undegraded":"2024-03-24T12:12:40.674261+0000","last_fullsized":"2024-03-24T12:12:40.674261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:11:20.906884+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.906884+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":231,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677567+0000","last_change":"2024-03-24T12:11:27.439531+0000","last_active":"2024-03-24T12:12:40.677567+0000","last_peered":"2024-03-24T12:12:40.677567+0000","last_clean":"2024-03-24T12:12:40.677567+0000","last_became_active":"2024-03-24T12:08:36.164337+0000","last_became_peered":"2024-03-24T12:08:36.164337+0000","last_unstale":"2024-03-24T12:12:40.677567+0000","last_undegraded":"2024-03-24T12:12:40.677567+0000","last_fullsized":"2024-03-24T12:12:40.677567+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:27.439409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.439409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564260+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":345,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682727+0000","last_change":"2024-03-24T12:11:22.405559+0000","last_active":"2024-03-24T12:12:40.682727+0000","last_peered":"2024-03-24T12:12:40.682727+0000","last_clean":"2024-03-24T12:12:40.682727+0000","last_became_active":"2024-03-24T12:08:36.155105+0000","last_became_peered":"2024-03-24T12:08:36.155105+0000","last_unstale":"2024-03-24T12:12:40.682727+0000","last_undegraded":"2024-03-24T12:12:40.682727+0000","last_fullsized":"2024-03-24T12:12:40.682727+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:22.404955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:22.404955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":867,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677246+0000","last_change":"2024-03-24T12:11:32.175174+0000","last_active":"2024-03-24T12:12:40.677246+0000","last_peered":"2024-03-24T12:12:40.677246+0000","last_clean":"2024-03-24T12:12:40.677246+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:12:40.677246+0000","last_undegraded":"2024-03-24T12:12:40.677246+0000","last_fullsized":"2024-03-24T12:12:40.677246+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.175048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.175048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:00.349651+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"454'10238","reported_seq":10322,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.616278+0000","last_change":"2024-03-24T12:10:42.555659+0000","last_active":"2024-03-24T12:12:46.616278+0000","last_peered":"2024-03-24T12:12:46.616278+0000","last_clean":"2024-03-24T12:12:46.616278+0000","last_became_active":"2024-03-24T12:10:42.555322+0000","last_became_peered":"2024-03-24T12:10:42.555322+0000","last_unstale":"2024-03-24T12:12:46.616278+0000","last_undegraded":"2024-03-24T12:12:46.616278+0000","last_fullsized":"2024-03-24T12:12:46.616278+0000","mapping_epoch":417,"log_start":"440'5800","ondisk_log_start":"440'5800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.203293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410648576,"num_objects":6266,"num_object_clones":0,"num_object_copies":18798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10238,"num_write_kb":528128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"450'53","reported_seq":953,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682937+0000","last_change":"2024-03-24T12:12:20.673896+0000","last_active":"2024-03-24T12:12:40.682937+0000","last_peered":"2024-03-24T12:12:40.682937+0000","last_clean":"2024-03-24T12:12:40.682937+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:12:40.682937+0000","last_undegraded":"2024-03-24T12:12:40.682937+0000","last_fullsized":"2024-03-24T12:12:40.682937+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:12:20.673798+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:12:20.673798+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.976568+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":636,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.680887+0000","last_change":"2024-03-24T12:09:04.230431+0000","last_active":"2024-03-24T12:12:40.680887+0000","last_peered":"2024-03-24T12:12:40.680887+0000","last_clean":"2024-03-24T12:12:40.680887+0000","last_became_active":"2024-03-24T12:09:04.230111+0000","last_became_peered":"2024-03-24T12:09:04.230111+0000","last_unstale":"2024-03-24T12:12:40.680887+0000","last_undegraded":"2024-03-24T12:12:40.680887+0000","last_fullsized":"2024-03-24T12:12:40.680887+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:08:42.842178+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:08:42.842178+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":599,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682543+0000","last_change":"2024-03-24T12:11:43.684235+0000","last_active":"2024-03-24T12:12:40.682543+0000","last_peered":"2024-03-24T12:12:40.682543+0000","last_clean":"2024-03-24T12:12:40.682543+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:12:40.682543+0000","last_undegraded":"2024-03-24T12:12:40.682543+0000","last_fullsized":"2024-03-24T12:12:40.682543+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.684070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.684070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:05.544597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":329,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674401+0000","last_change":"2024-03-24T12:08:34.445597+0000","last_active":"2024-03-24T12:12:40.674401+0000","last_peered":"2024-03-24T12:12:40.674401+0000","last_clean":"2024-03-24T12:12:40.674401+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:12:40.674401+0000","last_undegraded":"2024-03-24T12:12:40.674401+0000","last_fullsized":"2024-03-24T12:12:40.674401+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:34.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.445477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":792,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676924+0000","last_change":"2024-03-24T12:10:09.762381+0000","last_active":"2024-03-24T12:12:40.676924+0000","last_peered":"2024-03-24T12:12:40.676924+0000","last_clean":"2024-03-24T12:12:40.676924+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:12:40.676924+0000","last_undegraded":"2024-03-24T12:12:40.676924+0000","last_fullsized":"2024-03-24T12:12:40.676924+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:09.762247+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.762247+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:43.519564+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"454'10447","reported_seq":10532,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.853712+0000","last_change":"2024-03-24T12:10:42.557591+0000","last_active":"2024-03-24T12:12:46.853712+0000","last_peered":"2024-03-24T12:12:46.853712+0000","last_clean":"2024-03-24T12:12:46.853712+0000","last_became_active":"2024-03-24T12:10:42.557222+0000","last_became_peered":"2024-03-24T12:10:42.557222+0000","last_unstale":"2024-03-24T12:12:46.853712+0000","last_undegraded":"2024-03-24T12:12:46.853712+0000","last_fullsized":"2024-03-24T12:12:46.853712+0000","mapping_epoch":417,"log_start":"439'5700","ondisk_log_start":"439'5700","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419495936,"num_objects":6401,"num_object_clones":0,"num_object_copies":19203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10447,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"450'35","reported_seq":901,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682462+0000","last_change":"2024-03-24T12:12:28.791017+0000","last_active":"2024-03-24T12:12:40.682462+0000","last_peered":"2024-03-24T12:12:40.682462+0000","last_clean":"2024-03-24T12:12:40.682462+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:12:40.682462+0000","last_undegraded":"2024-03-24T12:12:40.682462+0000","last_fullsized":"2024-03-24T12:12:40.682462+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:12:28.790890+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:12:28.790890+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:30.631954+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":600,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674369+0000","last_change":"2024-03-24T12:10:31.694887+0000","last_active":"2024-03-24T12:12:40.674369+0000","last_peered":"2024-03-24T12:12:40.674369+0000","last_clean":"2024-03-24T12:12:40.674369+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:12:40.674369+0000","last_undegraded":"2024-03-24T12:12:40.674369+0000","last_fullsized":"2024-03-24T12:12:40.674369+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:31.694756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:10:31.694756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":1130,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.670720+0000","last_change":"2024-03-24T12:11:45.062251+0000","last_active":"2024-03-24T12:12:40.670720+0000","last_peered":"2024-03-24T12:12:40.670720+0000","last_clean":"2024-03-24T12:12:40.670720+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:12:40.670720+0000","last_undegraded":"2024-03-24T12:12:40.670720+0000","last_fullsized":"2024-03-24T12:12:40.670720+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:11:45.062114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.062114+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.519407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":855,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.684326+0000","last_change":"2024-03-24T12:10:09.602488+0000","last_active":"2024-03-24T12:12:40.684326+0000","last_peered":"2024-03-24T12:12:40.684326+0000","last_clean":"2024-03-24T12:12:40.684326+0000","last_became_active":"2024-03-24T12:08:36.161088+0000","last_became_peered":"2024-03-24T12:08:36.161088+0000","last_unstale":"2024-03-24T12:12:40.684326+0000","last_undegraded":"2024-03-24T12:12:40.684326+0000","last_fullsized":"2024-03-24T12:12:40.684326+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:10:09.602371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.602371+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765564+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":1188,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.675375+0000","last_change":"2024-03-24T12:11:42.368078+0000","last_active":"2024-03-24T12:12:40.675375+0000","last_peered":"2024-03-24T12:12:40.675375+0000","last_clean":"2024-03-24T12:12:40.675375+0000","last_became_active":"2024-03-24T12:08:36.161668+0000","last_became_peered":"2024-03-24T12:08:36.161668+0000","last_unstale":"2024-03-24T12:12:40.675375+0000","last_undegraded":"2024-03-24T12:12:40.675375+0000","last_fullsized":"2024-03-24T12:12:40.675375+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:42.367917+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:11:42.367917+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:57.952365+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"454'10173","reported_seq":10258,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.590749+0000","last_change":"2024-03-24T12:10:42.559881+0000","last_active":"2024-03-24T12:12:46.590749+0000","last_peered":"2024-03-24T12:12:46.590749+0000","last_clean":"2024-03-24T12:12:46.590749+0000","last_became_active":"2024-03-24T12:10:42.559566+0000","last_became_peered":"2024-03-24T12:10:42.559566+0000","last_unstale":"2024-03-24T12:12:46.590749+0000","last_undegraded":"2024-03-24T12:12:46.590749+0000","last_fullsized":"2024-03-24T12:12:46.590749+0000","mapping_epoch":417,"log_start":"440'5800","ondisk_log_start":"440'5800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.203383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412286976,"num_objects":6291,"num_object_clones":0,"num_object_copies":18873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10173,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"450'46","reported_seq":945,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.681982+0000","last_change":"2024-03-24T12:12:13.128854+0000","last_active":"2024-03-24T12:12:40.681982+0000","last_peered":"2024-03-24T12:12:40.681982+0000","last_clean":"2024-03-24T12:12:40.681982+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:12:40.681982+0000","last_undegraded":"2024-03-24T12:12:40.681982+0000","last_fullsized":"2024-03-24T12:12:40.681982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:12:13.128715+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:12:13.128715+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:25.266832+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":881,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.675086+0000","last_change":"2024-03-24T12:11:43.926041+0000","last_active":"2024-03-24T12:12:40.675086+0000","last_peered":"2024-03-24T12:12:40.675086+0000","last_clean":"2024-03-24T12:12:40.675086+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:12:40.675086+0000","last_undegraded":"2024-03-24T12:12:40.675086+0000","last_fullsized":"2024-03-24T12:12:40.675086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.925905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.925905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:02.893804+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":1108,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.684232+0000","last_change":"2024-03-24T12:10:14.509080+0000","last_active":"2024-03-24T12:12:40.684232+0000","last_peered":"2024-03-24T12:12:40.684232+0000","last_clean":"2024-03-24T12:12:40.684232+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:12:40.684232+0000","last_undegraded":"2024-03-24T12:12:40.684232+0000","last_fullsized":"2024-03-24T12:12:40.684232+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:14.508959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.508959+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":769,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677497+0000","last_change":"2024-03-24T12:11:13.744756+0000","last_active":"2024-03-24T12:12:40.677497+0000","last_peered":"2024-03-24T12:12:40.677497+0000","last_clean":"2024-03-24T12:12:40.677497+0000","last_became_active":"2024-03-24T12:08:36.166897+0000","last_became_peered":"2024-03-24T12:08:36.166897+0000","last_unstale":"2024-03-24T12:12:40.677497+0000","last_undegraded":"2024-03-24T12:12:40.677497+0000","last_fullsized":"2024-03-24T12:12:40.677497+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:13.744672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.744672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564316+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":792,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677312+0000","last_change":"2024-03-24T12:10:11.797368+0000","last_active":"2024-03-24T12:12:40.677312+0000","last_peered":"2024-03-24T12:12:40.677312+0000","last_clean":"2024-03-24T12:12:40.677312+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:12:40.677312+0000","last_undegraded":"2024-03-24T12:12:40.677312+0000","last_fullsized":"2024-03-24T12:12:40.677312+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:11.797287+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:11.797287+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:43.519598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"454'10241","reported_seq":10326,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.855844+0000","last_change":"2024-03-24T12:10:42.557448+0000","last_active":"2024-03-24T12:12:46.855844+0000","last_peered":"2024-03-24T12:12:46.855844+0000","last_clean":"2024-03-24T12:12:46.855844+0000","last_became_active":"2024-03-24T12:10:42.557136+0000","last_became_peered":"2024-03-24T12:10:42.557136+0000","last_unstale":"2024-03-24T12:12:46.855844+0000","last_undegraded":"2024-03-24T12:12:46.855844+0000","last_fullsized":"2024-03-24T12:12:46.855844+0000","mapping_epoch":417,"log_start":"439'5500","ondisk_log_start":"439'5500","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410058752,"num_objects":6257,"num_object_clones":0,"num_object_copies":18771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10241,"num_write_kb":527936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"450'32","reported_seq":891,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.684058+0000","last_change":"2024-03-24T12:11:21.537427+0000","last_active":"2024-03-24T12:12:40.684058+0000","last_peered":"2024-03-24T12:12:40.684058+0000","last_clean":"2024-03-24T12:12:40.684058+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:12:40.684058+0000","last_undegraded":"2024-03-24T12:12:40.684058+0000","last_fullsized":"2024-03-24T12:12:40.684058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:11:21.537273+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:11:21.537273+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":873,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677488+0000","last_change":"2024-03-24T12:11:18.149890+0000","last_active":"2024-03-24T12:12:40.677488+0000","last_peered":"2024-03-24T12:12:40.677488+0000","last_clean":"2024-03-24T12:12:40.677488+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:12:40.677488+0000","last_undegraded":"2024-03-24T12:12:40.677488+0000","last_fullsized":"2024-03-24T12:12:40.677488+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:18.149775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.149775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.402236+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":765,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677832+0000","last_change":"2024-03-24T12:12:29.601620+0000","last_active":"2024-03-24T12:12:40.677832+0000","last_peered":"2024-03-24T12:12:40.677832+0000","last_clean":"2024-03-24T12:12:40.677832+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:12:40.677832+0000","last_undegraded":"2024-03-24T12:12:40.677832+0000","last_fullsized":"2024-03-24T12:12:40.677832+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:29.601502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:29.601502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.074844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":847,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671339+0000","last_change":"2024-03-24T12:10:14.749596+0000","last_active":"2024-03-24T12:12:40.671339+0000","last_peered":"2024-03-24T12:12:40.671339+0000","last_clean":"2024-03-24T12:12:40.671339+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:12:40.671339+0000","last_undegraded":"2024-03-24T12:12:40.671339+0000","last_fullsized":"2024-03-24T12:12:40.671339+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:14.749468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.749468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"454'10077","reported_seq":10161,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.697459+0000","last_change":"2024-03-24T12:10:42.552380+0000","last_active":"2024-03-24T12:12:46.697459+0000","last_peered":"2024-03-24T12:12:46.697459+0000","last_clean":"2024-03-24T12:12:46.697459+0000","last_became_active":"2024-03-24T12:10:42.552054+0000","last_became_peered":"2024-03-24T12:10:42.552054+0000","last_unstale":"2024-03-24T12:12:46.697459+0000","last_undegraded":"2024-03-24T12:12:46.697459+0000","last_fullsized":"2024-03-24T12:12:46.697459+0000","mapping_epoch":417,"log_start":"439'5100","ondisk_log_start":"439'5100","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409534464,"num_objects":6249,"num_object_clones":0,"num_object_copies":18747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10077,"num_write_kb":522432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":1098,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.670142+0000","last_change":"2024-03-24T12:11:38.190269+0000","last_active":"2024-03-24T12:12:40.670142+0000","last_peered":"2024-03-24T12:12:40.670142+0000","last_clean":"2024-03-24T12:12:40.670142+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:12:40.670142+0000","last_undegraded":"2024-03-24T12:12:40.670142+0000","last_fullsized":"2024-03-24T12:12:40.670142+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:11:38.190121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:38.190121+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.913586+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"450'49","reported_seq":922,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.679748+0000","last_change":"2024-03-24T12:10:19.407534+0000","last_active":"2024-03-24T12:12:40.679748+0000","last_peered":"2024-03-24T12:12:40.679748+0000","last_clean":"2024-03-24T12:12:40.679748+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:12:40.679748+0000","last_undegraded":"2024-03-24T12:12:40.679748+0000","last_fullsized":"2024-03-24T12:12:40.679748+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:10:19.407433+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.407433+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":863,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674833+0000","last_change":"2024-03-24T12:11:16.896924+0000","last_active":"2024-03-24T12:12:40.674833+0000","last_peered":"2024-03-24T12:12:40.674833+0000","last_clean":"2024-03-24T12:12:40.674833+0000","last_became_active":"2024-03-24T11:52:15.138214+0000","last_became_peered":"2024-03-24T11:52:15.138214+0000","last_unstale":"2024-03-24T12:12:40.674833+0000","last_undegraded":"2024-03-24T12:12:40.674833+0000","last_fullsized":"2024-03-24T12:12:40.674833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:16.896801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:11:16.896801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":875,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676639+0000","last_change":"2024-03-24T12:12:24.532173+0000","last_active":"2024-03-24T12:12:40.676639+0000","last_peered":"2024-03-24T12:12:40.676639+0000","last_clean":"2024-03-24T12:12:40.676639+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:12:40.676639+0000","last_undegraded":"2024-03-24T12:12:40.676639+0000","last_fullsized":"2024-03-24T12:12:40.676639+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:24.531916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:12:24.531916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:30.506866+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":935,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671554+0000","last_change":"2024-03-24T12:12:28.260533+0000","last_active":"2024-03-24T12:12:40.671554+0000","last_peered":"2024-03-24T12:12:40.671554+0000","last_clean":"2024-03-24T12:12:40.671554+0000","last_became_active":"2024-03-24T12:10:47.971906+0000","last_became_peered":"2024-03-24T12:10:47.971906+0000","last_unstale":"2024-03-24T12:12:40.671554+0000","last_undegraded":"2024-03-24T12:12:40.671554+0000","last_fullsized":"2024-03-24T12:12:40.671554+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:12:28.260278+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:12:28.260278+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:30.568309+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"454'10376","reported_seq":10547,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.804232+0000","last_change":"2024-03-24T12:12:12.434955+0000","last_active":"2024-03-24T12:12:46.804232+0000","last_peered":"2024-03-24T12:12:46.804232+0000","last_clean":"2024-03-24T12:12:46.804232+0000","last_became_active":"2024-03-24T12:10:42.562110+0000","last_became_peered":"2024-03-24T12:10:42.562110+0000","last_unstale":"2024-03-24T12:12:46.804232+0000","last_undegraded":"2024-03-24T12:12:46.804232+0000","last_fullsized":"2024-03-24T12:12:46.804232+0000","mapping_epoch":417,"log_start":"440'6000","ondisk_log_start":"440'6000","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7838","last_scrub_stamp":"2024-03-24T12:12:12.434659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:12.434659+0000","objects_scrubbed":7761,"log_size":4376,"log_dups_size":0,"ondisk_log_size":4376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:20.803078+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424017920,"num_objects":6470,"num_object_clones":0,"num_object_copies":19410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":839,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.679938+0000","last_change":"2024-03-24T12:11:32.519689+0000","last_active":"2024-03-24T12:12:40.679938+0000","last_peered":"2024-03-24T12:12:40.679938+0000","last_clean":"2024-03-24T12:12:40.679938+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:12:40.679938+0000","last_undegraded":"2024-03-24T12:12:40.679938+0000","last_fullsized":"2024-03-24T12:12:40.679938+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.519559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.519559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:57.052977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"450'50","reported_seq":923,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.670033+0000","last_change":"2024-03-24T12:10:29.733172+0000","last_active":"2024-03-24T12:12:40.670033+0000","last_peered":"2024-03-24T12:12:40.670033+0000","last_clean":"2024-03-24T12:12:40.670033+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:12:40.670033+0000","last_undegraded":"2024-03-24T12:12:40.670033+0000","last_fullsized":"2024-03-24T12:12:40.670033+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_deep_scrub":"241'38","last_deep_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.733073+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":616,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674613+0000","last_change":"2024-03-24T12:11:27.010877+0000","last_active":"2024-03-24T12:12:40.674613+0000","last_peered":"2024-03-24T12:12:40.674613+0000","last_clean":"2024-03-24T12:12:40.674613+0000","last_became_active":"2024-03-24T11:59:19.097627+0000","last_became_peered":"2024-03-24T11:59:19.097627+0000","last_unstale":"2024-03-24T12:12:40.674613+0000","last_undegraded":"2024-03-24T12:12:40.674613+0000","last_fullsized":"2024-03-24T12:12:40.674613+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:11:27.010737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.010737+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.904171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":857,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.680609+0000","last_change":"2024-03-24T12:10:38.401132+0000","last_active":"2024-03-24T12:12:40.680609+0000","last_peered":"2024-03-24T12:12:40.680609+0000","last_clean":"2024-03-24T12:12:40.680609+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:12:40.680609+0000","last_undegraded":"2024-03-24T12:12:40.680609+0000","last_fullsized":"2024-03-24T12:12:40.680609+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:38.401028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:10:38.401028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":933,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.679289+0000","last_change":"2024-03-24T12:10:45.574511+0000","last_active":"2024-03-24T12:12:40.679289+0000","last_peered":"2024-03-24T12:12:40.679289+0000","last_clean":"2024-03-24T12:12:40.679289+0000","last_became_active":"2024-03-24T12:10:45.573960+0000","last_became_peered":"2024-03-24T12:10:45.573960+0000","last_unstale":"2024-03-24T12:12:40.679289+0000","last_undegraded":"2024-03-24T12:12:40.679289+0000","last_fullsized":"2024-03-24T12:12:40.679289+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:25.359883+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.359883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"454'10071","reported_seq":10241,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:47.305799+0000","last_change":"2024-03-24T12:12:19.183366+0000","last_active":"2024-03-24T12:12:47.305799+0000","last_peered":"2024-03-24T12:12:47.305799+0000","last_clean":"2024-03-24T12:12:47.305799+0000","last_became_active":"2024-03-24T12:10:42.552346+0000","last_became_peered":"2024-03-24T12:10:42.552346+0000","last_unstale":"2024-03-24T12:12:47.305799+0000","last_undegraded":"2024-03-24T12:12:47.305799+0000","last_fullsized":"2024-03-24T12:12:47.305799+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7988","last_scrub_stamp":"2024-03-24T12:12:19.183036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.183036+0000","objects_scrubbed":7889,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:25.262334+0000","scrub_duration":3159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398131200,"num_objects":6075,"num_object_clones":0,"num_object_copies":18225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10071,"num_write_kb":516672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":864,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674355+0000","last_change":"2024-03-24T12:12:30.833566+0000","last_active":"2024-03-24T12:12:40.674355+0000","last_peered":"2024-03-24T12:12:40.674355+0000","last_clean":"2024-03-24T12:12:40.674355+0000","last_became_active":"2024-03-24T12:08:36.161658+0000","last_became_peered":"2024-03-24T12:08:36.161658+0000","last_unstale":"2024-03-24T12:12:40.674355+0000","last_undegraded":"2024-03-24T12:12:40.674355+0000","last_fullsized":"2024-03-24T12:12:40.674355+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:30.833447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:12:30.833447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:59.276491+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"450'47","reported_seq":927,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682894+0000","last_change":"2024-03-24T12:11:37.405546+0000","last_active":"2024-03-24T12:12:40.682894+0000","last_peered":"2024-03-24T12:12:40.682894+0000","last_clean":"2024-03-24T12:12:40.682894+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:12:40.682894+0000","last_undegraded":"2024-03-24T12:12:40.682894+0000","last_fullsized":"2024-03-24T12:12:40.682894+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'37","last_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_deep_scrub":"369'37","last_deep_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.405400+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:04.594252+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":881,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.679425+0000","last_change":"2024-03-24T12:11:17.183566+0000","last_active":"2024-03-24T12:12:40.679425+0000","last_peered":"2024-03-24T12:12:40.679425+0000","last_clean":"2024-03-24T12:12:40.679425+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:12:40.679425+0000","last_undegraded":"2024-03-24T12:12:40.679425+0000","last_fullsized":"2024-03-24T12:12:40.679425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:17.183126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:11:17.183126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.643290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":614,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.673470+0000","last_change":"2024-03-24T12:12:20.922086+0000","last_active":"2024-03-24T12:12:40.673470+0000","last_peered":"2024-03-24T12:12:40.673470+0000","last_clean":"2024-03-24T12:12:40.673470+0000","last_became_active":"2024-03-24T12:05:44.738926+0000","last_became_peered":"2024-03-24T12:05:44.738926+0000","last_unstale":"2024-03-24T12:12:40.673470+0000","last_undegraded":"2024-03-24T12:12:40.673470+0000","last_fullsized":"2024-03-24T12:12:40.673470+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:20.921985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:20.921985+0000","last_clean_scrub_stamp":"2024-03-24T12:12:20.921985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:22.281652+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":946,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674734+0000","last_change":"2024-03-24T12:11:45.081667+0000","last_active":"2024-03-24T12:12:40.674734+0000","last_peered":"2024-03-24T12:12:40.674734+0000","last_clean":"2024-03-24T12:12:40.674734+0000","last_became_active":"2024-03-24T12:10:41.554328+0000","last_became_peered":"2024-03-24T12:10:41.554328+0000","last_unstale":"2024-03-24T12:12:40.674734+0000","last_undegraded":"2024-03-24T12:12:40.674734+0000","last_fullsized":"2024-03-24T12:12:40.674734+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.081517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.745034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"454'10137","reported_seq":10221,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.591106+0000","last_change":"2024-03-24T12:10:42.558775+0000","last_active":"2024-03-24T12:12:46.591106+0000","last_peered":"2024-03-24T12:12:46.591106+0000","last_clean":"2024-03-24T12:12:46.591106+0000","last_became_active":"2024-03-24T12:10:42.558462+0000","last_became_peered":"2024-03-24T12:10:42.558462+0000","last_unstale":"2024-03-24T12:12:46.591106+0000","last_undegraded":"2024-03-24T12:12:46.591106+0000","last_fullsized":"2024-03-24T12:12:46.591106+0000","mapping_epoch":417,"log_start":"440'5700","ondisk_log_start":"440'5700","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.203363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413728768,"num_objects":6313,"num_object_clones":0,"num_object_copies":18939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":526400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":731,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677932+0000","last_change":"2024-03-24T12:12:21.577856+0000","last_active":"2024-03-24T12:12:40.677932+0000","last_peered":"2024-03-24T12:12:40.677932+0000","last_clean":"2024-03-24T12:12:40.677932+0000","last_became_active":"2024-03-24T12:08:36.165265+0000","last_became_peered":"2024-03-24T12:08:36.165265+0000","last_unstale":"2024-03-24T12:12:40.677932+0000","last_undegraded":"2024-03-24T12:12:40.677932+0000","last_fullsized":"2024-03-24T12:12:40.677932+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:21.577764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:21.577764+0000","last_clean_scrub_stamp":"2024-03-24T12:12:21.577764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:45.211618+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"450'42","reported_seq":932,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.685162+0000","last_change":"2024-03-24T12:11:23.383853+0000","last_active":"2024-03-24T12:12:40.685162+0000","last_peered":"2024-03-24T12:12:40.685162+0000","last_clean":"2024-03-24T12:12:40.685162+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:12:40.685162+0000","last_undegraded":"2024-03-24T12:12:40.685162+0000","last_fullsized":"2024-03-24T12:12:40.685162+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:11:23.383616+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.383616+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":692,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.683036+0000","last_change":"2024-03-24T12:10:16.036972+0000","last_active":"2024-03-24T12:12:40.683036+0000","last_peered":"2024-03-24T12:12:40.683036+0000","last_clean":"2024-03-24T12:12:40.683036+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:12:40.683036+0000","last_undegraded":"2024-03-24T12:12:40.683036+0000","last_fullsized":"2024-03-24T12:12:40.683036+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:16.036850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:16.036850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":874,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.681685+0000","last_change":"2024-03-24T12:12:26.485548+0000","last_active":"2024-03-24T12:12:40.681685+0000","last_peered":"2024-03-24T12:12:40.681685+0000","last_clean":"2024-03-24T12:12:40.681685+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:12:40.681685+0000","last_undegraded":"2024-03-24T12:12:40.681685+0000","last_fullsized":"2024-03-24T12:12:40.681685+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:26.485429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:12:26.485429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:38.629715+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":870,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682609+0000","last_change":"2024-03-24T12:11:39.903255+0000","last_active":"2024-03-24T12:12:40.682609+0000","last_peered":"2024-03-24T12:12:40.682609+0000","last_clean":"2024-03-24T12:12:40.682609+0000","last_became_active":"2024-03-24T12:08:36.173357+0000","last_became_peered":"2024-03-24T12:08:36.173357+0000","last_unstale":"2024-03-24T12:12:40.682609+0000","last_undegraded":"2024-03-24T12:12:40.682609+0000","last_fullsized":"2024-03-24T12:12:40.682609+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_clean_scrub_stamp":"2024-03-24T12:11:39.903133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:04.706316+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":957,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.673687+0000","last_change":"2024-03-24T12:12:23.891768+0000","last_active":"2024-03-24T12:12:40.673687+0000","last_peered":"2024-03-24T12:12:40.673687+0000","last_clean":"2024-03-24T12:12:40.673687+0000","last_became_active":"2024-03-24T12:10:50.998877+0000","last_became_peered":"2024-03-24T12:10:50.998877+0000","last_unstale":"2024-03-24T12:12:40.673687+0000","last_undegraded":"2024-03-24T12:12:40.673687+0000","last_fullsized":"2024-03-24T12:12:40.673687+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:12:23.891623+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.891623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:32.530680+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"454'10187","reported_seq":10271,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.578039+0000","last_change":"2024-03-24T12:10:42.558754+0000","last_active":"2024-03-24T12:12:46.578039+0000","last_peered":"2024-03-24T12:12:46.578039+0000","last_clean":"2024-03-24T12:12:46.578039+0000","last_became_active":"2024-03-24T12:10:42.558225+0000","last_became_peered":"2024-03-24T12:10:42.558225+0000","last_unstale":"2024-03-24T12:12:46.578039+0000","last_undegraded":"2024-03-24T12:12:46.578039+0000","last_fullsized":"2024-03-24T12:12:46.578039+0000","mapping_epoch":417,"log_start":"440'5800","ondisk_log_start":"440'5800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406126592,"num_objects":6197,"num_object_clones":0,"num_object_copies":18591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10187,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":866,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676579+0000","last_change":"2024-03-24T12:10:37.011600+0000","last_active":"2024-03-24T12:12:40.676579+0000","last_peered":"2024-03-24T12:12:40.676579+0000","last_clean":"2024-03-24T12:12:40.676579+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:12:40.676579+0000","last_undegraded":"2024-03-24T12:12:40.676579+0000","last_fullsized":"2024-03-24T12:12:40.676579+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:37.011493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:10:37.011493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.402181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"450'40","reported_seq":1180,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674453+0000","last_change":"2024-03-24T12:11:32.027906+0000","last_active":"2024-03-24T12:12:40.674453+0000","last_peered":"2024-03-24T12:12:40.674453+0000","last_clean":"2024-03-24T12:12:40.674453+0000","last_became_active":"2024-03-24T11:52:17.425445+0000","last_became_peered":"2024-03-24T11:52:17.425445+0000","last_unstale":"2024-03-24T12:12:40.674453+0000","last_undegraded":"2024-03-24T12:12:40.674453+0000","last_fullsized":"2024-03-24T12:12:40.674453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:11:32.027804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.027804+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":1187,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677393+0000","last_change":"2024-03-24T12:11:35.238528+0000","last_active":"2024-03-24T12:12:40.677393+0000","last_peered":"2024-03-24T12:12:40.677393+0000","last_clean":"2024-03-24T12:12:40.677393+0000","last_became_active":"2024-03-24T12:10:33.461798+0000","last_became_peered":"2024-03-24T12:10:33.461798+0000","last_unstale":"2024-03-24T12:12:40.677393+0000","last_undegraded":"2024-03-24T12:12:40.677393+0000","last_fullsized":"2024-03-24T12:12:40.677393+0000","mapping_epoch":64,"log_start":"83'130","ondisk_log_start":"83'130","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:35.238422+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:10:25.984109+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.238422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:44.414733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":772,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677572+0000","last_change":"2024-03-24T12:12:30.636570+0000","last_active":"2024-03-24T12:12:40.677572+0000","last_peered":"2024-03-24T12:12:40.677572+0000","last_clean":"2024-03-24T12:12:40.677572+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:12:40.677572+0000","last_undegraded":"2024-03-24T12:12:40.677572+0000","last_fullsized":"2024-03-24T12:12:40.677572+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:30.636466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:30.636466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:52.420266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":860,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674269+0000","last_change":"2024-03-24T12:08:43.113674+0000","last_active":"2024-03-24T12:12:40.674269+0000","last_peered":"2024-03-24T12:12:40.674269+0000","last_clean":"2024-03-24T12:12:40.674269+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:12:40.674269+0000","last_undegraded":"2024-03-24T12:12:40.674269+0000","last_fullsized":"2024-03-24T12:12:40.674269+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:08:43.113510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.113510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":726,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677932+0000","last_change":"2024-03-24T12:11:35.488009+0000","last_active":"2024-03-24T12:12:40.677932+0000","last_peered":"2024-03-24T12:12:40.677932+0000","last_clean":"2024-03-24T12:12:40.677932+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:12:40.677932+0000","last_undegraded":"2024-03-24T12:12:40.677932+0000","last_fullsized":"2024-03-24T12:12:40.677932+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:35.487872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.487872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:03.379847+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"454'10202","reported_seq":10286,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.597746+0000","last_change":"2024-03-24T12:10:42.558943+0000","last_active":"2024-03-24T12:12:46.597746+0000","last_peered":"2024-03-24T12:12:46.597746+0000","last_clean":"2024-03-24T12:12:46.597746+0000","last_became_active":"2024-03-24T12:10:42.558601+0000","last_became_peered":"2024-03-24T12:10:42.558601+0000","last_unstale":"2024-03-24T12:12:46.597746+0000","last_undegraded":"2024-03-24T12:12:46.597746+0000","last_fullsized":"2024-03-24T12:12:46.597746+0000","mapping_epoch":417,"log_start":"440'5800","ondisk_log_start":"440'5800","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.203345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415498240,"num_objects":6340,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10202,"num_write_kb":529344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":253,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.678392+0000","last_change":"2024-03-24T12:12:18.565246+0000","last_active":"2024-03-24T12:12:40.678392+0000","last_peered":"2024-03-24T12:12:40.678392+0000","last_clean":"2024-03-24T12:12:40.678392+0000","last_became_active":"2024-03-24T12:08:36.162188+0000","last_became_peered":"2024-03-24T12:08:36.162188+0000","last_unstale":"2024-03-24T12:12:40.678392+0000","last_undegraded":"2024-03-24T12:12:40.678392+0000","last_fullsized":"2024-03-24T12:12:40.678392+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:18.565128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:12:18.565128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.801563+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"450'21","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.681901+0000","last_change":"2024-03-24T12:11:27.012324+0000","last_active":"2024-03-24T12:12:40.681901+0000","last_peered":"2024-03-24T12:12:40.681901+0000","last_clean":"2024-03-24T12:12:40.681901+0000","last_became_active":"2024-03-24T12:08:36.172879+0000","last_became_peered":"2024-03-24T12:08:36.172879+0000","last_unstale":"2024-03-24T12:12:40.681901+0000","last_undegraded":"2024-03-24T12:12:40.681901+0000","last_fullsized":"2024-03-24T12:12:40.681901+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'17","last_scrub_stamp":"2024-03-24T12:11:27.012204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.012204+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.287575+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":330,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.670583+0000","last_change":"2024-03-24T12:10:12.685963+0000","last_active":"2024-03-24T12:12:40.670583+0000","last_peered":"2024-03-24T12:12:40.670583+0000","last_clean":"2024-03-24T12:12:40.670583+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:12:40.670583+0000","last_undegraded":"2024-03-24T12:12:40.670583+0000","last_fullsized":"2024-03-24T12:12:40.670583+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.685815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.685815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"75'5","reported_seq":804,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:45.530085+0000","last_change":"2024-03-24T12:11:13.936448+0000","last_active":"2024-03-24T12:12:45.530085+0000","last_peered":"2024-03-24T12:12:45.530085+0000","last_clean":"2024-03-24T12:12:45.530085+0000","last_became_active":"2024-03-24T12:05:44.736437+0000","last_became_peered":"2024-03-24T12:05:44.736437+0000","last_unstale":"2024-03-24T12:12:45.530085+0000","last_undegraded":"2024-03-24T12:12:45.530085+0000","last_fullsized":"2024-03-24T12:12:45.530085+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:11:13.936256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.936256+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":428,"num_read_kb":423,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":861,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.679325+0000","last_change":"2024-03-24T12:11:31.240744+0000","last_active":"2024-03-24T12:12:40.679325+0000","last_peered":"2024-03-24T12:12:40.679325+0000","last_clean":"2024-03-24T12:12:40.679325+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:12:40.679325+0000","last_undegraded":"2024-03-24T12:12:40.679325+0000","last_fullsized":"2024-03-24T12:12:40.679325+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:31.240650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:31.240650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:46.619564+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":1186,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676024+0000","last_change":"2024-03-24T12:11:30.174297+0000","last_active":"2024-03-24T12:12:40.676024+0000","last_peered":"2024-03-24T12:12:40.676024+0000","last_clean":"2024-03-24T12:12:40.676024+0000","last_became_active":"2024-03-24T12:10:23.398177+0000","last_became_peered":"2024-03-24T12:10:23.398177+0000","last_unstale":"2024-03-24T12:12:40.676024+0000","last_undegraded":"2024-03-24T12:12:40.676024+0000","last_fullsized":"2024-03-24T12:12:40.676024+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:30.173569+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:11:30.173569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:51.650721+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"454'10349","reported_seq":10521,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.613259+0000","last_change":"2024-03-24T12:12:14.049841+0000","last_active":"2024-03-24T12:12:46.613259+0000","last_peered":"2024-03-24T12:12:46.613259+0000","last_clean":"2024-03-24T12:12:46.613259+0000","last_became_active":"2024-03-24T12:10:42.559548+0000","last_became_peered":"2024-03-24T12:10:42.559548+0000","last_unstale":"2024-03-24T12:12:46.613259+0000","last_undegraded":"2024-03-24T12:12:46.613259+0000","last_fullsized":"2024-03-24T12:12:46.613259+0000","mapping_epoch":417,"log_start":"441'6000","ondisk_log_start":"441'6000","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7951","last_scrub_stamp":"2024-03-24T12:12:14.049510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.049510+0000","objects_scrubbed":7849,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:39.018031+0000","scrub_duration":2948,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417529856,"num_objects":6371,"num_object_clones":0,"num_object_copies":19113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10349,"num_write_kb":535040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":848,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.683548+0000","last_change":"2024-03-24T12:11:38.437189+0000","last_active":"2024-03-24T12:12:40.683548+0000","last_peered":"2024-03-24T12:12:40.683548+0000","last_clean":"2024-03-24T12:12:40.683548+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:12:40.683548+0000","last_undegraded":"2024-03-24T12:12:40.683548+0000","last_fullsized":"2024-03-24T12:12:40.683548+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:38.437087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:11:38.437087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:46.086372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"450'56","reported_seq":1017,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:44.338041+0000","last_change":"2024-03-24T12:10:33.726470+0000","last_active":"2024-03-24T12:12:44.338041+0000","last_peered":"2024-03-24T12:12:44.338041+0000","last_clean":"2024-03-24T12:12:44.338041+0000","last_became_active":"2024-03-24T12:08:36.152866+0000","last_became_peered":"2024-03-24T12:08:36.152866+0000","last_unstale":"2024-03-24T12:12:44.338041+0000","last_undegraded":"2024-03-24T12:12:44.338041+0000","last_fullsized":"2024-03-24T12:12:44.338041+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'44","last_scrub_stamp":"2024-03-24T12:10:33.726357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:33.726357+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":736,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676929+0000","last_change":"2024-03-24T12:11:19.470678+0000","last_active":"2024-03-24T12:12:40.676929+0000","last_peered":"2024-03-24T12:12:40.676929+0000","last_clean":"2024-03-24T12:12:40.676929+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:12:40.676929+0000","last_undegraded":"2024-03-24T12:12:40.676929+0000","last_fullsized":"2024-03-24T12:12:40.676929+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:19.470485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.470485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.585453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":838,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.672967+0000","last_change":"2024-03-24T12:11:28.156543+0000","last_active":"2024-03-24T12:12:40.672967+0000","last_peered":"2024-03-24T12:12:40.672967+0000","last_clean":"2024-03-24T12:12:40.672967+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:12:40.672967+0000","last_undegraded":"2024-03-24T12:12:40.672967+0000","last_fullsized":"2024-03-24T12:12:40.672967+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:28.156411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.156411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":812,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676733+0000","last_change":"2024-03-24T12:10:26.959642+0000","last_active":"2024-03-24T12:12:40.676733+0000","last_peered":"2024-03-24T12:12:40.676733+0000","last_clean":"2024-03-24T12:12:40.676733+0000","last_became_active":"2024-03-24T12:10:26.959434+0000","last_became_peered":"2024-03-24T12:10:26.959434+0000","last_unstale":"2024-03-24T12:12:40.676733+0000","last_undegraded":"2024-03-24T12:12:40.676733+0000","last_fullsized":"2024-03-24T12:12:40.676733+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:10.760367+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.760367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.585366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"454'10164","reported_seq":10249,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:46.727836+0000","last_change":"2024-03-24T12:10:42.558548+0000","last_active":"2024-03-24T12:12:46.727836+0000","last_peered":"2024-03-24T12:12:46.727836+0000","last_clean":"2024-03-24T12:12:46.727836+0000","last_became_active":"2024-03-24T12:10:42.558230+0000","last_became_peered":"2024-03-24T12:10:42.558230+0000","last_unstale":"2024-03-24T12:12:46.727836+0000","last_undegraded":"2024-03-24T12:12:46.727836+0000","last_fullsized":"2024-03-24T12:12:46.727836+0000","mapping_epoch":417,"log_start":"439'5200","ondisk_log_start":"439'5200","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411369496,"num_objects":6278,"num_object_clones":0,"num_object_copies":18834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10164,"num_write_kb":526081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":1114,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.683453+0000","last_change":"2024-03-24T12:11:19.363311+0000","last_active":"2024-03-24T12:12:40.683453+0000","last_peered":"2024-03-24T12:12:40.683453+0000","last_clean":"2024-03-24T12:12:40.683453+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:12:40.683453+0000","last_undegraded":"2024-03-24T12:12:40.683453+0000","last_fullsized":"2024-03-24T12:12:40.683453+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:11:19.362922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.362922+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"450'35","reported_seq":928,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676290+0000","last_change":"2024-03-24T12:11:24.029766+0000","last_active":"2024-03-24T12:12:40.676290+0000","last_peered":"2024-03-24T12:12:40.676290+0000","last_clean":"2024-03-24T12:12:40.676290+0000","last_became_active":"2024-03-24T12:08:36.174663+0000","last_became_peered":"2024-03-24T12:08:36.174663+0000","last_unstale":"2024-03-24T12:12:40.676290+0000","last_undegraded":"2024-03-24T12:12:40.676290+0000","last_fullsized":"2024-03-24T12:12:40.676290+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:11:24.029563+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:11:24.029563+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.448106+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":607,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682669+0000","last_change":"2024-03-24T12:12:17.361772+0000","last_active":"2024-03-24T12:12:40.682669+0000","last_peered":"2024-03-24T12:12:40.682669+0000","last_clean":"2024-03-24T12:12:40.682669+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:12:40.682669+0000","last_undegraded":"2024-03-24T12:12:40.682669+0000","last_fullsized":"2024-03-24T12:12:40.682669+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:17.361648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:12:17.361648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:23.972127+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":852,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682658+0000","last_change":"2024-03-24T12:11:26.507311+0000","last_active":"2024-03-24T12:12:40.682658+0000","last_peered":"2024-03-24T12:12:40.682658+0000","last_clean":"2024-03-24T12:12:40.682658+0000","last_became_active":"2024-03-24T12:08:36.145936+0000","last_became_peered":"2024-03-24T12:08:36.145936+0000","last_unstale":"2024-03-24T12:12:40.682658+0000","last_undegraded":"2024-03-24T12:12:40.682658+0000","last_fullsized":"2024-03-24T12:12:40.682658+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:26.507184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:11:26.507184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.897133+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":938,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674626+0000","last_change":"2024-03-24T12:11:18.911425+0000","last_active":"2024-03-24T12:12:40.674626+0000","last_peered":"2024-03-24T12:12:40.674626+0000","last_clean":"2024-03-24T12:12:40.674626+0000","last_became_active":"2024-03-24T12:10:36.965307+0000","last_became_peered":"2024-03-24T12:10:36.965307+0000","last_unstale":"2024-03-24T12:12:40.674626+0000","last_undegraded":"2024-03-24T12:12:40.674626+0000","last_fullsized":"2024-03-24T12:12:40.674626+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:18.911296+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.911296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.904205+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"454'10241","reported_seq":10325,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:48.836520+0000","last_change":"2024-03-24T12:10:42.558716+0000","last_active":"2024-03-24T12:12:48.836520+0000","last_peered":"2024-03-24T12:12:48.836520+0000","last_clean":"2024-03-24T12:12:48.836520+0000","last_became_active":"2024-03-24T12:10:42.558369+0000","last_became_peered":"2024-03-24T12:10:42.558369+0000","last_unstale":"2024-03-24T12:12:48.836520+0000","last_undegraded":"2024-03-24T12:12:48.836520+0000","last_fullsized":"2024-03-24T12:12:48.836520+0000","mapping_epoch":417,"log_start":"439'5200","ondisk_log_start":"439'5200","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":5041,"log_dups_size":0,"ondisk_log_size":5041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:53.430389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402718720,"num_objects":6145,"num_object_clones":0,"num_object_copies":18435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10241,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":872,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.669714+0000","last_change":"2024-03-24T12:11:19.790253+0000","last_active":"2024-03-24T12:12:40.669714+0000","last_peered":"2024-03-24T12:12:40.669714+0000","last_clean":"2024-03-24T12:12:40.669714+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:12:40.669714+0000","last_undegraded":"2024-03-24T12:12:40.669714+0000","last_fullsized":"2024-03-24T12:12:40.669714+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:19.790086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.790086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.399995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"450'24","reported_seq":910,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.679849+0000","last_change":"2024-03-24T12:08:34.327181+0000","last_active":"2024-03-24T12:12:40.679849+0000","last_peered":"2024-03-24T12:12:40.679849+0000","last_clean":"2024-03-24T12:12:40.679849+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:12:40.679849+0000","last_undegraded":"2024-03-24T12:12:40.679849+0000","last_fullsized":"2024-03-24T12:12:40.679849+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'18","last_scrub_stamp":"2024-03-24T12:08:34.327037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:08:34.327037+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":867,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.684864+0000","last_change":"2024-03-24T12:12:23.704478+0000","last_active":"2024-03-24T12:12:40.684864+0000","last_peered":"2024-03-24T12:12:40.684864+0000","last_clean":"2024-03-24T12:12:40.684864+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:12:40.684864+0000","last_undegraded":"2024-03-24T12:12:40.684864+0000","last_fullsized":"2024-03-24T12:12:40.684864+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:12:23.704204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.704204+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:27.358814+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":838,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671842+0000","last_change":"2024-03-24T12:12:14.425125+0000","last_active":"2024-03-24T12:12:40.671842+0000","last_peered":"2024-03-24T12:12:40.671842+0000","last_clean":"2024-03-24T12:12:40.671842+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:12:40.671842+0000","last_undegraded":"2024-03-24T12:12:40.671842+0000","last_fullsized":"2024-03-24T12:12:40.671842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:14.424990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.424990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:36.533158+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"450'25","reported_seq":897,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.685263+0000","last_change":"2024-03-24T12:11:40.464348+0000","last_active":"2024-03-24T12:12:40.685263+0000","last_peered":"2024-03-24T12:12:40.685263+0000","last_clean":"2024-03-24T12:12:40.685263+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:12:40.685263+0000","last_undegraded":"2024-03-24T12:12:40.685263+0000","last_fullsized":"2024-03-24T12:12:40.685263+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:40.464206+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:11:40.464206+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:55.591744+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":345,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682925+0000","last_change":"2024-03-24T12:11:20.390445+0000","last_active":"2024-03-24T12:12:40.682925+0000","last_peered":"2024-03-24T12:12:40.682925+0000","last_clean":"2024-03-24T12:12:40.682925+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:12:40.682925+0000","last_undegraded":"2024-03-24T12:12:40.682925+0000","last_fullsized":"2024-03-24T12:12:40.682925+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:20.390312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.390312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:53.431524+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":833,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.673599+0000","last_change":"2024-03-24T12:10:30.721092+0000","last_active":"2024-03-24T12:12:40.673599+0000","last_peered":"2024-03-24T12:12:40.673599+0000","last_clean":"2024-03-24T12:12:40.673599+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:12:40.673599+0000","last_undegraded":"2024-03-24T12:12:40.673599+0000","last_fullsized":"2024-03-24T12:12:40.673599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:30.721003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:30.721003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682343+0000","last_change":"2024-03-24T12:12:30.857644+0000","last_active":"2024-03-24T12:12:40.682343+0000","last_peered":"2024-03-24T12:12:40.682343+0000","last_clean":"2024-03-24T12:12:40.682343+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:12:40.682343+0000","last_undegraded":"2024-03-24T12:12:40.682343+0000","last_fullsized":"2024-03-24T12:12:40.682343+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:30.857510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:12:30.857510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:36.994339+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"450'18","reported_seq":872,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.673496+0000","last_change":"2024-03-24T12:10:25.685130+0000","last_active":"2024-03-24T12:12:40.673496+0000","last_peered":"2024-03-24T12:12:40.673496+0000","last_clean":"2024-03-24T12:12:40.673496+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:12:40.673496+0000","last_undegraded":"2024-03-24T12:12:40.673496+0000","last_fullsized":"2024-03-24T12:12:40.673496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:10:25.685009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.685009+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":870,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.673939+0000","last_change":"2024-03-24T12:11:43.032559+0000","last_active":"2024-03-24T12:12:40.673939+0000","last_peered":"2024-03-24T12:12:40.673939+0000","last_clean":"2024-03-24T12:12:40.673939+0000","last_became_active":"2024-03-24T11:52:15.138897+0000","last_became_peered":"2024-03-24T11:52:15.138897+0000","last_unstale":"2024-03-24T12:12:40.673939+0000","last_undegraded":"2024-03-24T12:12:40.673939+0000","last_fullsized":"2024-03-24T12:12:40.673939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.032403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.032403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:12.416512+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":683,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.678530+0000","last_change":"2024-03-24T12:10:25.718633+0000","last_active":"2024-03-24T12:12:40.678530+0000","last_peered":"2024-03-24T12:12:40.678530+0000","last_clean":"2024-03-24T12:12:40.678530+0000","last_became_active":"2024-03-24T12:08:36.165715+0000","last_became_peered":"2024-03-24T12:08:36.165715+0000","last_unstale":"2024-03-24T12:12:40.678530+0000","last_undegraded":"2024-03-24T12:12:40.678530+0000","last_fullsized":"2024-03-24T12:12:40.678530+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:25.718555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.718555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":855,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.678775+0000","last_change":"2024-03-24T12:10:19.030271+0000","last_active":"2024-03-24T12:12:40.678775+0000","last_peered":"2024-03-24T12:12:40.678775+0000","last_clean":"2024-03-24T12:12:40.678775+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:12:40.678775+0000","last_undegraded":"2024-03-24T12:12:40.678775+0000","last_fullsized":"2024-03-24T12:12:40.678775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.030172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.030172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.402118+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"450'35","reported_seq":895,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671445+0000","last_change":"2024-03-24T12:10:28.696002+0000","last_active":"2024-03-24T12:12:40.671445+0000","last_peered":"2024-03-24T12:12:40.671445+0000","last_clean":"2024-03-24T12:12:40.671445+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:12:40.671445+0000","last_undegraded":"2024-03-24T12:12:40.671445+0000","last_fullsized":"2024-03-24T12:12:40.671445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:28.695890+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:10:28.695890+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":734,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677161+0000","last_change":"2024-03-24T12:11:32.439386+0000","last_active":"2024-03-24T12:12:40.677161+0000","last_peered":"2024-03-24T12:12:40.677161+0000","last_clean":"2024-03-24T12:12:40.677161+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:12:40.677161+0000","last_undegraded":"2024-03-24T12:12:40.677161+0000","last_fullsized":"2024-03-24T12:12:40.677161+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.439301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.439301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564287+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":997,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677230+0000","last_change":"2024-03-24T12:10:15.754439+0000","last_active":"2024-03-24T12:12:40.677230+0000","last_peered":"2024-03-24T12:12:40.677230+0000","last_clean":"2024-03-24T12:12:40.677230+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:12:40.677230+0000","last_undegraded":"2024-03-24T12:12:40.677230+0000","last_fullsized":"2024-03-24T12:12:40.677230+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:15.754352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:15.754352+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":843,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.685061+0000","last_change":"2024-03-24T12:11:17.508810+0000","last_active":"2024-03-24T12:12:40.685061+0000","last_peered":"2024-03-24T12:12:40.685061+0000","last_clean":"2024-03-24T12:12:40.685061+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:12:40.685061+0000","last_undegraded":"2024-03-24T12:12:40.685061+0000","last_fullsized":"2024-03-24T12:12:40.685061+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:17.508697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:17.508697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"450'21","reported_seq":768,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677690+0000","last_change":"2024-03-24T12:12:25.566925+0000","last_active":"2024-03-24T12:12:40.677690+0000","last_peered":"2024-03-24T12:12:40.677690+0000","last_clean":"2024-03-24T12:12:40.677690+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:12:40.677690+0000","last_undegraded":"2024-03-24T12:12:40.677690+0000","last_fullsized":"2024-03-24T12:12:40.677690+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:12:25.566647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:25.566647+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:29.850688+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":729,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.683082+0000","last_change":"2024-03-24T12:10:34.022259+0000","last_active":"2024-03-24T12:12:40.683082+0000","last_peered":"2024-03-24T12:12:40.683082+0000","last_clean":"2024-03-24T12:12:40.683082+0000","last_became_active":"2024-03-24T12:10:30.443191+0000","last_became_peered":"2024-03-24T12:10:30.443191+0000","last_unstale":"2024-03-24T12:12:40.683082+0000","last_undegraded":"2024-03-24T12:12:40.683082+0000","last_fullsized":"2024-03-24T12:12:40.683082+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:34.022182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:34.022182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":1106,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671762+0000","last_change":"2024-03-24T12:10:19.664090+0000","last_active":"2024-03-24T12:12:40.671762+0000","last_peered":"2024-03-24T12:12:40.671762+0000","last_clean":"2024-03-24T12:12:40.671762+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:12:40.671762+0000","last_undegraded":"2024-03-24T12:12:40.671762+0000","last_fullsized":"2024-03-24T12:12:40.671762+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:10:19.664009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.664009+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.203241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":843,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.685068+0000","last_change":"2024-03-24T12:10:17.566669+0000","last_active":"2024-03-24T12:12:40.685068+0000","last_peered":"2024-03-24T12:12:40.685068+0000","last_clean":"2024-03-24T12:12:40.685068+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:12:40.685068+0000","last_undegraded":"2024-03-24T12:12:40.685068+0000","last_fullsized":"2024-03-24T12:12:40.685068+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:17.566562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:10:17.566562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765670+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"450'21","reported_seq":660,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674078+0000","last_change":"2024-03-24T12:11:28.054589+0000","last_active":"2024-03-24T12:12:40.674078+0000","last_peered":"2024-03-24T12:12:40.674078+0000","last_clean":"2024-03-24T12:12:40.674078+0000","last_became_active":"2024-03-24T11:59:19.097526+0000","last_became_peered":"2024-03-24T11:59:19.097526+0000","last_unstale":"2024-03-24T12:12:40.674078+0000","last_undegraded":"2024-03-24T12:12:40.674078+0000","last_fullsized":"2024-03-24T12:12:40.674078+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:11:28.054480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.054480+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:48.923586+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":871,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674623+0000","last_change":"2024-03-24T12:12:15.943971+0000","last_active":"2024-03-24T12:12:40.674623+0000","last_peered":"2024-03-24T12:12:40.674623+0000","last_clean":"2024-03-24T12:12:40.674623+0000","last_became_active":"2024-03-24T12:08:36.152805+0000","last_became_peered":"2024-03-24T12:08:36.152805+0000","last_unstale":"2024-03-24T12:12:40.674623+0000","last_undegraded":"2024-03-24T12:12:40.674623+0000","last_fullsized":"2024-03-24T12:12:40.674623+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:15.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:12:15.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:16.265713+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":847,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671751+0000","last_change":"2024-03-24T12:11:37.185727+0000","last_active":"2024-03-24T12:12:40.671751+0000","last_peered":"2024-03-24T12:12:40.671751+0000","last_clean":"2024-03-24T12:12:40.671751+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:12:40.671751+0000","last_undegraded":"2024-03-24T12:12:40.671751+0000","last_fullsized":"2024-03-24T12:12:40.671751+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:37.185629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.185629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":852,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682518+0000","last_change":"2024-03-24T12:11:31.446015+0000","last_active":"2024-03-24T12:12:40.682518+0000","last_peered":"2024-03-24T12:12:40.682518+0000","last_clean":"2024-03-24T12:12:40.682518+0000","last_became_active":"2024-03-24T12:08:36.146995+0000","last_became_peered":"2024-03-24T12:08:36.146995+0000","last_unstale":"2024-03-24T12:12:40.682518+0000","last_undegraded":"2024-03-24T12:12:40.682518+0000","last_fullsized":"2024-03-24T12:12:40.682518+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:31.445925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:30.836662+0000","last_clean_scrub_stamp":"2024-03-24T12:11:31.445925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.589782+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"450'27","reported_seq":885,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.680979+0000","last_change":"2024-03-24T12:11:28.574992+0000","last_active":"2024-03-24T12:12:40.680979+0000","last_peered":"2024-03-24T12:12:40.680979+0000","last_clean":"2024-03-24T12:12:40.680979+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:12:40.680979+0000","last_undegraded":"2024-03-24T12:12:40.680979+0000","last_fullsized":"2024-03-24T12:12:40.680979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:28.574841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.574841+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":889,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.681221+0000","last_change":"2024-03-24T12:12:10.078427+0000","last_active":"2024-03-24T12:12:40.681221+0000","last_peered":"2024-03-24T12:12:40.681221+0000","last_clean":"2024-03-24T12:12:40.681221+0000","last_became_active":"2024-03-24T12:08:36.174093+0000","last_became_peered":"2024-03-24T12:08:36.174093+0000","last_unstale":"2024-03-24T12:12:40.681221+0000","last_undegraded":"2024-03-24T12:12:40.681221+0000","last_fullsized":"2024-03-24T12:12:40.681221+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:10.078142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:10.078142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:39.723464+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":485,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676346+0000","last_change":"2024-03-24T12:09:38.661845+0000","last_active":"2024-03-24T12:12:40.676346+0000","last_peered":"2024-03-24T12:12:40.676346+0000","last_clean":"2024-03-24T12:12:40.676346+0000","last_became_active":"2024-03-24T12:08:36.166281+0000","last_became_peered":"2024-03-24T12:08:36.166281+0000","last_unstale":"2024-03-24T12:12:40.676346+0000","last_undegraded":"2024-03-24T12:12:40.676346+0000","last_fullsized":"2024-03-24T12:12:40.676346+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:09:38.661755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:09:38.661755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.563954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":838,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.670226+0000","last_change":"2024-03-24T12:10:10.700383+0000","last_active":"2024-03-24T12:12:40.670226+0000","last_peered":"2024-03-24T12:12:40.670226+0000","last_clean":"2024-03-24T12:12:40.670226+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:12:40.670226+0000","last_undegraded":"2024-03-24T12:12:40.670226+0000","last_fullsized":"2024-03-24T12:12:40.670226+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:10.700237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.700237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"450'14","reported_seq":873,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.669840+0000","last_change":"2024-03-24T12:12:15.299189+0000","last_active":"2024-03-24T12:12:40.669840+0000","last_peered":"2024-03-24T12:12:40.669840+0000","last_clean":"2024-03-24T12:12:40.669840+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:12:40.669840+0000","last_undegraded":"2024-03-24T12:12:40.669840+0000","last_fullsized":"2024-03-24T12:12:40.669840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:12:15.299057+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:12:15.299057+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:15.663338+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":747,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.678312+0000","last_change":"2024-03-24T12:10:12.784984+0000","last_active":"2024-03-24T12:12:40.678312+0000","last_peered":"2024-03-24T12:12:40.678312+0000","last_clean":"2024-03-24T12:12:40.678312+0000","last_became_active":"2024-03-24T12:08:36.166947+0000","last_became_peered":"2024-03-24T12:08:36.166947+0000","last_unstale":"2024-03-24T12:12:40.678312+0000","last_undegraded":"2024-03-24T12:12:40.678312+0000","last_fullsized":"2024-03-24T12:12:40.678312+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.784866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.784866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":854,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.673668+0000","last_change":"2024-03-24T12:10:37.907768+0000","last_active":"2024-03-24T12:12:40.673668+0000","last_peered":"2024-03-24T12:12:40.673668+0000","last_clean":"2024-03-24T12:12:40.673668+0000","last_became_active":"2024-03-24T11:52:19.384325+0000","last_became_peered":"2024-03-24T11:52:19.384325+0000","last_unstale":"2024-03-24T12:12:40.673668+0000","last_undegraded":"2024-03-24T12:12:40.673668+0000","last_fullsized":"2024-03-24T12:12:40.673668+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:37.907635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:37.907635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":852,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676123+0000","last_change":"2024-03-24T12:10:35.026578+0000","last_active":"2024-03-24T12:12:40.676123+0000","last_peered":"2024-03-24T12:12:40.676123+0000","last_clean":"2024-03-24T12:12:40.676123+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:12:40.676123+0000","last_undegraded":"2024-03-24T12:12:40.676123+0000","last_fullsized":"2024-03-24T12:12:40.676123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:35.026455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.026455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.402087+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"450'27","reported_seq":880,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.670588+0000","last_change":"2024-03-24T12:10:22.696939+0000","last_active":"2024-03-24T12:12:40.670588+0000","last_peered":"2024-03-24T12:12:40.670588+0000","last_clean":"2024-03-24T12:12:40.670588+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:12:40.670588+0000","last_undegraded":"2024-03-24T12:12:40.670588+0000","last_fullsized":"2024-03-24T12:12:40.670588+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:10:22.696749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:22.696749+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":869,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674738+0000","last_change":"2024-03-24T12:12:13.011602+0000","last_active":"2024-03-24T12:12:40.674738+0000","last_peered":"2024-03-24T12:12:40.674738+0000","last_clean":"2024-03-24T12:12:40.674738+0000","last_became_active":"2024-03-24T12:08:36.157290+0000","last_became_peered":"2024-03-24T12:08:36.157290+0000","last_unstale":"2024-03-24T12:12:40.674738+0000","last_undegraded":"2024-03-24T12:12:40.674738+0000","last_fullsized":"2024-03-24T12:12:40.674738+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:13.011469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_clean_scrub_stamp":"2024-03-24T12:12:13.011469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:32.523753+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":864,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677686+0000","last_change":"2024-03-24T12:11:45.406146+0000","last_active":"2024-03-24T12:12:40.677686+0000","last_peered":"2024-03-24T12:12:40.677686+0000","last_clean":"2024-03-24T12:12:40.677686+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:12:40.677686+0000","last_undegraded":"2024-03-24T12:12:40.677686+0000","last_fullsized":"2024-03-24T12:12:40.677686+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:45.406037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.406037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:50.099159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":829,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.672679+0000","last_change":"2024-03-24T12:10:13.863177+0000","last_active":"2024-03-24T12:12:40.672679+0000","last_peered":"2024-03-24T12:12:40.672679+0000","last_clean":"2024-03-24T12:12:40.672679+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:12:40.672679+0000","last_undegraded":"2024-03-24T12:12:40.672679+0000","last_fullsized":"2024-03-24T12:12:40.672679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.862457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.862457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"450'35","reported_seq":903,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.685479+0000","last_change":"2024-03-24T12:10:35.061280+0000","last_active":"2024-03-24T12:12:40.685479+0000","last_peered":"2024-03-24T12:12:40.685479+0000","last_clean":"2024-03-24T12:12:40.685479+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:12:40.685479+0000","last_undegraded":"2024-03-24T12:12:40.685479+0000","last_fullsized":"2024-03-24T12:12:40.685479+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:35.061151+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.061151+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":699,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676851+0000","last_change":"2024-03-24T12:10:24.690361+0000","last_active":"2024-03-24T12:12:40.676851+0000","last_peered":"2024-03-24T12:12:40.676851+0000","last_clean":"2024-03-24T12:12:40.676851+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:12:40.676851+0000","last_undegraded":"2024-03-24T12:12:40.676851+0000","last_fullsized":"2024-03-24T12:12:40.676851+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.690274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.690274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"274'9","reported_seq":1114,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.675016+0000","last_change":"2024-03-24T12:11:30.060710+0000","last_active":"2024-03-24T12:12:40.675016+0000","last_peered":"2024-03-24T12:12:40.675016+0000","last_clean":"2024-03-24T12:12:40.675016+0000","last_became_active":"2024-03-24T11:52:19.386952+0000","last_became_peered":"2024-03-24T11:52:19.386952+0000","last_unstale":"2024-03-24T12:12:40.675016+0000","last_undegraded":"2024-03-24T12:12:40.675016+0000","last_fullsized":"2024-03-24T12:12:40.675016+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:11:30.060478+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:11:30.060478+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.661103+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":871,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676669+0000","last_change":"2024-03-24T12:10:27.009973+0000","last_active":"2024-03-24T12:12:40.676669+0000","last_peered":"2024-03-24T12:12:40.676669+0000","last_clean":"2024-03-24T12:12:40.676669+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:12:40.676669+0000","last_undegraded":"2024-03-24T12:12:40.676669+0000","last_fullsized":"2024-03-24T12:12:40.676669+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:10:27.009877+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:27.009877+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.402152+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"450'27","reported_seq":919,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.675049+0000","last_change":"2024-03-24T12:12:19.829113+0000","last_active":"2024-03-24T12:12:40.675049+0000","last_peered":"2024-03-24T12:12:40.675049+0000","last_clean":"2024-03-24T12:12:40.675049+0000","last_became_active":"2024-03-24T12:08:36.156563+0000","last_became_peered":"2024-03-24T12:08:36.156563+0000","last_unstale":"2024-03-24T12:12:40.675049+0000","last_undegraded":"2024-03-24T12:12:40.675049+0000","last_fullsized":"2024-03-24T12:12:40.675049+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:12:19.829011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.829011+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:49.797083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":842,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.670346+0000","last_change":"2024-03-24T12:10:23.652133+0000","last_active":"2024-03-24T12:12:40.670346+0000","last_peered":"2024-03-24T12:12:40.670346+0000","last_clean":"2024-03-24T12:12:40.670346+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:12:40.670346+0000","last_undegraded":"2024-03-24T12:12:40.670346+0000","last_fullsized":"2024-03-24T12:12:40.670346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:10:23.652045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.652045+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":863,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.685578+0000","last_change":"2024-03-24T12:12:19.386952+0000","last_active":"2024-03-24T12:12:40.685578+0000","last_peered":"2024-03-24T12:12:40.685578+0000","last_clean":"2024-03-24T12:12:40.685578+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:12:40.685578+0000","last_undegraded":"2024-03-24T12:12:40.685578+0000","last_fullsized":"2024-03-24T12:12:40.685578+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:19.386800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.386800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:31.593985+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":447,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677330+0000","last_change":"2024-03-24T12:11:15.598297+0000","last_active":"2024-03-24T12:12:40.677330+0000","last_peered":"2024-03-24T12:12:40.677330+0000","last_clean":"2024-03-24T12:12:40.677330+0000","last_became_active":"2024-03-24T12:08:36.166973+0000","last_became_peered":"2024-03-24T12:08:36.166973+0000","last_unstale":"2024-03-24T12:12:40.677330+0000","last_undegraded":"2024-03-24T12:12:40.677330+0000","last_fullsized":"2024-03-24T12:12:40.677330+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:15.598041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:11:15.598041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.713677+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"450'46","reported_seq":817,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671885+0000","last_change":"2024-03-24T12:11:35.124433+0000","last_active":"2024-03-24T12:12:40.671885+0000","last_peered":"2024-03-24T12:12:40.671885+0000","last_clean":"2024-03-24T12:12:40.671885+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:12:40.671885+0000","last_undegraded":"2024-03-24T12:12:40.671885+0000","last_fullsized":"2024-03-24T12:12:40.671885+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:11:35.124291+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.124291+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:47.181527+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":862,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.674284+0000","last_change":"2024-03-24T12:11:15.939612+0000","last_active":"2024-03-24T12:12:40.674284+0000","last_peered":"2024-03-24T12:12:40.674284+0000","last_clean":"2024-03-24T12:12:40.674284+0000","last_became_active":"2024-03-24T11:52:15.138569+0000","last_became_peered":"2024-03-24T11:52:15.138569+0000","last_unstale":"2024-03-24T12:12:40.674284+0000","last_undegraded":"2024-03-24T12:12:40.674284+0000","last_fullsized":"2024-03-24T12:12:40.674284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:15.939499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:11:15.939499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.904235+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":607,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.683122+0000","last_change":"2024-03-24T12:12:29.766921+0000","last_active":"2024-03-24T12:12:40.683122+0000","last_peered":"2024-03-24T12:12:40.683122+0000","last_clean":"2024-03-24T12:12:40.683122+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:12:40.683122+0000","last_undegraded":"2024-03-24T12:12:40.683122+0000","last_fullsized":"2024-03-24T12:12:40.683122+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:29.766811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:12:29.766811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.373912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":848,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.684087+0000","last_change":"2024-03-24T12:10:13.573691+0000","last_active":"2024-03-24T12:12:40.684087+0000","last_peered":"2024-03-24T12:12:40.684087+0000","last_clean":"2024-03-24T12:12:40.684087+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:12:40.684087+0000","last_undegraded":"2024-03-24T12:12:40.684087+0000","last_fullsized":"2024-03-24T12:12:40.684087+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.573504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.573504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765644+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"450'29","reported_seq":909,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.672203+0000","last_change":"2024-03-24T12:11:41.236799+0000","last_active":"2024-03-24T12:12:40.672203+0000","last_peered":"2024-03-24T12:12:40.672203+0000","last_clean":"2024-03-24T12:12:40.672203+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:12:40.672203+0000","last_undegraded":"2024-03-24T12:12:40.672203+0000","last_fullsized":"2024-03-24T12:12:40.672203+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:11:41.236628+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:11:41.236628+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:56.929372+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":855,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671633+0000","last_change":"2024-03-24T12:11:20.004402+0000","last_active":"2024-03-24T12:12:40.671633+0000","last_peered":"2024-03-24T12:12:40.671633+0000","last_clean":"2024-03-24T12:12:40.671633+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:12:40.671633+0000","last_undegraded":"2024-03-24T12:12:40.671633+0000","last_fullsized":"2024-03-24T12:12:40.671633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:11:20.004266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.004266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":850,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.682387+0000","last_change":"2024-03-24T12:11:23.602693+0000","last_active":"2024-03-24T12:12:40.682387+0000","last_peered":"2024-03-24T12:12:40.682387+0000","last_clean":"2024-03-24T12:12:40.682387+0000","last_became_active":"2024-03-24T12:11:16.022628+0000","last_became_peered":"2024-03-24T12:11:16.022628+0000","last_unstale":"2024-03-24T12:12:40.682387+0000","last_undegraded":"2024-03-24T12:12:40.682387+0000","last_fullsized":"2024-03-24T12:12:40.682387+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:23.602435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.602435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":842,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.673391+0000","last_change":"2024-03-24T12:12:37.644881+0000","last_active":"2024-03-24T12:12:40.673391+0000","last_peered":"2024-03-24T12:12:40.673391+0000","last_clean":"2024-03-24T12:12:40.673391+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:12:40.673391+0000","last_undegraded":"2024-03-24T12:12:40.673391+0000","last_fullsized":"2024-03-24T12:12:40.673391+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:39.416426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:08:39.416426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"450'21","reported_seq":884,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.680262+0000","last_change":"2024-03-24T12:12:09.683485+0000","last_active":"2024-03-24T12:12:40.680262+0000","last_peered":"2024-03-24T12:12:40.680262+0000","last_clean":"2024-03-24T12:12:40.680262+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:12:40.680262+0000","last_undegraded":"2024-03-24T12:12:40.680262+0000","last_fullsized":"2024-03-24T12:12:40.680262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:12:09.683324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:09.683324+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:20.470987+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":884,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676481+0000","last_change":"2024-03-24T12:11:41.354046+0000","last_active":"2024-03-24T12:12:40.676481+0000","last_peered":"2024-03-24T12:12:40.676481+0000","last_clean":"2024-03-24T12:12:40.676481+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:12:40.676481+0000","last_undegraded":"2024-03-24T12:12:40.676481+0000","last_fullsized":"2024-03-24T12:12:40.676481+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:41.353830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:11:41.353830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:55.943062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":760,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.677424+0000","last_change":"2024-03-24T12:10:23.722476+0000","last_active":"2024-03-24T12:12:40.677424+0000","last_peered":"2024-03-24T12:12:40.677424+0000","last_clean":"2024-03-24T12:12:40.677424+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:12:40.677424+0000","last_undegraded":"2024-03-24T12:12:40.677424+0000","last_fullsized":"2024-03-24T12:12:40.677424+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:23.722321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.722321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.564166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":843,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.683128+0000","last_change":"2024-03-24T12:12:27.562418+0000","last_active":"2024-03-24T12:12:40.683128+0000","last_peered":"2024-03-24T12:12:40.683128+0000","last_clean":"2024-03-24T12:12:40.683128+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:12:40.683128+0000","last_undegraded":"2024-03-24T12:12:40.683128+0000","last_fullsized":"2024-03-24T12:12:40.683128+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:27.562289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:12:27.562289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:29.066954+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"450'11","reported_seq":868,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.681074+0000","last_change":"2024-03-24T12:11:36.552488+0000","last_active":"2024-03-24T12:12:40.681074+0000","last_peered":"2024-03-24T12:12:40.681074+0000","last_clean":"2024-03-24T12:12:40.681074+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:12:40.681074+0000","last_undegraded":"2024-03-24T12:12:40.681074+0000","last_fullsized":"2024-03-24T12:12:40.681074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:11:36.552333+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:11:36.552333+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:01.517180+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":880,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.679162+0000","last_change":"2024-03-24T12:11:13.237903+0000","last_active":"2024-03-24T12:12:40.679162+0000","last_peered":"2024-03-24T12:12:40.679162+0000","last_clean":"2024-03-24T12:12:40.679162+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:12:40.679162+0000","last_undegraded":"2024-03-24T12:12:40.679162+0000","last_fullsized":"2024-03-24T12:12:40.679162+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:13.237829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.237829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.385120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":872,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.675922+0000","last_change":"2024-03-24T12:11:23.134759+0000","last_active":"2024-03-24T12:12:40.675922+0000","last_peered":"2024-03-24T12:12:40.675922+0000","last_clean":"2024-03-24T12:12:40.675922+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:12:40.675922+0000","last_undegraded":"2024-03-24T12:12:40.675922+0000","last_fullsized":"2024-03-24T12:12:40.675922+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:23.134643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.011992+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.134643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:42.402209+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":766,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.676482+0000","last_change":"2024-03-24T12:10:13.888112+0000","last_active":"2024-03-24T12:12:40.676482+0000","last_peered":"2024-03-24T12:12:40.676482+0000","last_clean":"2024-03-24T12:12:40.676482+0000","last_became_active":"2024-03-24T12:08:36.164166+0000","last_became_peered":"2024-03-24T12:08:36.164166+0000","last_unstale":"2024-03-24T12:12:40.676482+0000","last_undegraded":"2024-03-24T12:12:40.676482+0000","last_fullsized":"2024-03-24T12:12:40.676482+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.888002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.888002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:45.563988+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"450'25","reported_seq":888,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.679240+0000","last_change":"2024-03-24T12:11:44.652906+0000","last_active":"2024-03-24T12:12:40.679240+0000","last_peered":"2024-03-24T12:12:40.679240+0000","last_clean":"2024-03-24T12:12:40.679240+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:12:40.679240+0000","last_undegraded":"2024-03-24T12:12:40.679240+0000","last_fullsized":"2024-03-24T12:12:40.679240+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:44.652731+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:11:44.652731+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:13.964162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":852,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.672806+0000","last_change":"2024-03-24T12:08:40.183479+0000","last_active":"2024-03-24T12:12:40.672806+0000","last_peered":"2024-03-24T12:12:40.672806+0000","last_clean":"2024-03-24T12:12:40.672806+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:12:40.672806+0000","last_undegraded":"2024-03-24T12:12:40.672806+0000","last_fullsized":"2024-03-24T12:12:40.672806+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:08:40.183311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:08:40.183311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.673427+0000","last_change":"2024-03-24T12:12:22.837708+0000","last_active":"2024-03-24T12:12:40.673427+0000","last_peered":"2024-03-24T12:12:40.673427+0000","last_clean":"2024-03-24T12:12:40.673427+0000","last_became_active":"2024-03-24T12:08:36.152717+0000","last_became_peered":"2024-03-24T12:08:36.152717+0000","last_unstale":"2024-03-24T12:12:40.673427+0000","last_undegraded":"2024-03-24T12:12:40.673427+0000","last_fullsized":"2024-03-24T12:12:40.673427+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:22.837495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:22.837495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.498764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":843,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.680168+0000","last_change":"2024-03-24T12:10:12.547927+0000","last_active":"2024-03-24T12:12:40.680168+0000","last_peered":"2024-03-24T12:12:40.680168+0000","last_clean":"2024-03-24T12:12:40.680168+0000","last_became_active":"2024-03-24T12:08:36.165334+0000","last_became_peered":"2024-03-24T12:08:36.165334+0000","last_unstale":"2024-03-24T12:12:40.680168+0000","last_undegraded":"2024-03-24T12:12:40.680168+0000","last_fullsized":"2024-03-24T12:12:40.680168+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.547803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.547803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:41.765590+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"450'46","reported_seq":936,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.670701+0000","last_change":"2024-03-24T12:08:47.101220+0000","last_active":"2024-03-24T12:12:40.670701+0000","last_peered":"2024-03-24T12:12:40.670701+0000","last_clean":"2024-03-24T12:12:40.670701+0000","last_became_active":"2024-03-24T12:08:36.154117+0000","last_became_peered":"2024-03-24T12:08:36.154117+0000","last_unstale":"2024-03-24T12:12:40.670701+0000","last_undegraded":"2024-03-24T12:12:40.670701+0000","last_fullsized":"2024-03-24T12:12:40.670701+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:47.101082+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:08:47.101082+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":856,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.689494+0000","last_change":"2024-03-24T12:12:12.477784+0000","last_active":"2024-03-24T12:12:40.689494+0000","last_peered":"2024-03-24T12:12:40.689494+0000","last_clean":"2024-03-24T12:12:40.689494+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:12:40.689494+0000","last_undegraded":"2024-03-24T12:12:40.689494+0000","last_fullsized":"2024-03-24T12:12:40.689494+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:12.477654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:12:12.477654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:27.896776+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":319,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.669559+0000","last_change":"2024-03-24T12:12:27.315018+0000","last_active":"2024-03-24T12:12:40.669559+0000","last_peered":"2024-03-24T12:12:40.669559+0000","last_clean":"2024-03-24T12:12:40.669559+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:12:40.669559+0000","last_undegraded":"2024-03-24T12:12:40.669559+0000","last_fullsized":"2024-03-24T12:12:40.669559+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:27.314894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:12:27.314894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:52.284429+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":837,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-24T12:12:40.671223+0000","last_change":"2024-03-24T12:11:44.456204+0000","last_active":"2024-03-24T12:12:40.671223+0000","last_peered":"2024-03-24T12:12:40.671223+0000","last_clean":"2024-03-24T12:12:40.671223+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:12:40.671223+0000","last_undegraded":"2024-03-24T12:12:40.671223+0000","last_fullsized":"2024-03-24T12:12:40.671223+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:44.455779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:44.455779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:53.752100+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6576406552,"num_objects":100349,"num_object_clones":0,"num_object_copies":301047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100349,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163913,"num_write_kb":8456321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19654127616,"data_stored":19654115400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79013,"ondisk_log_size":79013,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":390,"ondisk_log_size":390,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":428,"num_read_kb":423,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1551,"num_read_kb":1308,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":1078,"ondisk_log_size":1078,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005306,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3239656,"kb_used_data":2442144,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":90505496,"statfs":{"total":95995035648,"available":92677627904,"internally_reserved":0,"allocated":2500755456,"data_stored":2500340283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":816642506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332353,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4000060,"kb_used_data":3256692,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":89745092,"statfs":{"total":95995035648,"available":91898974208,"internally_reserved":0,"allocated":3334852608,"data_stored":3334417265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":761199046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575129,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4904244,"kb_used_data":4024748,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":88840908,"statfs":{"total":95995035648,"available":90973089792,"internally_reserved":0,"allocated":4121341952,"data_stored":4120913912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738655,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3278640,"kb_used_data":2381096,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":90466512,"statfs":{"total":95995035648,"available":92637708288,"internally_reserved":0,"allocated":2438242304,"data_stored":2437818305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411605,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1374924,"kb_used_data":788420,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92370228,"statfs":{"total":95995035648,"available":94587113472,"internally_reserved":0,"allocated":807342080,"data_stored":806954850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117263,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3722116,"kb_used_data":2838524,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":90023036,"statfs":{"total":95995035648,"available":92183588864,"internally_reserved":0,"allocated":2906648576,"data_stored":2906205522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":904788423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921034,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2732152,"kb_used_data":2001264,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":91013000,"statfs":{"total":95995035648,"available":93197312000,"internally_reserved":0,"allocated":2049294336,"data_stored":2048862235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455540,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1894648,"kb_used_data":1569264,"kb_used_omap":7,"kb_used_meta":325368,"kb_avail":91850504,"statfs":{"total":95995035648,"available":94054916096,"internally_reserved":0,"allocated":1606926336,"data_stored":1606494990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":333176944},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2424045568,"data_stored":2424045568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4113567744,"data_stored":4113563672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":804716544,"data_stored":804716544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2887319552,"data_stored":2887319552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2048135168,"data_stored":2048131096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1591345152,"data_stored":1591345152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3295744000,"data_stored":3295739928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2489253888,"data_stored":2489253888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:12:52.009 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T12:12:52.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_4 pg_num 26 2024-03-24T12:12:52.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:52.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:52.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:52.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:52.253 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:52.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:52.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:52.397 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:52.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:52.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:52.524 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T12:12:52.542 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:52.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:52.603 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:52.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:52.679 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:52.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:52.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:52.822 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:52.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:52.885 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:52.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:52.975 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:52.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:53.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:53.108 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:53.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:53.210 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:53.255 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:53.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:53.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:53.294 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:53 smithi064 ceph-mon[30223]: pgmap v974: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:12:53.295 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:53 smithi064 ceph-mon[42243]: pgmap v974: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:12:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:53 smithi193 ceph-mon[33593]: pgmap v974: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:12:53.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:53.398 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:53.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:53.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:53.577 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:53.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:53.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:53.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:53.817 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:53.854 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:53.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:53.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:53.994 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:53.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:12:54.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:54.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:54.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:54.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:54.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1866033905' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:54.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:54 smithi064 ceph-mon[30223]: from='client.35987 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:54.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1866033905' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:54.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:54 smithi064 ceph-mon[42243]: from='client.35987 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1866033905' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:54 smithi193 ceph-mon[33593]: from='client.35987 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:12:54.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:54.419 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:54.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:54.521 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:54.555 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:54.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:54.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:54.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:54.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:54.804 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:54.838 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:54.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:54.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:54.976 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:54.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:55.078 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:55.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:55.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:55.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:55.253 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:55.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[30223]: pgmap v975: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-03-24T12:12:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[30223]: from='client.35987 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:12:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[30223]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T12:12:55.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[30223]: 5.12 scrub starts 2024-03-24T12:12:55.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[30223]: 5.12 scrub ok 2024-03-24T12:12:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[42243]: pgmap v975: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-03-24T12:12:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[42243]: from='client.35987 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:12:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[42243]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T12:12:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[42243]: 5.12 scrub starts 2024-03-24T12:12:55.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:55 smithi064 ceph-mon[42243]: 5.12 scrub ok 2024-03-24T12:12:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:55 smithi193 ceph-mon[33593]: pgmap v975: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-03-24T12:12:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:55 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:12:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:55 smithi193 ceph-mon[33593]: from='client.35987 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:12:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:55 smithi193 ceph-mon[33593]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T12:12:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:55 smithi193 ceph-mon[33593]: 5.12 scrub starts 2024-03-24T12:12:55.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:55 smithi193 ceph-mon[33593]: 5.12 scrub ok 2024-03-24T12:12:55.355 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:55.389 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:55.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:55.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:55.521 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:55.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:55.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:55.671 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:55.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:55.754 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T12:12:55.754 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-03-24T12:12:55.754 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-03-24T12:12:55.754 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-03-24T12:12:55.754 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T12:12:55.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:55.813 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:55.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:55.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:55.951 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:55.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:56.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:56.092 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:56.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:56.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:56.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:56 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T12:12:56.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:56 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T12:12:56.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:56 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:12:56.208 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:56 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:12:56.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:56 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T12:12:56.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:56 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T12:12:56.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:56 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:12:56.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:56 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:12:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:56.229 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:56.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:56.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:56 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T12:12:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:56 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T12:12:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:56 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:12:56.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:56 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:12:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:56.379 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:56.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:56.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:56.512 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:56.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:56.665 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T12:12:56.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:56.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:56.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:56.923 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:12:56.923 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:12:56.923 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:12:56.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:12:56.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:12:56.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:12:56.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:12:56.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:12:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:12:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:12:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:12:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:12:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:12:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:12:56.926 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:12:56.926 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:12:56.976 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:56.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:57.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:12:57.115 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:12:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:57.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:12:57.218 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:57.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T12:12:57.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: pgmap v977: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:12:57.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T12:12:57.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T12:12:57.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T12:12:57.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T12:12:57.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T12:12:57.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T12:12:57.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1599211303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:57.255 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:57.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:57.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: pgmap v977: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1599211303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: pgmap v977: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:12:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T12:12:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T12:12:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T12:12:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T12:12:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T12:12:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T12:12:57.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1599211303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:12:57.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:57.386 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:57.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:57.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:57.524 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:57.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:57.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:57.626 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:12:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:57.666 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:57.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:12:57.696 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:12:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:12:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:57.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:57.810 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:57.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:57.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:57.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:12:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:57.945 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:57.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:58.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:58.094 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:58.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:58.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 6.1f scrub ok 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 2.1e scrub starts 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 2.1e scrub ok 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:12:58.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 6.1f scrub ok 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 2.1e scrub starts 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 2.1e scrub ok 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:12:58.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:58.229 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:58.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:58.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 6.1f scrub ok 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 2.1e scrub starts 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 2.1e scrub ok 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:12:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:58.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:12:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:12:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:58.375 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:58.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:58.443 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:12:58.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:58.528 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:58.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:58.629 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:58.664 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:58.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:58.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:12:58.816 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:58.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:58.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:12:58.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:58.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:58.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:59.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:59.092 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:59.092 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 11 pg_num to 26 2024-03-24T12:12:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:59.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:59.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:12:59.230 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:59.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:12:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: pgmap v978: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:12:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 11.3 scrub starts 2024-03-24T12:12:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 1.1 scrub starts 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: 1.1 scrub ok 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1830746925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:12:59.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[30223]: from='client.35996 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: pgmap v978: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 11.3 scrub starts 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 1.1 scrub starts 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: 1.1 scrub ok 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1830746925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:12:59.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:12:59 smithi064 ceph-mon[42243]: from='client.35996 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:12:59.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:12:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: pgmap v978: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:12:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 11.3 scrub starts 2024-03-24T12:12:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T12:12:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T12:12:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:12:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:12:59.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T12:12:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T12:12:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 1.1 scrub starts 2024-03-24T12:12:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: 1.1 scrub ok 2024-03-24T12:12:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1830746925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:12:59.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:12:59 smithi193 ceph-mon[33593]: from='client.35996 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:12:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:59.363 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:59.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:12:59.464 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:12:59.500 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:12:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:59.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:12:59.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:59.633 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:59.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:12:59.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:12:59.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:12:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:59.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:12:59.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:12:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:12:59.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:12:59.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:00.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:00.072 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:00.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:00.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:00.250 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:00.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:00.301 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:13:00.302 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: from='client.35996 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T12:13:00.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: 11.3 scrub ok 2024-03-24T12:13:00.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:00 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:13:00.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:00.377 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:00.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:00.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:00.480 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T12:13:00.480 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T12:13:00.480 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: from='client.35996 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: 11.3 scrub ok 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: from='client.35996 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T12:13:00.481 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:13:00.482 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:13:00.482 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T12:13:00.482 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T12:13:00.482 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: 11.3 scrub ok 2024-03-24T12:13:00.482 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:00 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:13:00.504 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:00.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:00.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:00.630 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:00.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:00.731 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:00.756 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:00.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:00.857 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:00.884 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:00.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:00.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:01.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:01.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:01.152 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:01.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:01.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:01.292 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:01.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:01.394 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:01.428 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:01.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:01.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:01.530 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:01 smithi064 ceph-mon[30223]: pgmap v980: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T12:13:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:01 smithi064 ceph-mon[30223]: 11.1 scrub starts 2024-03-24T12:13:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:01 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:13:01.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:01 smithi064 ceph-mon[30223]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T12:13:01.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:01 smithi064 ceph-mon[42243]: pgmap v980: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T12:13:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:01 smithi064 ceph-mon[42243]: 11.1 scrub starts 2024-03-24T12:13:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:01 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:13:01.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:01 smithi064 ceph-mon[42243]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T12:13:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:01.563 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:01.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:01 smithi193 ceph-mon[33593]: pgmap v980: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T12:13:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:01 smithi193 ceph-mon[33593]: 11.1 scrub starts 2024-03-24T12:13:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:01 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:13:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:01 smithi193 ceph-mon[33593]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T12:13:01.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:01.689 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:01.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:01.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:01.830 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:01.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:01.932 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:01.967 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:01.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:02.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:02.106 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:02.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:02.207 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:02.236 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:02.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:02.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:02.363 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:02.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:02.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T12:13:02.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T12:13:02.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[42243]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T12:13:02.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[42243]: 11.4 scrub starts 2024-03-24T12:13:02.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:13:02.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T12:13:02.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T12:13:02.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[30223]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T12:13:02.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[30223]: 11.4 scrub starts 2024-03-24T12:13:02.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:02 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:13:02.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:02.493 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:02.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:02 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T12:13:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:02 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T12:13:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:02 smithi193 ceph-mon[33593]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T12:13:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:02 smithi193 ceph-mon[33593]: 11.4 scrub starts 2024-03-24T12:13:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:02 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:13:02.594 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:02.618 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:02.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:02.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:02.748 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:02.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:02.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:13:02.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:02.875 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:02.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:02.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:03.001 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:03.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:03.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:03.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:03.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:03.254 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:03.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:03.355 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:03.380 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:03.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:03.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:03.507 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:03.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:03.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.0 scrub starts 2024-03-24T12:13:03.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.6 scrub starts 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: pgmap v983: 168 pgs: 10 unknown, 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.4 scrub ok 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.7 scrub starts 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.0 scrub ok 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.3 scrub starts 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.6 scrub ok 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.5 scrub starts 2024-03-24T12:13:03.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.7 scrub ok 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.3 scrub ok 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 11.5 scrub ok 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 6.19 scrub starts 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[30223]: 6.19 scrub ok 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.0 scrub starts 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.6 scrub starts 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: pgmap v983: 168 pgs: 10 unknown, 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.4 scrub ok 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.7 scrub starts 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.0 scrub ok 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T12:13:03.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.3 scrub starts 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.6 scrub ok 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.5 scrub starts 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.7 scrub ok 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.3 scrub ok 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 11.5 scrub ok 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 6.19 scrub starts 2024-03-24T12:13:03.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:03 smithi064 ceph-mon[42243]: 6.19 scrub ok 2024-03-24T12:13:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.0 scrub starts 2024-03-24T12:13:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.6 scrub starts 2024-03-24T12:13:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: pgmap v983: 168 pgs: 10 unknown, 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.4 scrub ok 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.7 scrub starts 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.0 scrub ok 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.3 scrub starts 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.6 scrub ok 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.5 scrub starts 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.7 scrub ok 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.3 scrub ok 2024-03-24T12:13:03.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 11.5 scrub ok 2024-03-24T12:13:03.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 6.19 scrub starts 2024-03-24T12:13:03.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:03 smithi193 ceph-mon[33593]: 6.19 scrub ok 2024-03-24T12:13:03.609 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:03.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:03.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:03.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:03.757 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:03.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:03.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:03.884 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:03.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:03.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:04.010 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:04.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:04.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:04.136 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:04.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:04.157 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:13:04.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:04.275 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:04.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 11.2 scrub starts 2024-03-24T12:13:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 11.2 scrub ok 2024-03-24T12:13:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T12:13:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 11.1 scrub starts 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 11.1 scrub ok 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1931893498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:13:04.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[30223]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 11.2 scrub starts 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 11.2 scrub ok 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 11.1 scrub starts 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 11.1 scrub ok 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T12:13:04.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1931893498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:04.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:04.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:13:04.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:13:04.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:13:04.304 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:04 smithi064 ceph-mon[42243]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T12:13:04.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:04.412 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:04.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:04.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:04.556 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:04.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 11.2 scrub starts 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 11.2 scrub ok 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 11.1 scrub starts 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 11.1 scrub ok 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T12:13:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1931893498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:04.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:04.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:13:04.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:13:04.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:13:04.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:04 smithi193 ceph-mon[33593]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T12:13:04.658 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:04.693 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:04.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:04.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:04.833 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:04.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:04.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:04.979 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:04.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:05.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:05.118 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:05.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:05.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:05.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:05.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:05.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:05.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.9 scrub starts 2024-03-24T12:13:05.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: pgmap v986: 168 pgs: 2 active+clean+scrubbing, 3 unknown, 163 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:13:05.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.9 scrub ok 2024-03-24T12:13:05.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T12:13:05.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 458'11253 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11239 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 458'11261 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 458'11218 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 458'11168 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11448 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11271 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11261 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.10 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11253 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11218 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11168 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T12:13:05.489 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.9 scrub starts 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: pgmap v986: 168 pgs: 2 active+clean+scrubbing, 3 unknown, 163 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.9 scrub ok 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 458'11253 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11239 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 458'11261 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 458'11218 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 458'11168 2024-03-24T12:13:05.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11448 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11271 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11261 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.10 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11253 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11218 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11168 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T12:13:05.491 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:05 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T12:13:05.506 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:13:05.506 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:13:05.506 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-24T12:13:05.506 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.007483 2024-03-24T12:13:05.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 6 0.007482750665621318 2024-03-24T12:13:05.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.9 scrub starts 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: pgmap v986: 168 pgs: 2 active+clean+scrubbing, 3 unknown, 163 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.9 scrub ok 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 458'11253 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11239 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 458'11261 2024-03-24T12:13:05.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 458'11218 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 458'11168 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11448 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11271 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11261 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.10 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11253 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11218 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 458'11195 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11053 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 458'11168 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 458'11445 2024-03-24T12:13:05.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T12:13:05.588 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:05 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T12:13:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:05.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:05.629 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:13:05.675 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:13:05.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:05.716 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:13:05.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:05.735 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:05.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:05.759 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:13:05.800 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:13:05.836 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:05.842 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:13:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:05.869 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:05.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:05.890 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:13:05.944 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:13:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:13:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:13:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:13:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:13:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:13:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:13:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:13:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:13:05.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:06.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:06.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:06.177 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:06.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:06.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:06.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:06.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:06.462 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:06.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:06.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:06 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3071544393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:06.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:06 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3071544393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:06.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:06.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:06 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3071544393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:06.603 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:06.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:06.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:06.744 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:06.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:06.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:06.884 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:06.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:06.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:07.021 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:07.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:07.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:07.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:07.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:07.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:13:07.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:07.330 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:07.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:07 smithi193 ceph-mon[33593]: pgmap v988: 168 pgs: 1 active+remapped+backfill_toofull, 3 remapped+peering, 2 active+clean+scrubbing, 2 unknown, 160 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 5184/237081 objects misplaced (2.187%); 0 B/s, 0 objects/s recovering 2024-03-24T12:13:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:07 smithi193 ceph-mon[33593]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:07.431 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:07.466 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:07.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:07.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:13:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:13:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:07.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:07 smithi064 ceph-mon[42243]: pgmap v988: 168 pgs: 1 active+remapped+backfill_toofull, 3 remapped+peering, 2 active+clean+scrubbing, 2 unknown, 160 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 5184/237081 objects misplaced (2.187%); 0 B/s, 0 objects/s recovering 2024-03-24T12:13:07.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:07 smithi064 ceph-mon[42243]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:07 smithi064 ceph-mon[30223]: pgmap v988: 168 pgs: 1 active+remapped+backfill_toofull, 3 remapped+peering, 2 active+clean+scrubbing, 2 unknown, 160 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 5184/237081 objects misplaced (2.187%); 0 B/s, 0 objects/s recovering 2024-03-24T12:13:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:07 smithi064 ceph-mon[30223]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:07.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:07.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:07.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:07.747 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:07.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:07.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:07.889 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:07.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:07.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:08.029 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:08.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:08.130 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:08.163 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:08.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:08.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:08.309 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:08.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:08.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:13:07 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:13:08.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:08.458 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:08.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:08.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:08.594 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:08.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:08.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:08.737 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:08.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:08.838 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:08.875 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:08.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:08.977 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:09.012 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:09.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:09.114 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:09.146 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:09.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:09.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:09.280 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:09.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:09.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:09.412 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:09.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:09.486 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:09.493 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:09.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:09.544 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:09.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:09 smithi064 ceph-mon[30223]: pgmap v989: 168 pgs: 3 active+remapped+backfill_toofull, 3 remapped+peering, 4 active+remapped+backfilling, 158 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 33480/244404 objects misplaced (13.699%); 15 MiB/s, 248 objects/s recovering 2024-03-24T12:13:09.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:09 smithi064 ceph-mon[42243]: pgmap v989: 168 pgs: 3 active+remapped+backfill_toofull, 3 remapped+peering, 4 active+remapped+backfilling, 158 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 33480/244404 objects misplaced (13.699%); 15 MiB/s, 248 objects/s recovering 2024-03-24T12:13:09.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:09 smithi193 ceph-mon[33593]: pgmap v989: 168 pgs: 3 active+remapped+backfill_toofull, 3 remapped+peering, 4 active+remapped+backfilling, 158 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 33480/244404 objects misplaced (13.699%); 15 MiB/s, 248 objects/s recovering 2024-03-24T12:13:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:09.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:09.699 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:09.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:09.801 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:09.837 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:09.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:09.939 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:09.976 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:09.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:10.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:10.127 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:10.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:10.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:10.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:10.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:13:10.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:10.405 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:10.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:10.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:10.540 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:10.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:10.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:10.714 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:10.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:10.815 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:10.853 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:10.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:10.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:10.995 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:10.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:11.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:11.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:11.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:11.281 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:11.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:11.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:11.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:11.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:11.467 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:11 smithi064 ceph-mon[30223]: pgmap v990: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 687 B/s rd, 0 op/s; 46632/241644 objects misplaced (19.298%); 22 MiB/s, 346 objects/s recovering 2024-03-24T12:13:11.467 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:11 smithi064 ceph-mon[42243]: pgmap v990: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 687 B/s rd, 0 op/s; 46632/241644 objects misplaced (19.298%); 22 MiB/s, 346 objects/s recovering 2024-03-24T12:13:11.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:11.581 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:11.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:11 smithi193 ceph-mon[33593]: pgmap v990: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 687 B/s rd, 0 op/s; 46632/241644 objects misplaced (19.298%); 22 MiB/s, 346 objects/s recovering 2024-03-24T12:13:11.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:11.727 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:11.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:11.829 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:11.866 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:11.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:11.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:12.004 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:12.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:12.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:12.150 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:12.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:12.251 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:12.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:12.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:12.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:12.424 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:12.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:12.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:12.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:12 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:12 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:12 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:12.652 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:12.687 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:12.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:12.789 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:12.824 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:12.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:12.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:12.964 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:12.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:13.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:13.105 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:13.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:13.207 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:13.237 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:13.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:13.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:13.382 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:13.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:13.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:13.524 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:13.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:13 smithi064 ceph-mon[30223]: pgmap v991: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 607 B/s rd, 0 op/s; 46632/241644 objects misplaced (19.298%); 19 MiB/s, 305 objects/s recovering 2024-03-24T12:13:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:13 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:13 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:13.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:13 smithi064 ceph-mon[42243]: pgmap v991: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 607 B/s rd, 0 op/s; 46632/241644 objects misplaced (19.298%); 19 MiB/s, 305 objects/s recovering 2024-03-24T12:13:13.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:13 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:13.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:13 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:13 smithi193 ceph-mon[33593]: pgmap v991: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 607 B/s rd, 0 op/s; 46632/241644 objects misplaced (19.298%); 19 MiB/s, 305 objects/s recovering 2024-03-24T12:13:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:13 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:13 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:13.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:13.667 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:13.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:13.769 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:13.800 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:13.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:13.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:13.937 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:13.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:14.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:14.079 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:14.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:14.180 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:14.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:14.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:14.317 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:14.345 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:14.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:14.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:14.483 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:14.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:14.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:14.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:14.615 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:14.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:14.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:14.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:14.755 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:14.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:14.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:14.887 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:14.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:14.989 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:15.018 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:15.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:15.120 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:15.148 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:15.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:15.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:15.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:15.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:15 smithi064 ceph-mon[30223]: pgmap v992: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 40638/228399 objects misplaced (17.793%); 34 MiB/s, 549 objects/s recovering 2024-03-24T12:13:15.266 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:15 smithi064 ceph-mon[42243]: pgmap v992: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 40638/228399 objects misplaced (17.793%); 34 MiB/s, 549 objects/s recovering 2024-03-24T12:13:15.277 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:15.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:15.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:15.418 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:15.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:15.519 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:15 smithi064 ceph-mon[30223]: 11.14 scrub starts 2024-03-24T12:13:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:15 smithi064 ceph-mon[30223]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T12:13:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:15 smithi064 ceph-mon[42243]: 11.14 scrub starts 2024-03-24T12:13:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:15 smithi064 ceph-mon[42243]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T12:13:15.554 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:15.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:15 smithi193 ceph-mon[33593]: pgmap v992: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 40638/228399 objects misplaced (17.793%); 34 MiB/s, 549 objects/s recovering 2024-03-24T12:13:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:15 smithi193 ceph-mon[33593]: 11.14 scrub starts 2024-03-24T12:13:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:15 smithi193 ceph-mon[33593]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T12:13:15.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:15.700 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:15.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:15.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:15.839 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:15.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:15.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:15.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:15.985 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:15.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:16.087 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:16.129 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:16.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:16.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:16.273 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:16.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:16.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:16.420 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:16.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:16.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:16 smithi064 ceph-mon[30223]: 3.0 deep-scrub starts 2024-03-24T12:13:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:16 smithi064 ceph-mon[30223]: 3.0 deep-scrub ok 2024-03-24T12:13:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:16 smithi064 ceph-mon[30223]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T12:13:16.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:16 smithi064 ceph-mon[42243]: 3.0 deep-scrub starts 2024-03-24T12:13:16.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:16 smithi064 ceph-mon[42243]: 3.0 deep-scrub ok 2024-03-24T12:13:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:16 smithi064 ceph-mon[42243]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T12:13:16.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:16.561 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:16.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:16.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:16 smithi193 ceph-mon[33593]: 3.0 deep-scrub starts 2024-03-24T12:13:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:16 smithi193 ceph-mon[33593]: 3.0 deep-scrub ok 2024-03-24T12:13:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:16 smithi193 ceph-mon[33593]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T12:13:16.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:16.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:16.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:16.805 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:16.840 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:16.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:16.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:16.974 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:17.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:17.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:17.039 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:13:17.039 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:13:17.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.16023.0:187082 11.19 11:9e2069be:::benchmark_data_smithi064_206665_object54949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-24T12:13:17.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:10.412304+0000", 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.010469081, 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010503863, 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.16023", 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 187082 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.412304+0000", 2024-03-24T12:13:17.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.412304+0000", 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.412306+0000", 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.356e-06 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:13:17.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.412312+0000", 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8220000000000003e-06 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.412324+0000", 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1884e-05 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.412332+0000", 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9529999999999999e-06 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:13:17.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.415158+0000", 2024-03-24T12:13:17.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028257920000000001 2024-03-24T12:13:17.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:13:17.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.415298+0000", 2024-03-24T12:13:17.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014008599999999999 2024-03-24T12:13:17.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,7", 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.415636+0000", 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033819600000000003 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.418898+0000", 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032620650000000002 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.419186+0000", 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028825200000000002 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.419495+0000", 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000308282 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.421670+0000", 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021749949999999999 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:13:17.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:187086 11.f e461/417 11:f7967738:::benchmark_data_smithi064_206665_object54953:head v 461'11873, mlcod=461'11871)", 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:10.420472+0000", 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0023016159999999998, 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024589120000000002, 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.420472+0000", 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:13:17.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.420472+0000", 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.420473+0000", 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0049999999999999e-06 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.420479+0000", 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8750000000000002e-06 2024-03-24T12:13:17.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.420489+0000", 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4450000000000002e-06 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.420496+0000", 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7959999999999998e-06 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.420707+0000", 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021160100000000001 2024-03-24T12:13:17.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.420737+0000", 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0029e-05 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:187095 11.1 e461/457 11:859fbd77:::benchmark_data_smithi064_206665_object54962:head v 461'11446, mlcod=461'11444)", 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:10.422035+0000", 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.000737763, 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098483800000000003, 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:13:17.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.422035+0000", 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.422035+0000", 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.422037+0000", 2024-03-24T12:13:17.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.422044+0000", 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3930000000000002e-06 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.422054+0000", 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8660000000000001e-06 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.422063+0000", 2024-03-24T12:13:17.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.302e-06 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.422880+0000", 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081648799999999998 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:10.422899+0000", 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9606000000000002e-05 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:13:17.052 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 3 2024-03-24T12:13:17.053 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:13:17.114 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:17.152 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:17.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:17.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:17.297 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:17.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:17.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:17.440 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:17.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:17.541 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:17.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-mon[30223]: pgmap v995: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 791 op/s; 37915/225423 objects misplaced (16.819%); 40 MiB/s, 643 objects/s recovering 2024-03-24T12:13:17.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-mon[30223]: 11.14 scrub starts 2024-03-24T12:13:17.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-mon[30223]: 11.14 scrub ok 2024-03-24T12:13:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-mon[30223]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T12:13:17.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:13:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-mon[42243]: pgmap v995: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 791 op/s; 37915/225423 objects misplaced (16.819%); 40 MiB/s, 643 objects/s recovering 2024-03-24T12:13:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-mon[42243]: 11.14 scrub starts 2024-03-24T12:13:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-mon[42243]: 11.14 scrub ok 2024-03-24T12:13:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:17 smithi064 ceph-mon[42243]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T12:13:17.579 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:17.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:17 smithi193 ceph-mon[33593]: pgmap v995: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 791 op/s; 37915/225423 objects misplaced (16.819%); 40 MiB/s, 643 objects/s recovering 2024-03-24T12:13:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:17 smithi193 ceph-mon[33593]: 11.14 scrub starts 2024-03-24T12:13:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:17 smithi193 ceph-mon[33593]: 11.14 scrub ok 2024-03-24T12:13:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:17 smithi193 ceph-mon[33593]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T12:13:17.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:17.715 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:17.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:17.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:17.857 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:17.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:17.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:17.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:18.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:18.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:18.145 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:18.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:18.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:18.291 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:18.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:18.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:13:17 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:13:18.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:18.423 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:18.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:18.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:18.567 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:18.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:18.668 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:18.697 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:18.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:18.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:18.835 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:18.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:18.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:18.973 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:18.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:19.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:19.112 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:19.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:19.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:19.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:19.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:19.393 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:19.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:19.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:19.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:19 smithi064 ceph-mon[30223]: pgmap v997: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 517 op/s; 32875/214074 objects misplaced (15.357%); 39 MiB/s, 624 objects/s recovering 2024-03-24T12:13:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:19.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:19 smithi064 ceph-mon[42243]: pgmap v997: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 517 op/s; 32875/214074 objects misplaced (15.357%); 39 MiB/s, 624 objects/s recovering 2024-03-24T12:13:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:19 smithi193 ceph-mon[33593]: pgmap v997: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 517 op/s; 32875/214074 objects misplaced (15.357%); 39 MiB/s, 624 objects/s recovering 2024-03-24T12:13:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:19.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:19.682 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:19.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:19.783 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:19.807 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:19.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:19.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:19.950 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:19.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:19.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:20.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:20.077 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:20.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:20.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:20.218 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:20.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:20.320 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:20.355 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:20.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:20.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:20.494 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:20.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:20 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T12:13:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:20 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T12:13:20.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:20 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T12:13:20.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:20 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T12:13:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:20 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T12:13:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:20 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T12:13:20.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:20.632 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:20.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:20.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:20.762 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:20.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:20.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:20.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:20.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:21.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:21.052 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:21.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:21.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:21.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:21.206 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:21.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:21.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:21.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:21.390 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.6 primary-affinity to 0.00748275 (84902) 2024-03-24T12:13:21.449 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:21.478 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:21.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:21.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:21 smithi064 ceph-mon[30223]: pgmap v998: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30032/211497 objects misplaced (14.200%); 23 MiB/s, 366 objects/s recovering 2024-03-24T12:13:21.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/54389363' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.007482750665621318}]: dispatch 2024-03-24T12:13:21.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:21 smithi064 ceph-mon[42243]: pgmap v998: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30032/211497 objects misplaced (14.200%); 23 MiB/s, 366 objects/s recovering 2024-03-24T12:13:21.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/54389363' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.007482750665621318}]: dispatch 2024-03-24T12:13:21.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:21 smithi193 ceph-mon[33593]: pgmap v998: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30032/211497 objects misplaced (14.200%); 23 MiB/s, 366 objects/s recovering 2024-03-24T12:13:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/54389363' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.007482750665621318}]: dispatch 2024-03-24T12:13:21.610 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:21.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:21.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:21.742 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:21.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:21.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:21.871 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:21.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:21.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:22.004 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:22.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:22.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:22.143 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:22.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:22.244 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:22.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:22.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:22.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:22.414 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:22.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:22.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:22.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:22.542 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:22.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:13:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:13:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/54389363' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.007482750665621318}]': finished 2024-03-24T12:13:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[30223]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T12:13:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[30223]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T12:13:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:13:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:13:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/54389363' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.007482750665621318}]': finished 2024-03-24T12:13:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[42243]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T12:13:22.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:22 smithi064 ceph-mon[42243]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T12:13:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:22 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:13:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:22 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:13:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:22 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/54389363' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.007482750665621318}]': finished 2024-03-24T12:13:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:22 smithi193 ceph-mon[33593]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T12:13:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:22 smithi193 ceph-mon[33593]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T12:13:22.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:22.678 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:22.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:22.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:22.809 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:22.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:22.910 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:22.934 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:22.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:23.036 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:23.067 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:23.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:23.168 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:23.204 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:23.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:23.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:23.337 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:23.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:23.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:23.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:23.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:23.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:23 smithi064 ceph-mon[30223]: pgmap v1000: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 30032/211497 objects misplaced (14.200%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:13:23.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:23 smithi064 ceph-mon[30223]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T12:13:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:23 smithi064 ceph-mon[42243]: pgmap v1000: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 30032/211497 objects misplaced (14.200%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:13:23.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:23 smithi064 ceph-mon[42243]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T12:13:23.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:23 smithi193 ceph-mon[33593]: pgmap v1000: 168 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 159 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 30032/211497 objects misplaced (14.200%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:13:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:23 smithi193 ceph-mon[33593]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T12:13:23.615 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:23.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:23.717 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:23.751 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:23.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:23.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:23.888 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:23.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:23.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:24.027 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:24.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:24.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:24.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:24.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:24.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:24.304 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:24.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:24.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:24.440 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:24.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:24.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:24.578 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:24.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:24.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:24.691 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:24 smithi064 ceph-mon[30223]: pgmap v1003: 168 pgs: 16 peering, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 144 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4.7 KiB/s rd, 43 KiB/s wr, 769 op/s; 25553/200328 objects misplaced (12.756%); 35 MiB/s, 555 objects/s recovering 2024-03-24T12:13:24.691 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:24 smithi064 ceph-mon[30223]: Health check failed: Reduced data availability: 8 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-24T12:13:24.691 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:24 smithi064 ceph-mon[30223]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T12:13:24.691 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:24 smithi064 ceph-mon[42243]: pgmap v1003: 168 pgs: 16 peering, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 144 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4.7 KiB/s rd, 43 KiB/s wr, 769 op/s; 25553/200328 objects misplaced (12.756%); 35 MiB/s, 555 objects/s recovering 2024-03-24T12:13:24.692 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:24 smithi064 ceph-mon[42243]: Health check failed: Reduced data availability: 8 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-24T12:13:24.692 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:24 smithi064 ceph-mon[42243]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T12:13:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:24.707 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:24.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:24.808 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:24.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:24 smithi193 ceph-mon[33593]: pgmap v1003: 168 pgs: 16 peering, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 144 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4.7 KiB/s rd, 43 KiB/s wr, 769 op/s; 25553/200328 objects misplaced (12.756%); 35 MiB/s, 555 objects/s recovering 2024-03-24T12:13:24.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:24 smithi193 ceph-mon[33593]: Health check failed: Reduced data availability: 8 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-24T12:13:24.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:24 smithi193 ceph-mon[33593]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T12:13:24.838 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:24.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:24.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:24.975 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:24.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:25.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:25.101 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:25.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:25.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:25.240 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:25.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:25.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:25.369 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:25.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:25.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:25.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:25.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:25.640 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:25.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:25.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:25.742 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:25.766 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:25.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:25.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:25.902 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:25.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:26.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:26.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:26.033 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:26.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:26.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:26.162 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:26.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:26.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:26.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:26.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:26.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:26.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:26.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:26.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:26 smithi064 ceph-mon[30223]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T12:13:26.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:26 smithi064 ceph-mon[42243]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T12:13:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:26.580 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:26.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:26 smithi193 ceph-mon[33593]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T12:13:26.682 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:26.707 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:26.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:26.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:26.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:26.835 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:26.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:26.937 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:26.973 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:26.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:27.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:27.115 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:27.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:27.217 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:27.255 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:27.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:27.344 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:13:27.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:27.395 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:27.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:27.496 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:27.529 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:27.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:27 smithi193 ceph-mon[33593]: pgmap v1006: 168 pgs: 16 peering, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 144 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 13 KiB/s rd, 119 KiB/s wr, 1.09k op/s; 23550/198606 objects misplaced (11.858%); 41 MiB/s, 653 objects/s recovering 2024-03-24T12:13:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:27 smithi193 ceph-mon[33593]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T12:13:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:27 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T12:13:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:27 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T12:13:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:27 smithi193 ceph-mon[33593]: 11.18 scrub starts 2024-03-24T12:13:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:27 smithi193 ceph-mon[33593]: 11.8 scrub starts 2024-03-24T12:13:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:27 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/647346811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:13:27.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:27.655 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:27.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:27.709 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:13:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:27.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[30223]: pgmap v1006: 168 pgs: 16 peering, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 144 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 13 KiB/s rd, 119 KiB/s wr, 1.09k op/s; 23550/198606 objects misplaced (11.858%); 41 MiB/s, 653 objects/s recovering 2024-03-24T12:13:27.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[30223]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T12:13:27.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T12:13:27.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T12:13:27.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[30223]: 11.18 scrub starts 2024-03-24T12:13:27.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[30223]: 11.8 scrub starts 2024-03-24T12:13:27.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/647346811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:13:27.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[42243]: pgmap v1006: 168 pgs: 16 peering, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 144 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 13 KiB/s rd, 119 KiB/s wr, 1.09k op/s; 23550/198606 objects misplaced (11.858%); 41 MiB/s, 653 objects/s recovering 2024-03-24T12:13:27.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[42243]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T12:13:27.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T12:13:27.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T12:13:27.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[42243]: 11.18 scrub starts 2024-03-24T12:13:27.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[42243]: 11.8 scrub starts 2024-03-24T12:13:27.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:27 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/647346811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:13:27.756 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:27.782 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:27.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:27.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:27.916 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:27.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:28.017 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:28.043 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:28.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:28.145 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:28.168 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:28.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:28.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:28.295 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:28.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:28.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:13:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:13:28.325 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:13:28.325 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:13:28.325 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: 11.18 scrub ok 2024-03-24T12:13:28.325 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: 11.8 scrub ok 2024-03-24T12:13:28.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:28.446 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:28.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:28.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:13:28.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/647346811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:13:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T12:13:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: pgmap v1009: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 161 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 6.2 KiB/s rd, 3 op/s; 20343/190896 objects misplaced (10.657%) 2024-03-24T12:13:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:13:28.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:28 smithi193 ceph-mon[33593]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T12:13:28.587 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:28.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:28.689 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:28.734 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:28.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:13:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:13:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: 11.18 scrub ok 2024-03-24T12:13:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: 11.8 scrub ok 2024-03-24T12:13:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:13:28.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/647346811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:13:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T12:13:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: pgmap v1009: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 161 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 6.2 KiB/s rd, 3 op/s; 20343/190896 objects misplaced (10.657%) 2024-03-24T12:13:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:13:28.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[30223]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T12:13:28.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: 11.18 scrub ok 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: 11.8 scrub ok 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/647346811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: pgmap v1009: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 161 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 6.2 KiB/s rd, 3 op/s; 20343/190896 objects misplaced (10.657%) 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:28 smithi064 ceph-mon[42243]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T12:13:28.836 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:28.878 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:28.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:28.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:28.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:29.015 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:29.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:29.117 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:29.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:29.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:29.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:29.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:29.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:29.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:29.425 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:29.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:29.527 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:29.562 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:29.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:13:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:13:29.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:13:29.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:13:29.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-03-24T12:13:29.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:13:29.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:13:29.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:29.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:29 smithi193 ceph-mon[33593]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T12:13:29.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:29.704 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:29.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:13:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:13:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:13:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:13:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[30223]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:13:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:13:29.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:29.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:29 smithi064 ceph-mon[42243]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T12:13:29.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:29.840 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:29.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:29.942 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:29.984 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:29.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:30.086 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:30.124 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:30.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:30.225 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:30.258 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:30.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:30.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:30.398 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:30.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:30.500 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:30.534 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:30.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:30.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:30.672 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:30.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:30.774 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:30.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:30 smithi064 ceph-mon[30223]: pgmap v1012: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19416/189069 objects misplaced (10.269%) 2024-03-24T12:13:30.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:30 smithi064 ceph-mon[42243]: pgmap v1012: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19416/189069 objects misplaced (10.269%) 2024-03-24T12:13:30.810 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:30.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:30 smithi193 ceph-mon[33593]: pgmap v1012: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19416/189069 objects misplaced (10.269%) 2024-03-24T12:13:30.912 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:30.943 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:30.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:31.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:31.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:31.183 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:31.216 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:31.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:31.255 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:13:31.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:31.371 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:31.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:31.472 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:31.505 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:31.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:31.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:31.651 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:31.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:31.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:31.791 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:31.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:31.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:31.937 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:31.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:32.040 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:32.069 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:32.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:32.170 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:32.207 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:32.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:32.309 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:32.347 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:32.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:32.450 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:32.486 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:32.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:32.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:32 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:13:32.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:32 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:13:32.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:32 smithi064 ceph-mon[30223]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T12:13:32.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:32 smithi064 ceph-mon[30223]: pgmap v1014: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19416/189069 objects misplaced (10.269%) 2024-03-24T12:13:32.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:32 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:13:32.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:32 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:13:32.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:32 smithi064 ceph-mon[42243]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T12:13:32.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:32 smithi064 ceph-mon[42243]: pgmap v1014: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19416/189069 objects misplaced (10.269%) 2024-03-24T12:13:32.588 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:32.622 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:32.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:32.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:32.756 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:32.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:32 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:13:32.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:32 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:13:32.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:32 smithi193 ceph-mon[33593]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T12:13:32.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:32 smithi193 ceph-mon[33593]: pgmap v1014: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19416/189069 objects misplaced (10.269%) 2024-03-24T12:13:32.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:32.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:32.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:33.025 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:33.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:33.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:33.164 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:33.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:33.198 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:33.266 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:33.302 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:33.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:33.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:33.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:13:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:33.436 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:33.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:33.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:33.592 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:33.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:33.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:33.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:33.762 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:33.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:33.863 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:33.898 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:33.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:33.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:34.041 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:34.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:34.087 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:34.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:34.185 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:34.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:34.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:34.332 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:34.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:34.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:34.481 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:34.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:34.583 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:34.612 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:34.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:34.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:34.750 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:34.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:34.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:34 smithi064 ceph-mon[30223]: pgmap v1015: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18525/174798 objects misplaced (10.598%); 1.9 MiB/s, 30 objects/s recovering 2024-03-24T12:13:34.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:34 smithi064 ceph-mon[42243]: pgmap v1015: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18525/174798 objects misplaced (10.598%); 1.9 MiB/s, 30 objects/s recovering 2024-03-24T12:13:34.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:34.898 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:34.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:34.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:35.036 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:35.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:35.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:34 smithi193 ceph-mon[33593]: pgmap v1015: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18525/174798 objects misplaced (10.598%); 1.9 MiB/s, 30 objects/s recovering 2024-03-24T12:13:35.138 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:35.173 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:35.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:35.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:35.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:35.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:35.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:35.452 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:35.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:35.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:35.567 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:13:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:35.598 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:35.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:35.616 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:13:35.659 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:13:35.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:35.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/411931027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:35.703 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:13:35.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/411931027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:35.741 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:35.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:35.744 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:13:35.785 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:13:35.827 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:13:35.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:35.875 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:13:35.877 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:35.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:35.879 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:13:35.879 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:13:35.879 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:13:35.879 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:13:35.879 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:13:35.879 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:13:35.879 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:13:35.879 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:13:35.979 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:36.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:36.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/411931027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:36.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:36.160 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:36.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:36.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:36.261 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:36.291 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:36.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:36.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:36.436 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:36.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:36.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:36.582 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:36.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:36.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:36.725 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:36.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:36.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:36.865 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:36.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:36.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:37.002 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:37.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:37.037 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:36 smithi064 ceph-mon[30223]: pgmap v1016: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 55 op/s; 17426/170814 objects misplaced (10.202%); 1.6 MiB/s, 25 objects/s recovering 2024-03-24T12:13:37.038 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:36 smithi064 ceph-mon[42243]: pgmap v1016: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 55 op/s; 17426/170814 objects misplaced (10.202%); 1.6 MiB/s, 25 objects/s recovering 2024-03-24T12:13:37.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:36 smithi193 ceph-mon[33593]: pgmap v1016: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 55 op/s; 17426/170814 objects misplaced (10.202%); 1.6 MiB/s, 25 objects/s recovering 2024-03-24T12:13:37.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:37.186 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:37.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:37.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:37.322 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:37.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:37.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:37.433 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:37.475 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:37.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:37.577 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:37.608 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:37.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:37.669 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:13:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:37.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:37.758 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:37.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:37.790 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:13:37.790 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:13:37.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:37.893 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:37.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:37.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:37.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:38.007 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:37 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:13:38.007 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:37 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:13:38.008 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:37 smithi193 ceph-mon[33593]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T12:13:38.008 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:37 smithi193 ceph-mon[33593]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T12:13:38.042 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:38.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:13:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:13:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-mon[30223]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T12:13:38.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-mon[30223]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T12:13:38.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:13:38.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:13:38.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-mon[42243]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T12:13:38.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:37 smithi064 ceph-mon[42243]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T12:13:38.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:38.183 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:38.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:38.220 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:13:38.220 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:13:38.220 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-03-24T12:13:38.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:13:38.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:38.330 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:38.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:38.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:13:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:13:38.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:38.469 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:38.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:38.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:38.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:38.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:38.602 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:38.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:38.703 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:38.746 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:38.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:38.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:38 smithi064 ceph-mon[30223]: pgmap v1019: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 821 B/s rd, 0 B/s wr, 850 op/s; 15741/154287 objects misplaced (10.202%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T12:13:38.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:38.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:38 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:38.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:38 smithi064 ceph-mon[42243]: pgmap v1019: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 821 B/s rd, 0 B/s wr, 850 op/s; 15741/154287 objects misplaced (10.202%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T12:13:38.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:38.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:38 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:38.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:38.884 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:38.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:38.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:39.023 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:39.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:38 smithi193 ceph-mon[33593]: pgmap v1019: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 821 B/s rd, 0 B/s wr, 850 op/s; 15741/154287 objects misplaced (10.202%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T12:13:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:39.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:38 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:13:39.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:13:39.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:13:39.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:39.223 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:39.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:39.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:39.355 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:39.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:39.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:39.496 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:39.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:39.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:39.661 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:39.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:39.762 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:39.799 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:39.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:39.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:39.939 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:39.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:40.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:40.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:39 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:13:40.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:39 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:13:40.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:39 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:13:40.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:39 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:13:40.084 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:40.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:40.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:39 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:13:40.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:39 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:13:40.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:40.261 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:40.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:40.307 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:40.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:40.408 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:40.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:40.510 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:40.552 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:40.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:40.654 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:40.701 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:40.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:40.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:40 smithi064 ceph-mon[30223]: pgmap v1020: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 811 op/s; 14767/150714 objects misplaced (9.798%); 2.5 MiB/s, 39 objects/s recovering 2024-03-24T12:13:40.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:40 smithi064 ceph-mon[30223]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T12:13:40.708 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:40 smithi064 ceph-mon[42243]: pgmap v1020: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 811 op/s; 14767/150714 objects misplaced (9.798%); 2.5 MiB/s, 39 objects/s recovering 2024-03-24T12:13:40.708 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:40 smithi064 ceph-mon[42243]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T12:13:40.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:40.846 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:40.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:40.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:40.982 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:40.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:41.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:41.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:40 smithi193 ceph-mon[33593]: pgmap v1020: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 811 op/s; 14767/150714 objects misplaced (9.798%); 2.5 MiB/s, 39 objects/s recovering 2024-03-24T12:13:41.085 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:40 smithi193 ceph-mon[33593]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T12:13:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:41.126 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:41.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:41.227 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:41.265 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:41.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:41.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:41.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:41.497 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:41.531 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:41.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:41.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:41.672 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:41.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:41.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:41.811 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:41.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:41.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:41.939 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:41.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:42.041 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:42.077 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:42.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:42.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:42.210 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:42.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:42.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:42.337 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:42.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:42.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:42.475 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:42.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:42.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:42.617 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:42.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:42.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:42.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:42.858 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:42.898 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:42.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:43.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:43.041 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:43.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:43.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:43.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:43.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:43.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:43 smithi064 ceph-mon[30223]: pgmap v1022: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s; 14767/150714 objects misplaced (9.798%); 3.3 MiB/s, 52 objects/s recovering 2024-03-24T12:13:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:43 smithi064 ceph-mon[30223]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T12:13:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:43.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:43.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:43 smithi064 ceph-mon[42243]: pgmap v1022: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s; 14767/150714 objects misplaced (9.798%); 3.3 MiB/s, 52 objects/s recovering 2024-03-24T12:13:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:43 smithi064 ceph-mon[42243]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T12:13:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:43.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:43.321 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:43.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:43 smithi193 ceph-mon[33593]: pgmap v1022: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s; 14767/150714 objects misplaced (9.798%); 3.3 MiB/s, 52 objects/s recovering 2024-03-24T12:13:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:43 smithi193 ceph-mon[33593]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T12:13:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:13:43.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:43.423 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:43.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:43.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:43.596 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:43.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:43.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:43.735 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:43.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:43.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:43.876 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:43.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:43.977 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:44.012 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:44.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:44.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:44.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:44.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:44.140 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:44.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:44.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:44.278 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:44.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:44.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:44.421 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:44.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:44.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:44.558 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:44.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:44.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:44.701 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:44.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:44.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:44.835 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:44.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:44.937 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:44.976 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:44.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:45.027 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:45 smithi064 ceph-mon[30223]: pgmap v1024: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.15k op/s; 12871/134928 objects misplaced (9.539%); 7.9 MiB/s, 126 objects/s recovering 2024-03-24T12:13:45.027 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:45 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:13:45.027 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:45 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:13:45.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:45.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:45.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:45.277 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:45.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:45.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:45 smithi064 ceph-mon[42243]: pgmap v1024: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.15k op/s; 12871/134928 objects misplaced (9.539%); 7.9 MiB/s, 126 objects/s recovering 2024-03-24T12:13:45.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:45 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:13:45.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:45 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:13:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:45 smithi193 ceph-mon[33593]: pgmap v1024: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.15k op/s; 12871/134928 objects misplaced (9.539%); 7.9 MiB/s, 126 objects/s recovering 2024-03-24T12:13:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:45 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:13:45.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:45 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:13:45.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:45.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:45.421 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:45.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:45.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:45.564 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:45.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:45.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:45.709 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:45.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:45.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:45.843 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:45.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:45.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:45.977 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:45.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:46.070 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-03-24T12:13:46.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:46.123 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:46.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:46.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:46.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:46.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:46 smithi064 ceph-mon[42243]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T12:13:46.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:46 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/179849356' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:46.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:46 smithi064 ceph-mon[30223]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T12:13:46.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:46 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/179849356' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:46 smithi193 ceph-mon[33593]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T12:13:46.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:46 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/179849356' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:13:46.365 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:46.398 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:46.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:46.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:46.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:46.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:46.686 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:46.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:46.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:46.825 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:46.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:46.927 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:46.959 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:46.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:47.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:47.111 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:47.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:47.213 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:47.251 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:47.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[30223]: pgmap v1026: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 11945/131505 objects misplaced (9.083%); 5.3 MiB/s, 85 objects/s recovering 2024-03-24T12:13:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[30223]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:13:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/179849356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:13:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[30223]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T12:13:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[30223]: from='client.16209 v1:172.21.15.64:0/195172952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:13:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[42243]: pgmap v1026: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 11945/131505 objects misplaced (9.083%); 5.3 MiB/s, 85 objects/s recovering 2024-03-24T12:13:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[42243]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:13:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/179849356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:13:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[42243]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T12:13:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-mon[42243]: from='client.16209 v1:172.21.15.64:0/195172952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:13:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:47 smithi193 ceph-mon[33593]: pgmap v1026: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 11945/131505 objects misplaced (9.083%); 5.3 MiB/s, 85 objects/s recovering 2024-03-24T12:13:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:47 smithi193 ceph-mon[33593]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:13:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:47 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/179849356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:13:47.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:47 smithi193 ceph-mon[33593]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T12:13:47.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:47 smithi193 ceph-mon[33593]: from='client.16209 v1:172.21.15.64:0/195172952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:13:47.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:47.391 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:47.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:47.493 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:47.531 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:47.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:47.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:47.660 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:13:47.661 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:13:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:13:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:47.671 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:47.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:47.673 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:13:47.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:47.803 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:47.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:47.905 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:47.972 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:47.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:48.073 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:48.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:48.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:48.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:48.250 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:48.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:48.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:13:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:13:48.352 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:48.388 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:48.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:48.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:48.530 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:48.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:48.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:48.665 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:48.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:48.766 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:48.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:48.901 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:48.941 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:48.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:49.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:49.084 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:49.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:49.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:49.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:49.224 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:49.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:49.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:49 smithi193 ceph-mon[33593]: pgmap v1028: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.91k op/s; 10298/116355 objects misplaced (8.851%); 10 MiB/s, 167 objects/s recovering 2024-03-24T12:13:49.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:49.361 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:49.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:49.462 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:49.487 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:49.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:49.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:49 smithi064 ceph-mon[30223]: pgmap v1028: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.91k op/s; 10298/116355 objects misplaced (8.851%); 10 MiB/s, 167 objects/s recovering 2024-03-24T12:13:49.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:49.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:49 smithi064 ceph-mon[42243]: pgmap v1028: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.91k op/s; 10298/116355 objects misplaced (8.851%); 10 MiB/s, 167 objects/s recovering 2024-03-24T12:13:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:49.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:49.614 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:49.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:49.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:49.739 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:49.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:49.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:49.866 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:49.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:49.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:49.991 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:49.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:50.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:50.120 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:50.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:50.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:50.260 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:50.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:50.362 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:50.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:50.399 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:50.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:50.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:50.517 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:13:50.517 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:13:50.517 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:50 smithi064 ceph-mon[30223]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T12:13:50.518 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:50 smithi064 ceph-mon[42243]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T12:13:50.535 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:50.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:50 smithi193 ceph-mon[33593]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T12:13:50.637 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:50.668 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:50.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:50.770 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:50.811 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:50.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:50.913 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:50.946 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:50.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:51.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:51.088 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:51.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:51.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:51.224 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:51.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:51.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:51.362 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:51.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:51.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:51.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:51.500 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:51.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:51 smithi064 ceph-mon[30223]: pgmap v1030: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 9392/112890 objects misplaced (8.320%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:13:51.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:51 smithi064 ceph-mon[30223]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T12:13:51.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:51 smithi064 ceph-mon[42243]: pgmap v1030: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 9392/112890 objects misplaced (8.320%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:13:51.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:51 smithi064 ceph-mon[42243]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T12:13:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:51 smithi193 ceph-mon[33593]: pgmap v1030: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 9392/112890 objects misplaced (8.320%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:13:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:51 smithi193 ceph-mon[33593]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T12:13:51.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:51.645 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:51.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:51.746 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:51.781 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:51.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:51.883 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:51.919 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:51.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:52.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:52.066 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:52.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:52.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:52.168 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:52.209 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:52.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:52.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:52.349 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:52.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:52.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:52.494 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:52.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:52.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:52.632 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:52.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:52.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:52.765 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:52.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:52.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:52.909 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:52.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:53.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:53.048 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:53.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:53.150 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:53.184 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:53.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:53.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:53 smithi064 ceph-mon[30223]: pgmap v1032: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 9392/112890 objects misplaced (8.320%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:13:53.240 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:53 smithi064 ceph-mon[42243]: pgmap v1032: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 9392/112890 objects misplaced (8.320%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:13:53.286 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:53.319 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:53.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:53.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:53 smithi193 ceph-mon[33593]: pgmap v1032: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 9392/112890 objects misplaced (8.320%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:13:53.335 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:13:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a[66939]: logger=cleanup t=2024-03-24T12:13:52.889390547Z level=info msg="Completed cleanup jobs" duration=3.491064ms 2024-03-24T12:13:53.356 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:13:53.404 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:13:53.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:53.447 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:13:53.455 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:53.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:53.488 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:13:53.530 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:13:53.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:53.571 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:13:53.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:53.599 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:53.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:53.621 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:13:53.662 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:13:53.668 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:13:53.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:13:53.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:13:53.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:13:53.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:13:53.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:13:53.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:13:53.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:13:53.701 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:53.733 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:53.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:53.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:13:53.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:53.871 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:53.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:53.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:54.015 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:54.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:54.117 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:54.153 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:54.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:54.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:54.298 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:54.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/249593222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:54.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:54.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:54.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:54.550 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/249593222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:54.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/249593222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:13:54.575 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:54.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:54.602 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1027,"stamp":"2024-03-24T12:13:46.054494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2860277683,"num_objects":43835,"num_object_clones":0,"num_object_copies":131505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11945,"num_objects_unfound":0,"num_objects_dirty":43835,"num_whiteouts":0,"num_read":2081,"num_read_kb":1852,"num_write":152076,"num_write_kb":5278954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10157,"num_bytes_recovered":662657597,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150060,"ondisk_log_size":150060,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":520,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15452564,"kb_used_data":9176632,"kb_used_omap":18,"kb_used_meta":6275821,"kb_avail":734508652,"statfs":{"total":767960285184,"available":752136859648,"internally_reserved":0,"allocated":9396871168,"data_stored":9392823834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":6426440923},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-419627008,"num_objects":-6403,"num_object_clones":0,"num_object_copies":-19209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2822,"num_objects_unfound":0,"num_objects_dirty":-6403,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.365287"},"pg_stats":[{"pgid":"11.7","version":"478'12531","reported_seq":12707,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:44.328683+0000","last_change":"2024-03-24T12:13:02.388718+0000","last_active":"2024-03-24T12:13:44.328683+0000","last_peered":"2024-03-24T12:13:44.328683+0000","last_clean":"2024-03-24T12:13:44.328683+0000","last_became_active":"2024-03-24T12:13:01.126268+0000","last_became_peered":"2024-03-24T12:13:01.126268+0000","last_unstale":"2024-03-24T12:13:44.328683+0000","last_undegraded":"2024-03-24T12:13:44.328683+0000","last_fullsized":"2024-03-24T12:13:44.328683+0000","mapping_epoch":417,"log_start":"441'6300","ondisk_log_start":"441'6300","created":417,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'11208","last_scrub_stamp":"2024-03-24T12:13:02.388585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.388585+0000","objects_scrubbed":2642,"log_size":6231,"log_dups_size":0,"ondisk_log_size":6231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:31.681109+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"176'9","reported_seq":1186,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.690089+0000","last_change":"2024-03-24T12:12:23.293867+0000","last_active":"2024-03-24T12:13:39.690089+0000","last_peered":"2024-03-24T12:13:39.690089+0000","last_clean":"2024-03-24T12:13:39.690089+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:13:39.690089+0000","last_undegraded":"2024-03-24T12:13:39.690089+0000","last_fullsized":"2024-03-24T12:13:39.690089+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:12:23.293737+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.293737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:48.062463+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"450'39","reported_seq":844,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031098+0000","last_change":"2024-03-24T12:12:17.532582+0000","last_active":"2024-03-24T12:13:42.031098+0000","last_peered":"2024-03-24T12:13:42.031098+0000","last_clean":"2024-03-24T12:13:42.031098+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:13:42.031098+0000","last_undegraded":"2024-03-24T12:13:42.031098+0000","last_fullsized":"2024-03-24T12:13:42.031098+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-24T12:12:17.532295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:17.532295+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":384,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.696267+0000","last_change":"2024-03-24T12:12:21.258445+0000","last_active":"2024-03-24T12:13:39.696267+0000","last_peered":"2024-03-24T12:13:39.696267+0000","last_clean":"2024-03-24T12:13:39.696267+0000","last_became_active":"2024-03-24T12:05:44.731883+0000","last_became_peered":"2024-03-24T12:05:44.731883+0000","last_unstale":"2024-03-24T12:13:39.696267+0000","last_undegraded":"2024-03-24T12:13:39.696267+0000","last_fullsized":"2024-03-24T12:13:39.696267+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:21.258323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:21.258323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":810,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031654+0000","last_change":"2024-03-24T12:10:19.713100+0000","last_active":"2024-03-24T12:13:42.031654+0000","last_peered":"2024-03-24T12:13:42.031654+0000","last_clean":"2024-03-24T12:13:42.031654+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:13:42.031654+0000","last_undegraded":"2024-03-24T12:13:42.031654+0000","last_fullsized":"2024-03-24T12:13:42.031654+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:19.713016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:45.874555+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.713016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"477'12693","reported_seq":12955,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.725603+0000","last_change":"2024-03-24T12:13:01.923611+0000","last_active":"2024-03-24T12:13:41.725603+0000","last_peered":"2024-03-24T12:13:41.725603+0000","last_clean":"2024-03-24T12:13:41.725603+0000","last_became_active":"2024-03-24T12:13:01.126904+0000","last_became_peered":"2024-03-24T12:13:01.126904+0000","last_unstale":"2024-03-24T12:13:41.725603+0000","last_undegraded":"2024-03-24T12:13:41.725603+0000","last_fullsized":"2024-03-24T12:13:41.725603+0000","mapping_epoch":417,"log_start":"443'7000","ondisk_log_start":"443'7000","created":417,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'11437","last_scrub_stamp":"2024-03-24T12:13:01.923337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:01.923337+0000","objects_scrubbed":2601,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:17.784642+0000","scrub_duration":667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87490560,"num_objects":1335,"num_object_clones":0,"num_object_copies":4005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1266,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"450'21","reported_seq":934,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035488+0000","last_change":"2024-03-24T12:11:18.368554+0000","last_active":"2024-03-24T12:13:42.035488+0000","last_peered":"2024-03-24T12:13:42.035488+0000","last_clean":"2024-03-24T12:13:42.035488+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:13:42.035488+0000","last_undegraded":"2024-03-24T12:13:42.035488+0000","last_fullsized":"2024-03-24T12:13:42.035488+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:11:18.368416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.368416+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":945,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031269+0000","last_change":"2024-03-24T12:12:59.147957+0000","last_active":"2024-03-24T12:13:42.031269+0000","last_peered":"2024-03-24T12:13:42.031269+0000","last_clean":"2024-03-24T12:13:42.031269+0000","last_became_active":"2024-03-24T11:52:15.140150+0000","last_became_peered":"2024-03-24T11:52:15.140150+0000","last_unstale":"2024-03-24T12:13:42.031269+0000","last_undegraded":"2024-03-24T12:13:42.031269+0000","last_fullsized":"2024-03-24T12:13:42.031269+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:59.147768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:12:59.147768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:14.967977+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":382,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697732+0000","last_change":"2024-03-24T12:11:37.592094+0000","last_active":"2024-03-24T12:13:39.697732+0000","last_peered":"2024-03-24T12:13:39.697732+0000","last_clean":"2024-03-24T12:13:39.697732+0000","last_became_active":"2024-03-24T12:05:44.728164+0000","last_became_peered":"2024-03-24T12:05:44.728164+0000","last_unstale":"2024-03-24T12:13:39.697732+0000","last_undegraded":"2024-03-24T12:13:39.697732+0000","last_fullsized":"2024-03-24T12:13:39.697732+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:37.591971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.591971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075707+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":898,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035777+0000","last_change":"2024-03-24T12:12:55.607092+0000","last_active":"2024-03-24T12:13:42.035777+0000","last_peered":"2024-03-24T12:13:42.035777+0000","last_clean":"2024-03-24T12:13:42.035777+0000","last_became_active":"2024-03-24T12:05:44.736608+0000","last_became_peered":"2024-03-24T12:05:44.736608+0000","last_unstale":"2024-03-24T12:13:42.035777+0000","last_undegraded":"2024-03-24T12:13:42.035777+0000","last_fullsized":"2024-03-24T12:13:42.035777+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:55.606984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:12:55.606984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:56.962223+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"477'12463","reported_seq":12720,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.797436+0000","last_change":"2024-03-24T12:13:02.601279+0000","last_active":"2024-03-24T12:13:41.797436+0000","last_peered":"2024-03-24T12:13:41.797436+0000","last_clean":"2024-03-24T12:13:41.797436+0000","last_became_active":"2024-03-24T12:13:01.150371+0000","last_became_peered":"2024-03-24T12:13:01.150371+0000","last_unstale":"2024-03-24T12:13:41.797436+0000","last_undegraded":"2024-03-24T12:13:41.797436+0000","last_fullsized":"2024-03-24T12:13:41.797436+0000","mapping_epoch":417,"log_start":"442'6800","ondisk_log_start":"442'6800","created":417,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'11269","last_scrub_stamp":"2024-03-24T12:13:02.601097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.601097+0000","objects_scrubbed":2587,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:19.538034+0000","scrub_duration":340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91291648,"num_objects":1393,"num_object_clones":0,"num_object_copies":4179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1194,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"450'60","reported_seq":971,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.692741+0000","last_change":"2024-03-24T12:13:22.052036+0000","last_active":"2024-03-24T12:13:39.692741+0000","last_peered":"2024-03-24T12:13:39.692741+0000","last_clean":"2024-03-24T12:13:39.692741+0000","last_became_active":"2024-03-24T12:13:22.026501+0000","last_became_peered":"2024-03-24T12:13:22.026501+0000","last_unstale":"2024-03-24T12:13:39.692741+0000","last_undegraded":"2024-03-24T12:13:39.692741+0000","last_fullsized":"2024-03-24T12:13:39.692741+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'44","last_scrub_stamp":"2024-03-24T12:11:20.906884+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.906884+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":280,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031861+0000","last_change":"2024-03-24T12:11:27.439531+0000","last_active":"2024-03-24T12:13:42.031861+0000","last_peered":"2024-03-24T12:13:42.031861+0000","last_clean":"2024-03-24T12:13:42.031861+0000","last_became_active":"2024-03-24T12:08:36.164337+0000","last_became_peered":"2024-03-24T12:08:36.164337+0000","last_unstale":"2024-03-24T12:13:42.031861+0000","last_undegraded":"2024-03-24T12:13:42.031861+0000","last_fullsized":"2024-03-24T12:13:42.031861+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:27.439409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:11:27.439409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":393,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.034705+0000","last_change":"2024-03-24T12:11:22.405559+0000","last_active":"2024-03-24T12:13:42.034705+0000","last_peered":"2024-03-24T12:13:42.034705+0000","last_clean":"2024-03-24T12:13:42.034705+0000","last_became_active":"2024-03-24T12:08:36.155105+0000","last_became_peered":"2024-03-24T12:08:36.155105+0000","last_unstale":"2024-03-24T12:13:42.034705+0000","last_undegraded":"2024-03-24T12:13:42.034705+0000","last_fullsized":"2024-03-24T12:13:42.034705+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:22.404955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:22.404955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":913,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.689275+0000","last_change":"2024-03-24T12:11:32.175174+0000","last_active":"2024-03-24T12:13:39.689275+0000","last_peered":"2024-03-24T12:13:39.689275+0000","last_clean":"2024-03-24T12:13:39.689275+0000","last_became_active":"2024-03-24T12:05:44.737032+0000","last_became_peered":"2024-03-24T12:05:44.737032+0000","last_unstale":"2024-03-24T12:13:39.689275+0000","last_undegraded":"2024-03-24T12:13:39.689275+0000","last_fullsized":"2024-03-24T12:13:39.689275+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.175048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.175048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"477'12509","reported_seq":12681,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.770548+0000","last_change":"2024-03-24T12:13:02.521495+0000","last_active":"2024-03-24T12:13:41.770548+0000","last_peered":"2024-03-24T12:13:41.770548+0000","last_clean":"2024-03-24T12:13:41.770548+0000","last_became_active":"2024-03-24T12:13:01.136344+0000","last_became_peered":"2024-03-24T12:13:01.136344+0000","last_unstale":"2024-03-24T12:13:41.770548+0000","last_undegraded":"2024-03-24T12:13:41.770548+0000","last_fullsized":"2024-03-24T12:13:41.770548+0000","mapping_epoch":417,"log_start":"442'6800","ondisk_log_start":"442'6800","created":417,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'11266","last_scrub_stamp":"2024-03-24T12:13:02.521353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.521353+0000","objects_scrubbed":2573,"log_size":5709,"log_dups_size":0,"ondisk_log_size":5709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:31.424558+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87162880,"num_objects":1330,"num_object_clones":0,"num_object_copies":3990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1243,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"450'53","reported_seq":999,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.699091+0000","last_change":"2024-03-24T12:12:20.673896+0000","last_active":"2024-03-24T12:13:39.699091+0000","last_peered":"2024-03-24T12:13:39.699091+0000","last_clean":"2024-03-24T12:13:39.699091+0000","last_became_active":"2024-03-24T12:05:44.734527+0000","last_became_peered":"2024-03-24T12:05:44.734527+0000","last_unstale":"2024-03-24T12:13:39.699091+0000","last_undegraded":"2024-03-24T12:13:39.699091+0000","last_fullsized":"2024-03-24T12:13:39.699091+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-24T12:12:20.673798+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:12:20.673798+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.976568+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":691,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698061+0000","last_change":"2024-03-24T12:12:56.754911+0000","last_active":"2024-03-24T12:13:39.698061+0000","last_peered":"2024-03-24T12:13:39.698061+0000","last_clean":"2024-03-24T12:13:39.698061+0000","last_became_active":"2024-03-24T12:09:04.230111+0000","last_became_peered":"2024-03-24T12:09:04.230111+0000","last_unstale":"2024-03-24T12:13:39.698061+0000","last_undegraded":"2024-03-24T12:13:39.698061+0000","last_fullsized":"2024-03-24T12:13:39.698061+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:12:56.754763+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:56.754763+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:13.808280+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":646,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.699171+0000","last_change":"2024-03-24T12:11:43.684235+0000","last_active":"2024-03-24T12:13:39.699171+0000","last_peered":"2024-03-24T12:13:39.699171+0000","last_clean":"2024-03-24T12:13:39.699171+0000","last_became_active":"2024-03-24T11:59:19.099927+0000","last_became_peered":"2024-03-24T11:59:19.099927+0000","last_unstale":"2024-03-24T12:13:39.699171+0000","last_undegraded":"2024-03-24T12:13:39.699171+0000","last_fullsized":"2024-03-24T12:13:39.699171+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.684070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:48.250404+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.684070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075885+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":383,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695496+0000","last_change":"2024-03-24T12:13:00.471271+0000","last_active":"2024-03-24T12:13:39.695496+0000","last_peered":"2024-03-24T12:13:39.695496+0000","last_clean":"2024-03-24T12:13:39.695496+0000","last_became_active":"2024-03-24T12:05:44.733737+0000","last_became_peered":"2024-03-24T12:05:44.733737+0000","last_unstale":"2024-03-24T12:13:39.695496+0000","last_undegraded":"2024-03-24T12:13:39.695496+0000","last_fullsized":"2024-03-24T12:13:39.695496+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:00.471173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:13:00.471173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:21.910921+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":841,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.030917+0000","last_change":"2024-03-24T12:10:09.762381+0000","last_active":"2024-03-24T12:13:42.030917+0000","last_peered":"2024-03-24T12:13:42.030917+0000","last_clean":"2024-03-24T12:13:42.030917+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:13:42.030917+0000","last_undegraded":"2024-03-24T12:13:42.030917+0000","last_fullsized":"2024-03-24T12:13:42.030917+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:09.762247+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:09.762247+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"477'12695","reported_seq":12941,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.781878+0000","last_change":"2024-03-24T12:13:22.041196+0000","last_active":"2024-03-24T12:13:41.781878+0000","last_peered":"2024-03-24T12:13:41.781878+0000","last_clean":"2024-03-24T12:13:41.781878+0000","last_became_active":"2024-03-24T12:13:22.024920+0000","last_became_peered":"2024-03-24T12:13:22.024920+0000","last_unstale":"2024-03-24T12:13:41.781878+0000","last_undegraded":"2024-03-24T12:13:41.781878+0000","last_fullsized":"2024-03-24T12:13:41.781878+0000","mapping_epoch":465,"log_start":"441'6700","ondisk_log_start":"441'6700","created":417,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'11435","last_scrub_stamp":"2024-03-24T12:13:02.538156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.538156+0000","objects_scrubbed":2637,"log_size":5995,"log_dups_size":0,"ondisk_log_size":5995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:07.883901+0000","scrub_duration":636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":4131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"450'35","reported_seq":947,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698880+0000","last_change":"2024-03-24T12:12:28.791017+0000","last_active":"2024-03-24T12:13:39.698880+0000","last_peered":"2024-03-24T12:13:39.698880+0000","last_clean":"2024-03-24T12:13:39.698880+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:13:39.698880+0000","last_undegraded":"2024-03-24T12:13:39.698880+0000","last_fullsized":"2024-03-24T12:13:39.698880+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:12:28.790890+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:12:28.790890+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:42.068189+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":646,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.696850+0000","last_change":"2024-03-24T12:10:31.694887+0000","last_active":"2024-03-24T12:13:39.696850+0000","last_peered":"2024-03-24T12:13:39.696850+0000","last_clean":"2024-03-24T12:13:39.696850+0000","last_became_active":"2024-03-24T11:59:19.116589+0000","last_became_peered":"2024-03-24T11:59:19.116589+0000","last_unstale":"2024-03-24T12:13:39.696850+0000","last_undegraded":"2024-03-24T12:13:39.696850+0000","last_fullsized":"2024-03-24T12:13:39.696850+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:31.694756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:10:31.694756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503086+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":1197,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.024203+0000","last_change":"2024-03-24T12:12:58.102547+0000","last_active":"2024-03-24T12:13:42.024203+0000","last_peered":"2024-03-24T12:13:42.024203+0000","last_clean":"2024-03-24T12:13:42.024203+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:13:42.024203+0000","last_undegraded":"2024-03-24T12:13:42.024203+0000","last_fullsized":"2024-03-24T12:13:42.024203+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:12:58.102424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:58.102424+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:58.499354+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":909,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.699470+0000","last_change":"2024-03-24T12:12:58.847886+0000","last_active":"2024-03-24T12:13:39.699470+0000","last_peered":"2024-03-24T12:13:39.699470+0000","last_clean":"2024-03-24T12:13:39.699470+0000","last_became_active":"2024-03-24T12:08:36.161088+0000","last_became_peered":"2024-03-24T12:08:36.161088+0000","last_unstale":"2024-03-24T12:13:39.699470+0000","last_undegraded":"2024-03-24T12:13:39.699470+0000","last_fullsized":"2024-03-24T12:13:39.699470+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:12:58.847759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:12:58.847759+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:13.141618+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":1235,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.689501+0000","last_change":"2024-03-24T12:11:42.368078+0000","last_active":"2024-03-24T12:13:39.689501+0000","last_peered":"2024-03-24T12:13:39.689501+0000","last_clean":"2024-03-24T12:13:39.689501+0000","last_became_active":"2024-03-24T12:08:36.161668+0000","last_became_peered":"2024-03-24T12:08:36.161668+0000","last_unstale":"2024-03-24T12:13:39.689501+0000","last_undegraded":"2024-03-24T12:13:39.689501+0000","last_fullsized":"2024-03-24T12:13:39.689501+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:42.367917+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:11:42.367917+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"477'12450","reported_seq":12625,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.755065+0000","last_change":"2024-03-24T12:13:02.009097+0000","last_active":"2024-03-24T12:13:41.755065+0000","last_peered":"2024-03-24T12:13:41.755065+0000","last_clean":"2024-03-24T12:13:41.755065+0000","last_became_active":"2024-03-24T12:13:01.128077+0000","last_became_peered":"2024-03-24T12:13:01.128077+0000","last_unstale":"2024-03-24T12:13:41.755065+0000","last_undegraded":"2024-03-24T12:13:41.755065+0000","last_fullsized":"2024-03-24T12:13:41.755065+0000","mapping_epoch":417,"log_start":"443'6800","ondisk_log_start":"443'6800","created":417,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'11180","last_scrub_stamp":"2024-03-24T12:13:02.007606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.007606+0000","objects_scrubbed":2603,"log_size":5650,"log_dups_size":0,"ondisk_log_size":5650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:08.619747+0000","scrub_duration":759,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86573056,"num_objects":1321,"num_object_clones":0,"num_object_copies":3963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1282,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"450'46","reported_seq":993,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032220+0000","last_change":"2024-03-24T12:12:13.128854+0000","last_active":"2024-03-24T12:13:42.032220+0000","last_peered":"2024-03-24T12:13:42.032220+0000","last_clean":"2024-03-24T12:13:42.032220+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:13:42.032220+0000","last_undegraded":"2024-03-24T12:13:42.032220+0000","last_fullsized":"2024-03-24T12:13:42.032220+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:12:13.128715+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:12:13.128715+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":937,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.030465+0000","last_change":"2024-03-24T12:13:03.153143+0000","last_active":"2024-03-24T12:13:42.030465+0000","last_peered":"2024-03-24T12:13:42.030465+0000","last_clean":"2024-03-24T12:13:42.030465+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:13:42.030465+0000","last_undegraded":"2024-03-24T12:13:42.030465+0000","last_fullsized":"2024-03-24T12:13:42.030465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:03.153016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:13:03.153016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:30.607442+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":1173,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.699403+0000","last_change":"2024-03-24T12:13:02.743315+0000","last_active":"2024-03-24T12:13:39.699403+0000","last_peered":"2024-03-24T12:13:39.699403+0000","last_clean":"2024-03-24T12:13:39.699403+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:13:39.699403+0000","last_undegraded":"2024-03-24T12:13:39.699403+0000","last_fullsized":"2024-03-24T12:13:39.699403+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:13:02.743203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.743203+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:09.248486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":818,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032322+0000","last_change":"2024-03-24T12:11:13.744756+0000","last_active":"2024-03-24T12:13:42.032322+0000","last_peered":"2024-03-24T12:13:42.032322+0000","last_clean":"2024-03-24T12:13:42.032322+0000","last_became_active":"2024-03-24T12:08:36.166897+0000","last_became_peered":"2024-03-24T12:08:36.166897+0000","last_unstale":"2024-03-24T12:13:42.032322+0000","last_undegraded":"2024-03-24T12:13:42.032322+0000","last_fullsized":"2024-03-24T12:13:42.032322+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:13.744672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.744672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":841,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032152+0000","last_change":"2024-03-24T12:10:11.797368+0000","last_active":"2024-03-24T12:13:42.032152+0000","last_peered":"2024-03-24T12:13:42.032152+0000","last_clean":"2024-03-24T12:13:42.032152+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:13:42.032152+0000","last_undegraded":"2024-03-24T12:13:42.032152+0000","last_fullsized":"2024-03-24T12:13:42.032152+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:11.797287+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:11.797287+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"477'12463","reported_seq":12650,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.643738+0000","last_change":"2024-03-24T12:13:22.043747+0000","last_active":"2024-03-24T12:13:41.643738+0000","last_peered":"2024-03-24T12:13:41.643738+0000","last_clean":"2024-03-24T12:13:41.643738+0000","last_became_active":"2024-03-24T12:13:22.028860+0000","last_became_peered":"2024-03-24T12:13:22.028860+0000","last_unstale":"2024-03-24T12:13:41.643738+0000","last_undegraded":"2024-03-24T12:13:41.643738+0000","last_fullsized":"2024-03-24T12:13:41.643738+0000","mapping_epoch":465,"log_start":"441'6500","ondisk_log_start":"441'6500","created":417,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'11251","last_scrub_stamp":"2024-03-24T12:13:03.203237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:03.203237+0000","objects_scrubbed":2567,"log_size":5963,"log_dups_size":0,"ondisk_log_size":5963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:27.423128+0000","scrub_duration":286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88801280,"num_objects":1355,"num_object_clones":0,"num_object_copies":4065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1212,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"450'32","reported_seq":937,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698566+0000","last_change":"2024-03-24T12:11:21.537427+0000","last_active":"2024-03-24T12:13:39.698566+0000","last_peered":"2024-03-24T12:13:39.698566+0000","last_clean":"2024-03-24T12:13:39.698566+0000","last_became_active":"2024-03-24T11:52:17.350867+0000","last_became_peered":"2024-03-24T11:52:17.350867+0000","last_unstale":"2024-03-24T12:13:39.698566+0000","last_undegraded":"2024-03-24T12:13:39.698566+0000","last_fullsized":"2024-03-24T12:13:39.698566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-24T12:11:21.537273+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:11:21.537273+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075605+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":927,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.689835+0000","last_change":"2024-03-24T12:13:26.353252+0000","last_active":"2024-03-24T12:13:39.689835+0000","last_peered":"2024-03-24T12:13:39.689835+0000","last_clean":"2024-03-24T12:13:39.689835+0000","last_became_active":"2024-03-24T11:52:15.141474+0000","last_became_peered":"2024-03-24T11:52:15.141474+0000","last_unstale":"2024-03-24T12:13:39.689835+0000","last_undegraded":"2024-03-24T12:13:39.689835+0000","last_fullsized":"2024-03-24T12:13:39.689835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:26.353174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:20.725884+0000","last_clean_scrub_stamp":"2024-03-24T12:13:26.353174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:41.316683+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":814,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032891+0000","last_change":"2024-03-24T12:12:29.601620+0000","last_active":"2024-03-24T12:13:42.032891+0000","last_peered":"2024-03-24T12:13:42.032891+0000","last_clean":"2024-03-24T12:13:42.032891+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:13:42.032891+0000","last_undegraded":"2024-03-24T12:13:42.032891+0000","last_fullsized":"2024-03-24T12:13:42.032891+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:29.601502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:29.601502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.074844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":893,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.694610+0000","last_change":"2024-03-24T12:10:14.749596+0000","last_active":"2024-03-24T12:13:39.694610+0000","last_peered":"2024-03-24T12:13:39.694610+0000","last_clean":"2024-03-24T12:13:39.694610+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:13:39.694610+0000","last_undegraded":"2024-03-24T12:13:39.694610+0000","last_fullsized":"2024-03-24T12:13:39.694610+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:14.749468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:14.749468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"477'13576","reported_seq":13710,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.813625+0000","last_change":"2024-03-24T12:10:42.552380+0000","last_active":"2024-03-24T12:13:41.813625+0000","last_peered":"2024-03-24T12:13:41.813625+0000","last_clean":"2024-03-24T12:13:41.813625+0000","last_became_active":"2024-03-24T12:10:42.552054+0000","last_became_peered":"2024-03-24T12:10:42.552054+0000","last_unstale":"2024-03-24T12:13:41.813625+0000","last_undegraded":"2024-03-24T12:13:41.813625+0000","last_fullsized":"2024-03-24T12:13:41.813625+0000","mapping_epoch":417,"log_start":"451'9070","ondisk_log_start":"451'9070","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4506,"log_dups_size":0,"ondisk_log_size":4506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180224000,"num_objects":2750,"num_object_clones":0,"num_object_copies":8250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13576,"num_write_kb":522432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":1155,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.700477+0000","last_change":"2024-03-24T12:11:38.190269+0000","last_active":"2024-03-24T12:13:39.700477+0000","last_peered":"2024-03-24T12:13:39.700477+0000","last_clean":"2024-03-24T12:13:39.700477+0000","last_became_active":"2024-03-24T11:52:19.382304+0000","last_became_peered":"2024-03-24T11:52:19.382304+0000","last_unstale":"2024-03-24T12:13:39.700477+0000","last_undegraded":"2024-03-24T12:13:39.700477+0000","last_fullsized":"2024-03-24T12:13:39.700477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:11:38.190121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:38.190121+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503277+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"450'49","reported_seq":969,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697311+0000","last_change":"2024-03-24T12:10:19.407534+0000","last_active":"2024-03-24T12:13:39.697311+0000","last_peered":"2024-03-24T12:13:39.697311+0000","last_clean":"2024-03-24T12:13:39.697311+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:13:39.697311+0000","last_undegraded":"2024-03-24T12:13:39.697311+0000","last_fullsized":"2024-03-24T12:13:39.697311+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-03-24T12:10:19.407433+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.407433+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.693577+0000","last_change":"2024-03-24T12:13:22.025076+0000","last_active":"2024-03-24T12:13:39.693577+0000","last_peered":"2024-03-24T12:13:39.693577+0000","last_clean":"2024-03-24T12:13:39.693577+0000","last_became_active":"2024-03-24T12:13:22.024768+0000","last_became_peered":"2024-03-24T12:13:22.024768+0000","last_unstale":"2024-03-24T12:13:39.693577+0000","last_undegraded":"2024-03-24T12:13:39.693577+0000","last_fullsized":"2024-03-24T12:13:39.693577+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:03.950368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:13:03.950368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:32.872018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":922,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.691608+0000","last_change":"2024-03-24T12:12:24.532173+0000","last_active":"2024-03-24T12:13:39.691608+0000","last_peered":"2024-03-24T12:13:39.691608+0000","last_clean":"2024-03-24T12:13:39.691608+0000","last_became_active":"2024-03-24T12:05:44.740242+0000","last_became_peered":"2024-03-24T12:05:44.740242+0000","last_unstale":"2024-03-24T12:13:39.691608+0000","last_undegraded":"2024-03-24T12:13:39.691608+0000","last_fullsized":"2024-03-24T12:13:39.691608+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:24.531916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:12:24.531916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":981,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.694822+0000","last_change":"2024-03-24T12:12:28.260533+0000","last_active":"2024-03-24T12:13:39.694822+0000","last_peered":"2024-03-24T12:13:39.694822+0000","last_clean":"2024-03-24T12:13:39.694822+0000","last_became_active":"2024-03-24T12:10:47.971906+0000","last_became_peered":"2024-03-24T12:10:47.971906+0000","last_unstale":"2024-03-24T12:13:39.694822+0000","last_undegraded":"2024-03-24T12:13:39.694822+0000","last_fullsized":"2024-03-24T12:13:39.694822+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:12:28.260278+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:12:28.260278+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503450+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"477'14059","reported_seq":14279,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.826936+0000","last_change":"2024-03-24T12:12:12.434955+0000","last_active":"2024-03-24T12:13:41.826936+0000","last_peered":"2024-03-24T12:13:41.826936+0000","last_clean":"2024-03-24T12:13:41.826936+0000","last_became_active":"2024-03-24T12:10:42.562110+0000","last_became_peered":"2024-03-24T12:10:42.562110+0000","last_unstale":"2024-03-24T12:13:41.826936+0000","last_undegraded":"2024-03-24T12:13:41.826936+0000","last_fullsized":"2024-03-24T12:13:41.826936+0000","mapping_epoch":417,"log_start":"454'10127","ondisk_log_start":"454'10127","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7838","last_scrub_stamp":"2024-03-24T12:12:12.434659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:12.434659+0000","objects_scrubbed":7761,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182648832,"num_objects":2787,"num_object_clones":0,"num_object_copies":8361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14059,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":886,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697442+0000","last_change":"2024-03-24T12:11:32.519689+0000","last_active":"2024-03-24T12:13:39.697442+0000","last_peered":"2024-03-24T12:13:39.697442+0000","last_clean":"2024-03-24T12:13:39.697442+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:13:39.697442+0000","last_undegraded":"2024-03-24T12:13:39.697442+0000","last_fullsized":"2024-03-24T12:13:39.697442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.519559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:24.679110+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.519559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"450'50","reported_seq":969,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697558+0000","last_change":"2024-03-24T12:10:29.733172+0000","last_active":"2024-03-24T12:13:39.697558+0000","last_peered":"2024-03-24T12:13:39.697558+0000","last_clean":"2024-03-24T12:13:39.697558+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:13:39.697558+0000","last_undegraded":"2024-03-24T12:13:39.697558+0000","last_fullsized":"2024-03-24T12:13:39.697558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'38","last_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_deep_scrub":"241'38","last_deep_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_clean_scrub_stamp":"2024-03-24T12:10:29.733073+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":42,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698796+0000","last_change":"2024-03-24T12:13:22.032208+0000","last_active":"2024-03-24T12:13:39.698796+0000","last_peered":"2024-03-24T12:13:39.698796+0000","last_clean":"2024-03-24T12:13:39.698796+0000","last_became_active":"2024-03-24T12:13:22.031839+0000","last_became_peered":"2024-03-24T12:13:22.031839+0000","last_unstale":"2024-03-24T12:13:39.698796+0000","last_undegraded":"2024-03-24T12:13:39.698796+0000","last_fullsized":"2024-03-24T12:13:39.698796+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:13:18.950105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:13:18.950105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:42.191287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":904,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697832+0000","last_change":"2024-03-24T12:10:38.401132+0000","last_active":"2024-03-24T12:13:39.697832+0000","last_peered":"2024-03-24T12:13:39.697832+0000","last_clean":"2024-03-24T12:13:39.697832+0000","last_became_active":"2024-03-24T12:05:44.733553+0000","last_became_peered":"2024-03-24T12:05:44.733553+0000","last_unstale":"2024-03-24T12:13:39.697832+0000","last_undegraded":"2024-03-24T12:13:39.697832+0000","last_fullsized":"2024-03-24T12:13:39.697832+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:38.401028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:22.782610+0000","last_clean_scrub_stamp":"2024-03-24T12:10:38.401028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":979,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698365+0000","last_change":"2024-03-24T12:10:45.574511+0000","last_active":"2024-03-24T12:13:39.698365+0000","last_peered":"2024-03-24T12:13:39.698365+0000","last_clean":"2024-03-24T12:13:39.698365+0000","last_became_active":"2024-03-24T12:10:45.573960+0000","last_became_peered":"2024-03-24T12:10:45.573960+0000","last_unstale":"2024-03-24T12:13:39.698365+0000","last_undegraded":"2024-03-24T12:13:39.698365+0000","last_fullsized":"2024-03-24T12:13:39.698365+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:10:25.359883+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.359883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"478'13516","reported_seq":13739,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.294988+0000","last_change":"2024-03-24T12:12:19.183366+0000","last_active":"2024-03-24T12:13:42.294988+0000","last_peered":"2024-03-24T12:13:42.294988+0000","last_clean":"2024-03-24T12:13:42.294988+0000","last_became_active":"2024-03-24T12:10:42.552346+0000","last_became_peered":"2024-03-24T12:10:42.552346+0000","last_unstale":"2024-03-24T12:13:42.294988+0000","last_undegraded":"2024-03-24T12:13:42.294988+0000","last_fullsized":"2024-03-24T12:13:42.294988+0000","mapping_epoch":417,"log_start":"437'3500","ondisk_log_start":"437'3500","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7988","last_scrub_stamp":"2024-03-24T12:12:19.183036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.183036+0000","objects_scrubbed":7889,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:43.264884+0000","scrub_duration":3159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172359680,"num_objects":2630,"num_object_clones":0,"num_object_copies":7890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13516,"num_write_kb":516672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":673,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.033410+0000","last_change":"2024-03-24T12:13:22.024167+0000","last_active":"2024-03-24T12:13:42.033410+0000","last_peered":"2024-03-24T12:13:42.033410+0000","last_clean":"2024-03-24T12:13:42.033410+0000","last_became_active":"2024-03-24T12:13:22.023853+0000","last_became_peered":"2024-03-24T12:13:22.023853+0000","last_unstale":"2024-03-24T12:13:42.033410+0000","last_undegraded":"2024-03-24T12:13:42.033410+0000","last_fullsized":"2024-03-24T12:13:42.033410+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:30.833447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.319837+0000","last_clean_scrub_stamp":"2024-03-24T12:12:30.833447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"450'47","reported_seq":976,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035234+0000","last_change":"2024-03-24T12:11:37.405546+0000","last_active":"2024-03-24T12:13:42.035234+0000","last_peered":"2024-03-24T12:13:42.035234+0000","last_clean":"2024-03-24T12:13:42.035234+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:13:42.035234+0000","last_undegraded":"2024-03-24T12:13:42.035234+0000","last_fullsized":"2024-03-24T12:13:42.035234+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'37","last_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_deep_scrub":"369'37","last_deep_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.405400+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":928,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.693114+0000","last_change":"2024-03-24T12:11:17.183566+0000","last_active":"2024-03-24T12:13:39.693114+0000","last_peered":"2024-03-24T12:13:39.693114+0000","last_clean":"2024-03-24T12:13:39.693114+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:13:39.693114+0000","last_undegraded":"2024-03-24T12:13:39.693114+0000","last_fullsized":"2024-03-24T12:13:39.693114+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:17.183126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:02.041192+0000","last_clean_scrub_stamp":"2024-03-24T12:11:17.183126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":306,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.689407+0000","last_change":"2024-03-24T12:13:22.026949+0000","last_active":"2024-03-24T12:13:39.689407+0000","last_peered":"2024-03-24T12:13:39.689407+0000","last_clean":"2024-03-24T12:13:39.689407+0000","last_became_active":"2024-03-24T12:13:22.026572+0000","last_became_peered":"2024-03-24T12:13:22.026572+0000","last_unstale":"2024-03-24T12:13:39.689407+0000","last_undegraded":"2024-03-24T12:13:39.689407+0000","last_fullsized":"2024-03-24T12:13:39.689407+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:20.921985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:20.921985+0000","last_clean_scrub_stamp":"2024-03-24T12:12:20.921985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:48.236215+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":331,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035243+0000","last_change":"2024-03-24T12:13:22.024988+0000","last_active":"2024-03-24T12:13:42.035243+0000","last_peered":"2024-03-24T12:13:42.035243+0000","last_clean":"2024-03-24T12:13:42.035243+0000","last_became_active":"2024-03-24T12:13:22.024645+0000","last_became_peered":"2024-03-24T12:13:22.024645+0000","last_unstale":"2024-03-24T12:13:42.035243+0000","last_undegraded":"2024-03-24T12:13:42.035243+0000","last_fullsized":"2024-03-24T12:13:42.035243+0000","mapping_epoch":465,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.081517+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.528858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"477'12382","reported_seq":12555,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.738397+0000","last_change":"2024-03-24T12:13:03.626923+0000","last_active":"2024-03-24T12:13:41.738397+0000","last_peered":"2024-03-24T12:13:41.738397+0000","last_clean":"2024-03-24T12:13:41.738397+0000","last_became_active":"2024-03-24T12:13:01.136574+0000","last_became_peered":"2024-03-24T12:13:01.136574+0000","last_unstale":"2024-03-24T12:13:41.738397+0000","last_undegraded":"2024-03-24T12:13:41.738397+0000","last_fullsized":"2024-03-24T12:13:41.738397+0000","mapping_epoch":417,"log_start":"442'6700","ondisk_log_start":"442'6700","created":417,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'11168","last_scrub_stamp":"2024-03-24T12:13:03.626811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:03.626811+0000","objects_scrubbed":2602,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:20.382927+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90963968,"num_objects":1388,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1214,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":780,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.033101+0000","last_change":"2024-03-24T12:12:21.577856+0000","last_active":"2024-03-24T12:13:42.033101+0000","last_peered":"2024-03-24T12:13:42.033101+0000","last_clean":"2024-03-24T12:13:42.033101+0000","last_became_active":"2024-03-24T12:08:36.165265+0000","last_became_peered":"2024-03-24T12:08:36.165265+0000","last_unstale":"2024-03-24T12:13:42.033101+0000","last_undegraded":"2024-03-24T12:13:42.033101+0000","last_fullsized":"2024-03-24T12:13:42.033101+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:21.577764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:21.577764+0000","last_clean_scrub_stamp":"2024-03-24T12:12:21.577764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:45.211618+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"450'42","reported_seq":980,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.034885+0000","last_change":"2024-03-24T12:11:23.383853+0000","last_active":"2024-03-24T12:13:42.034885+0000","last_peered":"2024-03-24T12:13:42.034885+0000","last_clean":"2024-03-24T12:13:42.034885+0000","last_became_active":"2024-03-24T12:05:44.742635+0000","last_became_peered":"2024-03-24T12:05:44.742635+0000","last_unstale":"2024-03-24T12:13:42.034885+0000","last_undegraded":"2024-03-24T12:13:42.034885+0000","last_fullsized":"2024-03-24T12:13:42.034885+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-24T12:11:23.383616+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.383616+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":749,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032571+0000","last_change":"2024-03-24T12:12:54.407163+0000","last_active":"2024-03-24T12:13:42.032571+0000","last_peered":"2024-03-24T12:13:42.032571+0000","last_clean":"2024-03-24T12:13:42.032571+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:13:42.032571+0000","last_undegraded":"2024-03-24T12:13:42.032571+0000","last_fullsized":"2024-03-24T12:13:42.032571+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:54.407046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:54.407046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:13.910351+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":923,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035622+0000","last_change":"2024-03-24T12:12:26.485548+0000","last_active":"2024-03-24T12:13:42.035622+0000","last_peered":"2024-03-24T12:13:42.035622+0000","last_clean":"2024-03-24T12:13:42.035622+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:13:42.035622+0000","last_undegraded":"2024-03-24T12:13:42.035622+0000","last_fullsized":"2024-03-24T12:13:42.035622+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:26.485429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:12:26.485429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":919,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031804+0000","last_change":"2024-03-24T12:11:39.903255+0000","last_active":"2024-03-24T12:13:42.031804+0000","last_peered":"2024-03-24T12:13:42.031804+0000","last_clean":"2024-03-24T12:13:42.031804+0000","last_became_active":"2024-03-24T12:08:36.173357+0000","last_became_peered":"2024-03-24T12:08:36.173357+0000","last_unstale":"2024-03-24T12:13:42.031804+0000","last_undegraded":"2024-03-24T12:13:42.031804+0000","last_fullsized":"2024-03-24T12:13:42.031804+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_clean_scrub_stamp":"2024-03-24T12:11:39.903133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":333,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698034+0000","last_change":"2024-03-24T12:13:22.030568+0000","last_active":"2024-03-24T12:13:39.698034+0000","last_peered":"2024-03-24T12:13:39.698034+0000","last_clean":"2024-03-24T12:13:39.698034+0000","last_became_active":"2024-03-24T12:13:22.029299+0000","last_became_peered":"2024-03-24T12:13:22.029299+0000","last_unstale":"2024-03-24T12:13:39.698034+0000","last_undegraded":"2024-03-24T12:13:39.698034+0000","last_fullsized":"2024-03-24T12:13:39.698034+0000","mapping_epoch":465,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:12:23.891623+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.891623+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:43.056701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":18,"num_write":11,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"477'13660","reported_seq":13794,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.782813+0000","last_change":"2024-03-24T12:10:42.558754+0000","last_active":"2024-03-24T12:13:41.782813+0000","last_peered":"2024-03-24T12:13:41.782813+0000","last_clean":"2024-03-24T12:13:41.782813+0000","last_became_active":"2024-03-24T12:10:42.558225+0000","last_became_peered":"2024-03-24T12:10:42.558225+0000","last_unstale":"2024-03-24T12:13:41.782813+0000","last_undegraded":"2024-03-24T12:13:41.782813+0000","last_fullsized":"2024-03-24T12:13:41.782813+0000","mapping_epoch":417,"log_start":"453'9654","ondisk_log_start":"453'9654","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178520064,"num_objects":2724,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13660,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":921,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.690321+0000","last_change":"2024-03-24T12:13:03.231000+0000","last_active":"2024-03-24T12:13:39.690321+0000","last_peered":"2024-03-24T12:13:39.690321+0000","last_clean":"2024-03-24T12:13:39.690321+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:13:39.690321+0000","last_undegraded":"2024-03-24T12:13:39.690321+0000","last_fullsized":"2024-03-24T12:13:39.690321+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:03.230924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:13:03.230924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:17.454338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"469'41","reported_seq":1203,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.699034+0000","last_change":"2024-03-24T12:13:22.030185+0000","last_active":"2024-03-24T12:13:39.699034+0000","last_peered":"2024-03-24T12:13:39.699034+0000","last_clean":"2024-03-24T12:13:39.699034+0000","last_became_active":"2024-03-24T12:13:22.029830+0000","last_became_peered":"2024-03-24T12:13:22.029830+0000","last_unstale":"2024-03-24T12:13:39.699034+0000","last_undegraded":"2024-03-24T12:13:39.699034+0000","last_fullsized":"2024-03-24T12:13:39.699034+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-24T12:11:32.027804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.027804+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075671+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":1233,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.689419+0000","last_change":"2024-03-24T12:11:35.238528+0000","last_active":"2024-03-24T12:13:39.689419+0000","last_peered":"2024-03-24T12:13:39.689419+0000","last_clean":"2024-03-24T12:13:39.689419+0000","last_became_active":"2024-03-24T12:10:33.461798+0000","last_became_peered":"2024-03-24T12:10:33.461798+0000","last_unstale":"2024-03-24T12:13:39.689419+0000","last_undegraded":"2024-03-24T12:13:39.689419+0000","last_fullsized":"2024-03-24T12:13:39.689419+0000","mapping_epoch":64,"log_start":"83'130","ondisk_log_start":"83'130","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:35.238422+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:10:25.984109+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.238422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":820,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031908+0000","last_change":"2024-03-24T12:12:30.636570+0000","last_active":"2024-03-24T12:13:42.031908+0000","last_peered":"2024-03-24T12:13:42.031908+0000","last_clean":"2024-03-24T12:13:42.031908+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:13:42.031908+0000","last_undegraded":"2024-03-24T12:13:42.031908+0000","last_fullsized":"2024-03-24T12:13:42.031908+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:30.636466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:30.636466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:52.420266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":907,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.696683+0000","last_change":"2024-03-24T12:08:43.113674+0000","last_active":"2024-03-24T12:13:39.696683+0000","last_peered":"2024-03-24T12:13:39.696683+0000","last_clean":"2024-03-24T12:13:39.696683+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:13:39.696683+0000","last_undegraded":"2024-03-24T12:13:39.696683+0000","last_fullsized":"2024-03-24T12:13:39.696683+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:08:43.113510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:08:43.113510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502625+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":775,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.033021+0000","last_change":"2024-03-24T12:11:35.488009+0000","last_active":"2024-03-24T12:13:42.033021+0000","last_peered":"2024-03-24T12:13:42.033021+0000","last_clean":"2024-03-24T12:13:42.033021+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:13:42.033021+0000","last_undegraded":"2024-03-24T12:13:42.033021+0000","last_fullsized":"2024-03-24T12:13:42.033021+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:35.487872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.487872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"477'12478","reported_seq":12646,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.689456+0000","last_change":"2024-03-24T12:13:26.988691+0000","last_active":"2024-03-24T12:13:41.689456+0000","last_peered":"2024-03-24T12:13:41.689456+0000","last_clean":"2024-03-24T12:13:41.689456+0000","last_became_active":"2024-03-24T12:13:01.148794+0000","last_became_peered":"2024-03-24T12:13:01.148794+0000","last_unstale":"2024-03-24T12:13:41.689456+0000","last_undegraded":"2024-03-24T12:13:41.689456+0000","last_fullsized":"2024-03-24T12:13:41.689456+0000","mapping_epoch":417,"log_start":"442'6800","ondisk_log_start":"442'6800","created":417,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'11892","last_scrub_stamp":"2024-03-24T12:13:26.988191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:26.988191+0000","objects_scrubbed":1969,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:55.390080+0000","scrub_duration":581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":4131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":592,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":302,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032205+0000","last_change":"2024-03-24T12:12:18.565246+0000","last_active":"2024-03-24T12:13:42.032205+0000","last_peered":"2024-03-24T12:13:42.032205+0000","last_clean":"2024-03-24T12:13:42.032205+0000","last_became_active":"2024-03-24T12:08:36.162188+0000","last_became_peered":"2024-03-24T12:08:36.162188+0000","last_unstale":"2024-03-24T12:13:42.032205+0000","last_undegraded":"2024-03-24T12:13:42.032205+0000","last_fullsized":"2024-03-24T12:13:42.032205+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:18.565128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:12:18.565128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.801563+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"450'21","reported_seq":973,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031889+0000","last_change":"2024-03-24T12:12:58.130589+0000","last_active":"2024-03-24T12:13:42.031889+0000","last_peered":"2024-03-24T12:13:42.031889+0000","last_clean":"2024-03-24T12:13:42.031889+0000","last_became_active":"2024-03-24T12:08:36.172879+0000","last_became_peered":"2024-03-24T12:08:36.172879+0000","last_unstale":"2024-03-24T12:13:42.031889+0000","last_undegraded":"2024-03-24T12:13:42.031889+0000","last_fullsized":"2024-03-24T12:13:42.031889+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:12:58.129252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:58.129252+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:04.557719+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":376,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697539+0000","last_change":"2024-03-24T12:10:12.685963+0000","last_active":"2024-03-24T12:13:39.697539+0000","last_peered":"2024-03-24T12:13:39.697539+0000","last_clean":"2024-03-24T12:13:39.697539+0000","last_became_active":"2024-03-24T12:05:44.738608+0000","last_became_peered":"2024-03-24T12:05:44.738608+0000","last_unstale":"2024-03-24T12:13:39.697539+0000","last_undegraded":"2024-03-24T12:13:39.697539+0000","last_fullsized":"2024-03-24T12:13:39.697539+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.685815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.685815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502690+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"462'6","reported_seq":909,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.056240+0000","last_change":"2024-03-24T12:13:22.026448+0000","last_active":"2024-03-24T12:13:42.056240+0000","last_peered":"2024-03-24T12:13:42.056240+0000","last_clean":"2024-03-24T12:13:42.056240+0000","last_became_active":"2024-03-24T12:13:22.025820+0000","last_became_peered":"2024-03-24T12:13:22.025820+0000","last_unstale":"2024-03-24T12:13:42.056240+0000","last_undegraded":"2024-03-24T12:13:42.056240+0000","last_fullsized":"2024-03-24T12:13:42.056240+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'6","last_scrub_stamp":"2024-03-24T12:13:14.961156+0000","last_deep_scrub":"462'6","last_deep_scrub_stamp":"2024-03-24T12:13:14.961156+0000","last_clean_scrub_stamp":"2024-03-24T12:13:14.961156+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:19.575262+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":477,"num_read_kb":472,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":908,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.692325+0000","last_change":"2024-03-24T12:11:31.240744+0000","last_active":"2024-03-24T12:13:39.692325+0000","last_peered":"2024-03-24T12:13:39.692325+0000","last_clean":"2024-03-24T12:13:39.692325+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:13:39.692325+0000","last_undegraded":"2024-03-24T12:13:39.692325+0000","last_fullsized":"2024-03-24T12:13:39.692325+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:31.240650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:31.240650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":1232,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.692748+0000","last_change":"2024-03-24T12:11:30.174297+0000","last_active":"2024-03-24T12:13:39.692748+0000","last_peered":"2024-03-24T12:13:39.692748+0000","last_clean":"2024-03-24T12:13:39.692748+0000","last_became_active":"2024-03-24T12:10:23.398177+0000","last_became_peered":"2024-03-24T12:10:23.398177+0000","last_unstale":"2024-03-24T12:13:39.692748+0000","last_undegraded":"2024-03-24T12:13:39.692748+0000","last_fullsized":"2024-03-24T12:13:39.692748+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:30.173569+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:11:30.173569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"477'13900","reported_seq":14121,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.775159+0000","last_change":"2024-03-24T12:12:14.049841+0000","last_active":"2024-03-24T12:13:41.775159+0000","last_peered":"2024-03-24T12:13:41.775159+0000","last_clean":"2024-03-24T12:13:41.775159+0000","last_became_active":"2024-03-24T12:10:42.559548+0000","last_became_peered":"2024-03-24T12:10:42.559548+0000","last_unstale":"2024-03-24T12:13:41.775159+0000","last_undegraded":"2024-03-24T12:13:41.775159+0000","last_fullsized":"2024-03-24T12:13:41.775159+0000","mapping_epoch":417,"log_start":"453'9825","ondisk_log_start":"453'9825","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7951","last_scrub_stamp":"2024-03-24T12:12:14.049510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.049510+0000","objects_scrubbed":7849,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503737+0000","scrub_duration":2948,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184811520,"num_objects":2820,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13900,"num_write_kb":535040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":897,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.034003+0000","last_change":"2024-03-24T12:11:38.437189+0000","last_active":"2024-03-24T12:13:42.034003+0000","last_peered":"2024-03-24T12:13:42.034003+0000","last_clean":"2024-03-24T12:13:42.034003+0000","last_became_active":"2024-03-24T11:52:19.386077+0000","last_became_peered":"2024-03-24T11:52:19.386077+0000","last_unstale":"2024-03-24T12:13:42.034003+0000","last_undegraded":"2024-03-24T12:13:42.034003+0000","last_fullsized":"2024-03-24T12:13:42.034003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:38.437087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:11:38.437087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"450'56","reported_seq":1076,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032359+0000","last_change":"2024-03-24T12:10:33.726470+0000","last_active":"2024-03-24T12:13:42.032359+0000","last_peered":"2024-03-24T12:13:42.032359+0000","last_clean":"2024-03-24T12:13:42.032359+0000","last_became_active":"2024-03-24T12:08:36.152866+0000","last_became_peered":"2024-03-24T12:08:36.152866+0000","last_unstale":"2024-03-24T12:13:42.032359+0000","last_undegraded":"2024-03-24T12:13:42.032359+0000","last_fullsized":"2024-03-24T12:13:42.032359+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'44","last_scrub_stamp":"2024-03-24T12:10:33.726357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:33.726357+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":784,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031395+0000","last_change":"2024-03-24T12:11:19.470678+0000","last_active":"2024-03-24T12:13:42.031395+0000","last_peered":"2024-03-24T12:13:42.031395+0000","last_clean":"2024-03-24T12:13:42.031395+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:13:42.031395+0000","last_undegraded":"2024-03-24T12:13:42.031395+0000","last_fullsized":"2024-03-24T12:13:42.031395+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:19.470485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:09.913946+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.470485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":884,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.696581+0000","last_change":"2024-03-24T12:11:28.156543+0000","last_active":"2024-03-24T12:13:39.696581+0000","last_peered":"2024-03-24T12:13:39.696581+0000","last_clean":"2024-03-24T12:13:39.696581+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:13:39.696581+0000","last_undegraded":"2024-03-24T12:13:39.696581+0000","last_fullsized":"2024-03-24T12:13:39.696581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:28.156411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.156411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503141+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":868,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031066+0000","last_change":"2024-03-24T12:12:58.573186+0000","last_active":"2024-03-24T12:13:42.031066+0000","last_peered":"2024-03-24T12:13:42.031066+0000","last_clean":"2024-03-24T12:13:42.031066+0000","last_became_active":"2024-03-24T12:10:26.959434+0000","last_became_peered":"2024-03-24T12:10:26.959434+0000","last_unstale":"2024-03-24T12:13:42.031066+0000","last_undegraded":"2024-03-24T12:13:42.031066+0000","last_fullsized":"2024-03-24T12:13:42.031066+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:12:58.573061+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:12:58.573061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:25.297540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"477'13639","reported_seq":13774,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.752758+0000","last_change":"2024-03-24T12:10:42.558548+0000","last_active":"2024-03-24T12:13:41.752758+0000","last_peered":"2024-03-24T12:13:41.752758+0000","last_clean":"2024-03-24T12:13:41.752758+0000","last_became_active":"2024-03-24T12:10:42.558230+0000","last_became_peered":"2024-03-24T12:10:42.558230+0000","last_unstale":"2024-03-24T12:13:41.752758+0000","last_undegraded":"2024-03-24T12:13:41.752758+0000","last_fullsized":"2024-03-24T12:13:41.752758+0000","mapping_epoch":417,"log_start":"451'9158","ondisk_log_start":"451'9158","created":417,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183631896,"num_objects":2803,"num_object_clones":0,"num_object_copies":8409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13639,"num_write_kb":526081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":1174,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035519+0000","last_change":"2024-03-24T12:11:19.363311+0000","last_active":"2024-03-24T12:13:42.035519+0000","last_peered":"2024-03-24T12:13:42.035519+0000","last_clean":"2024-03-24T12:13:42.035519+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:13:42.035519+0000","last_undegraded":"2024-03-24T12:13:42.035519+0000","last_fullsized":"2024-03-24T12:13:42.035519+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:11:19.362922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:19.362922+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"450'35","reported_seq":985,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031745+0000","last_change":"2024-03-24T12:12:57.090207+0000","last_active":"2024-03-24T12:13:42.031745+0000","last_peered":"2024-03-24T12:13:42.031745+0000","last_clean":"2024-03-24T12:13:42.031745+0000","last_became_active":"2024-03-24T12:08:36.174663+0000","last_became_peered":"2024-03-24T12:08:36.174663+0000","last_unstale":"2024-03-24T12:13:42.031745+0000","last_undegraded":"2024-03-24T12:13:42.031745+0000","last_fullsized":"2024-03-24T12:13:42.031745+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:12:57.090061+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:12:57.090061+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:25.636827+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":655,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032914+0000","last_change":"2024-03-24T12:12:17.361772+0000","last_active":"2024-03-24T12:13:42.032914+0000","last_peered":"2024-03-24T12:13:42.032914+0000","last_clean":"2024-03-24T12:13:42.032914+0000","last_became_active":"2024-03-24T11:59:19.116896+0000","last_became_peered":"2024-03-24T11:59:19.116896+0000","last_unstale":"2024-03-24T12:13:42.032914+0000","last_undegraded":"2024-03-24T12:13:42.032914+0000","last_fullsized":"2024-03-24T12:13:42.032914+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:17.361648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:50.950565+0000","last_clean_scrub_stamp":"2024-03-24T12:12:17.361648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":900,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032308+0000","last_change":"2024-03-24T12:11:26.507311+0000","last_active":"2024-03-24T12:13:42.032308+0000","last_peered":"2024-03-24T12:13:42.032308+0000","last_clean":"2024-03-24T12:13:42.032308+0000","last_became_active":"2024-03-24T12:08:36.145936+0000","last_became_peered":"2024-03-24T12:08:36.145936+0000","last_unstale":"2024-03-24T12:13:42.032308+0000","last_undegraded":"2024-03-24T12:13:42.032308+0000","last_fullsized":"2024-03-24T12:13:42.032308+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:26.507184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:11:26.507184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":333,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.694929+0000","last_change":"2024-03-24T12:13:22.025077+0000","last_active":"2024-03-24T12:13:39.694929+0000","last_peered":"2024-03-24T12:13:39.694929+0000","last_clean":"2024-03-24T12:13:39.694929+0000","last_became_active":"2024-03-24T12:13:22.024721+0000","last_became_peered":"2024-03-24T12:13:22.024721+0000","last_unstale":"2024-03-24T12:13:39.694929+0000","last_undegraded":"2024-03-24T12:13:39.694929+0000","last_fullsized":"2024-03-24T12:13:39.694929+0000","mapping_epoch":465,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:11:18.911296+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:11:18.911296+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:43.504084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"478'12344","reported_seq":12518,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:43.897472+0000","last_change":"2024-03-24T12:13:02.244163+0000","last_active":"2024-03-24T12:13:43.897472+0000","last_peered":"2024-03-24T12:13:43.897472+0000","last_clean":"2024-03-24T12:13:43.897472+0000","last_became_active":"2024-03-24T12:13:01.127045+0000","last_became_peered":"2024-03-24T12:13:01.127045+0000","last_unstale":"2024-03-24T12:13:43.897472+0000","last_undegraded":"2024-03-24T12:13:43.897472+0000","last_fullsized":"2024-03-24T12:13:43.897472+0000","mapping_epoch":417,"log_start":"441'6000","ondisk_log_start":"441'6000","created":417,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'11061","last_scrub_stamp":"2024-03-24T12:13:02.244017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.244017+0000","objects_scrubbed":2667,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:16.814311+0000","scrub_duration":857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89915392,"num_objects":1372,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1295,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":928,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.023704+0000","last_change":"2024-03-24T12:12:55.136997+0000","last_active":"2024-03-24T12:13:42.023704+0000","last_peered":"2024-03-24T12:13:42.023704+0000","last_clean":"2024-03-24T12:13:42.023704+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:13:42.023704+0000","last_undegraded":"2024-03-24T12:13:42.023704+0000","last_fullsized":"2024-03-24T12:13:42.023704+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:55.136877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:55.136877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:00.247711+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"450'24","reported_seq":964,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697350+0000","last_change":"2024-03-24T12:12:55.719873+0000","last_active":"2024-03-24T12:13:39.697350+0000","last_peered":"2024-03-24T12:13:39.697350+0000","last_clean":"2024-03-24T12:13:39.697350+0000","last_became_active":"2024-03-24T11:52:37.789583+0000","last_became_peered":"2024-03-24T11:52:37.789583+0000","last_unstale":"2024-03-24T12:13:39.697350+0000","last_undegraded":"2024-03-24T12:13:39.697350+0000","last_fullsized":"2024-03-24T12:13:39.697350+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'24","last_scrub_stamp":"2024-03-24T12:12:55.719725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:55.719725+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:13.988370+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":913,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697390+0000","last_change":"2024-03-24T12:12:23.704478+0000","last_active":"2024-03-24T12:13:39.697390+0000","last_peered":"2024-03-24T12:13:39.697390+0000","last_clean":"2024-03-24T12:13:39.697390+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:13:39.697390+0000","last_undegraded":"2024-03-24T12:13:39.697390+0000","last_fullsized":"2024-03-24T12:13:39.697390+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:12:23.704204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:23.704204+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075977+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":884,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695075+0000","last_change":"2024-03-24T12:12:14.425125+0000","last_active":"2024-03-24T12:13:39.695075+0000","last_peered":"2024-03-24T12:13:39.695075+0000","last_clean":"2024-03-24T12:13:39.695075+0000","last_became_active":"2024-03-24T11:52:21.388681+0000","last_became_peered":"2024-03-24T11:52:21.388681+0000","last_unstale":"2024-03-24T12:13:39.695075+0000","last_undegraded":"2024-03-24T12:13:39.695075+0000","last_fullsized":"2024-03-24T12:13:39.695075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:14.424990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.424990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.564571+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"478'12439","reported_seq":19433,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:43.805333+0000","last_change":"2024-03-24T12:13:26.376552+0000","last_active":"2024-03-24T12:13:43.805333+0000","last_peered":"2024-03-24T12:13:43.805333+0000","last_clean":"2024-03-24T12:13:43.805333+0000","last_became_active":"2024-03-24T12:13:26.352304+0000","last_became_peered":"2024-03-24T12:13:26.352304+0000","last_unstale":"2024-03-24T12:13:43.805333+0000","last_undegraded":"2024-03-24T12:13:43.805333+0000","last_fullsized":"2024-03-24T12:13:43.805333+0000","mapping_epoch":469,"log_start":"442'6700","ondisk_log_start":"442'6700","created":457,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.521946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88506368,"num_objects":1351,"num_object_clones":0,"num_object_copies":4053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":263200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2280,"num_bytes_recovered":149422080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"450'25","reported_seq":945,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035023+0000","last_change":"2024-03-24T12:11:40.464348+0000","last_active":"2024-03-24T12:13:42.035023+0000","last_peered":"2024-03-24T12:13:42.035023+0000","last_clean":"2024-03-24T12:13:42.035023+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:13:42.035023+0000","last_undegraded":"2024-03-24T12:13:42.035023+0000","last_fullsized":"2024-03-24T12:13:42.035023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:40.464206+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:11:40.464206+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":393,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032481+0000","last_change":"2024-03-24T12:11:20.390445+0000","last_active":"2024-03-24T12:13:42.032481+0000","last_peered":"2024-03-24T12:13:42.032481+0000","last_clean":"2024-03-24T12:13:42.032481+0000","last_became_active":"2024-03-24T12:05:44.742856+0000","last_became_peered":"2024-03-24T12:05:44.742856+0000","last_unstale":"2024-03-24T12:13:42.032481+0000","last_undegraded":"2024-03-24T12:13:42.032481+0000","last_fullsized":"2024-03-24T12:13:42.032481+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:20.390312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.390312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":879,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695346+0000","last_change":"2024-03-24T12:10:30.721092+0000","last_active":"2024-03-24T12:13:39.695346+0000","last_peered":"2024-03-24T12:13:39.695346+0000","last_clean":"2024-03-24T12:13:39.695346+0000","last_became_active":"2024-03-24T11:52:19.376534+0000","last_became_peered":"2024-03-24T11:52:19.376534+0000","last_unstale":"2024-03-24T12:13:39.695346+0000","last_undegraded":"2024-03-24T12:13:39.695346+0000","last_fullsized":"2024-03-24T12:13:39.695346+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:30.721003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:30.721003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":382,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698598+0000","last_change":"2024-03-24T12:12:30.857644+0000","last_active":"2024-03-24T12:13:39.698598+0000","last_peered":"2024-03-24T12:13:39.698598+0000","last_clean":"2024-03-24T12:13:39.698598+0000","last_became_active":"2024-03-24T12:05:44.735889+0000","last_became_peered":"2024-03-24T12:05:44.735889+0000","last_unstale":"2024-03-24T12:13:39.698598+0000","last_undegraded":"2024-03-24T12:13:39.698598+0000","last_fullsized":"2024-03-24T12:13:39.698598+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:30.857510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:12:30.857510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:42.068220+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"478'12532","reported_seq":19579,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.284428+0000","last_change":"2024-03-24T12:13:26.909610+0000","last_active":"2024-03-24T12:13:42.284428+0000","last_peered":"2024-03-24T12:13:42.284428+0000","last_clean":"2024-03-24T12:13:42.284428+0000","last_became_active":"2024-03-24T12:13:25.371914+0000","last_became_peered":"2024-03-24T12:13:25.371914+0000","last_unstale":"2024-03-24T12:13:42.284428+0000","last_undegraded":"2024-03-24T12:13:42.284428+0000","last_fullsized":"2024-03-24T12:13:42.284428+0000","mapping_epoch":468,"log_start":"442'6800","ondisk_log_start":"442'6800","created":457,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":5,"last_scrub":"470'11879","last_scrub_stamp":"2024-03-24T12:13:26.909344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:26.909344+0000","objects_scrubbed":2017,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:39.243852+0000","scrub_duration":526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89260032,"num_objects":1362,"num_object_clones":0,"num_object_copies":4086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":655,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"450'18","reported_seq":918,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695235+0000","last_change":"2024-03-24T12:10:25.685130+0000","last_active":"2024-03-24T12:13:39.695235+0000","last_peered":"2024-03-24T12:13:39.695235+0000","last_clean":"2024-03-24T12:13:39.695235+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:13:39.695235+0000","last_undegraded":"2024-03-24T12:13:39.695235+0000","last_fullsized":"2024-03-24T12:13:39.695235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-03-24T12:10:25.685009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.685009+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697211+0000","last_change":"2024-03-24T12:13:22.030615+0000","last_active":"2024-03-24T12:13:39.697211+0000","last_peered":"2024-03-24T12:13:39.697211+0000","last_clean":"2024-03-24T12:13:39.697211+0000","last_became_active":"2024-03-24T12:13:22.029278+0000","last_became_peered":"2024-03-24T12:13:22.029278+0000","last_unstale":"2024-03-24T12:13:39.697211+0000","last_undegraded":"2024-03-24T12:13:39.697211+0000","last_fullsized":"2024-03-24T12:13:39.697211+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:43.032403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:11:43.032403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":732,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032353+0000","last_change":"2024-03-24T12:10:25.718633+0000","last_active":"2024-03-24T12:13:42.032353+0000","last_peered":"2024-03-24T12:13:42.032353+0000","last_clean":"2024-03-24T12:13:42.032353+0000","last_became_active":"2024-03-24T12:08:36.165715+0000","last_became_peered":"2024-03-24T12:08:36.165715+0000","last_unstale":"2024-03-24T12:13:42.032353+0000","last_undegraded":"2024-03-24T12:13:42.032353+0000","last_fullsized":"2024-03-24T12:13:42.032353+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:25.718555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:25.718555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":909,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.690539+0000","last_change":"2024-03-24T12:12:58.434494+0000","last_active":"2024-03-24T12:13:39.690539+0000","last_peered":"2024-03-24T12:13:39.690539+0000","last_clean":"2024-03-24T12:13:39.690539+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:13:39.690539+0000","last_undegraded":"2024-03-24T12:13:39.690539+0000","last_fullsized":"2024-03-24T12:13:39.690539+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:58.434378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:12:58.434378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:07.808326+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"450'35","reported_seq":942,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.694713+0000","last_change":"2024-03-24T12:10:28.696002+0000","last_active":"2024-03-24T12:13:39.694713+0000","last_peered":"2024-03-24T12:13:39.694713+0000","last_clean":"2024-03-24T12:13:39.694713+0000","last_became_active":"2024-03-24T11:52:17.423720+0000","last_became_peered":"2024-03-24T11:52:17.423720+0000","last_unstale":"2024-03-24T12:13:39.694713+0000","last_undegraded":"2024-03-24T12:13:39.694713+0000","last_fullsized":"2024-03-24T12:13:39.694713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:28.695890+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:10:28.695890+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":783,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031881+0000","last_change":"2024-03-24T12:11:32.439386+0000","last_active":"2024-03-24T12:13:42.031881+0000","last_peered":"2024-03-24T12:13:42.031881+0000","last_clean":"2024-03-24T12:13:42.031881+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:13:42.031881+0000","last_undegraded":"2024-03-24T12:13:42.031881+0000","last_fullsized":"2024-03-24T12:13:42.031881+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:32.439301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:54.857860+0000","last_clean_scrub_stamp":"2024-03-24T12:11:32.439301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":1057,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032074+0000","last_change":"2024-03-24T12:10:15.754439+0000","last_active":"2024-03-24T12:13:42.032074+0000","last_peered":"2024-03-24T12:13:42.032074+0000","last_clean":"2024-03-24T12:13:42.032074+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:13:42.032074+0000","last_undegraded":"2024-03-24T12:13:42.032074+0000","last_fullsized":"2024-03-24T12:13:42.032074+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:10:15.754352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:15.754352+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":891,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.034721+0000","last_change":"2024-03-24T12:11:17.508810+0000","last_active":"2024-03-24T12:13:42.034721+0000","last_peered":"2024-03-24T12:13:42.034721+0000","last_clean":"2024-03-24T12:13:42.034721+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:13:42.034721+0000","last_undegraded":"2024-03-24T12:13:42.034721+0000","last_fullsized":"2024-03-24T12:13:42.034721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:17.508697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:17.508697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"450'21","reported_seq":817,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032520+0000","last_change":"2024-03-24T12:12:25.566925+0000","last_active":"2024-03-24T12:13:42.032520+0000","last_peered":"2024-03-24T12:13:42.032520+0000","last_clean":"2024-03-24T12:13:42.032520+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:13:42.032520+0000","last_undegraded":"2024-03-24T12:13:42.032520+0000","last_fullsized":"2024-03-24T12:13:42.032520+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:12:25.566647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:25.566647+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.376381+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":778,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035431+0000","last_change":"2024-03-24T12:10:34.022259+0000","last_active":"2024-03-24T12:13:42.035431+0000","last_peered":"2024-03-24T12:13:42.035431+0000","last_clean":"2024-03-24T12:13:42.035431+0000","last_became_active":"2024-03-24T12:10:30.443191+0000","last_became_peered":"2024-03-24T12:10:30.443191+0000","last_unstale":"2024-03-24T12:13:42.035431+0000","last_undegraded":"2024-03-24T12:13:42.035431+0000","last_fullsized":"2024-03-24T12:13:42.035431+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:34.022182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:34.022182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"301'9","reported_seq":1164,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695888+0000","last_change":"2024-03-24T12:10:19.664090+0000","last_active":"2024-03-24T12:13:39.695888+0000","last_peered":"2024-03-24T12:13:39.695888+0000","last_clean":"2024-03-24T12:13:39.695888+0000","last_became_active":"2024-03-24T12:05:44.741015+0000","last_became_peered":"2024-03-24T12:05:44.741015+0000","last_unstale":"2024-03-24T12:13:39.695888+0000","last_undegraded":"2024-03-24T12:13:39.695888+0000","last_fullsized":"2024-03-24T12:13:39.695888+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'9","last_scrub_stamp":"2024-03-24T12:10:19.664009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:19.664009+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":889,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697692+0000","last_change":"2024-03-24T12:10:17.566669+0000","last_active":"2024-03-24T12:13:39.697692+0000","last_peered":"2024-03-24T12:13:39.697692+0000","last_clean":"2024-03-24T12:13:39.697692+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:13:39.697692+0000","last_undegraded":"2024-03-24T12:13:39.697692+0000","last_fullsized":"2024-03-24T12:13:39.697692+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:17.566562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:10:17.566562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075344+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"450'21","reported_seq":686,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.024409+0000","last_change":"2024-03-24T12:13:22.025972+0000","last_active":"2024-03-24T12:13:42.024409+0000","last_peered":"2024-03-24T12:13:42.024409+0000","last_clean":"2024-03-24T12:13:42.024409+0000","last_became_active":"2024-03-24T12:13:22.025550+0000","last_became_peered":"2024-03-24T12:13:22.025550+0000","last_unstale":"2024-03-24T12:13:42.024409+0000","last_undegraded":"2024-03-24T12:13:42.024409+0000","last_fullsized":"2024-03-24T12:13:42.024409+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:11:28.054480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.054480+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":678,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695481+0000","last_change":"2024-03-24T12:13:22.021430+0000","last_active":"2024-03-24T12:13:39.695481+0000","last_peered":"2024-03-24T12:13:39.695481+0000","last_clean":"2024-03-24T12:13:39.695481+0000","last_became_active":"2024-03-24T12:13:22.021125+0000","last_became_peered":"2024-03-24T12:13:22.021125+0000","last_unstale":"2024-03-24T12:13:39.695481+0000","last_undegraded":"2024-03-24T12:13:39.695481+0000","last_fullsized":"2024-03-24T12:13:39.695481+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:15.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:12:15.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503483+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":894,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.694988+0000","last_change":"2024-03-24T12:11:37.185727+0000","last_active":"2024-03-24T12:13:39.694988+0000","last_peered":"2024-03-24T12:13:39.694988+0000","last_clean":"2024-03-24T12:13:39.694988+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:13:39.694988+0000","last_undegraded":"2024-03-24T12:13:39.694988+0000","last_fullsized":"2024-03-24T12:13:39.694988+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:37.185629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:37.185629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503202+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":901,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032250+0000","last_change":"2024-03-24T12:11:31.446015+0000","last_active":"2024-03-24T12:13:42.032250+0000","last_peered":"2024-03-24T12:13:42.032250+0000","last_clean":"2024-03-24T12:13:42.032250+0000","last_became_active":"2024-03-24T12:08:36.146995+0000","last_became_peered":"2024-03-24T12:08:36.146995+0000","last_unstale":"2024-03-24T12:13:42.032250+0000","last_undegraded":"2024-03-24T12:13:42.032250+0000","last_fullsized":"2024-03-24T12:13:42.032250+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:31.445925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:30.836662+0000","last_clean_scrub_stamp":"2024-03-24T12:11:31.445925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"450'27","reported_seq":932,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698327+0000","last_change":"2024-03-24T12:11:28.574992+0000","last_active":"2024-03-24T12:13:39.698327+0000","last_peered":"2024-03-24T12:13:39.698327+0000","last_clean":"2024-03-24T12:13:39.698327+0000","last_became_active":"2024-03-24T11:52:17.358870+0000","last_became_peered":"2024-03-24T11:52:17.358870+0000","last_unstale":"2024-03-24T12:13:39.698327+0000","last_undegraded":"2024-03-24T12:13:39.698327+0000","last_fullsized":"2024-03-24T12:13:39.698327+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:28.574841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:11:28.574841+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":937,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031122+0000","last_change":"2024-03-24T12:12:10.078427+0000","last_active":"2024-03-24T12:13:42.031122+0000","last_peered":"2024-03-24T12:13:42.031122+0000","last_clean":"2024-03-24T12:13:42.031122+0000","last_became_active":"2024-03-24T12:08:36.174093+0000","last_became_peered":"2024-03-24T12:08:36.174093+0000","last_unstale":"2024-03-24T12:13:42.031122+0000","last_undegraded":"2024-03-24T12:13:42.031122+0000","last_fullsized":"2024-03-24T12:13:42.031122+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:10.078142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:12:10.078142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":542,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031263+0000","last_change":"2024-03-24T12:12:54.662555+0000","last_active":"2024-03-24T12:13:42.031263+0000","last_peered":"2024-03-24T12:13:42.031263+0000","last_clean":"2024-03-24T12:13:42.031263+0000","last_became_active":"2024-03-24T12:08:36.166281+0000","last_became_peered":"2024-03-24T12:08:36.166281+0000","last_unstale":"2024-03-24T12:13:42.031263+0000","last_undegraded":"2024-03-24T12:13:42.031263+0000","last_fullsized":"2024-03-24T12:13:42.031263+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:54.662442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:12:54.662442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:08.641775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":885,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.696096+0000","last_change":"2024-03-24T12:10:10.700383+0000","last_active":"2024-03-24T12:13:39.696096+0000","last_peered":"2024-03-24T12:13:39.696096+0000","last_clean":"2024-03-24T12:13:39.696096+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:13:39.696096+0000","last_undegraded":"2024-03-24T12:13:39.696096+0000","last_fullsized":"2024-03-24T12:13:39.696096+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:10.700237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:10.700237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502785+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"450'14","reported_seq":920,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697392+0000","last_change":"2024-03-24T12:12:15.299189+0000","last_active":"2024-03-24T12:13:39.697392+0000","last_peered":"2024-03-24T12:13:39.697392+0000","last_clean":"2024-03-24T12:13:39.697392+0000","last_became_active":"2024-03-24T11:52:17.356354+0000","last_became_peered":"2024-03-24T11:52:17.356354+0000","last_unstale":"2024-03-24T12:13:39.697392+0000","last_undegraded":"2024-03-24T12:13:39.697392+0000","last_fullsized":"2024-03-24T12:13:39.697392+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-24T12:12:15.299057+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:12:15.299057+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503391+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":796,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032114+0000","last_change":"2024-03-24T12:10:12.784984+0000","last_active":"2024-03-24T12:13:42.032114+0000","last_peered":"2024-03-24T12:13:42.032114+0000","last_clean":"2024-03-24T12:13:42.032114+0000","last_became_active":"2024-03-24T12:08:36.166947+0000","last_became_peered":"2024-03-24T12:08:36.166947+0000","last_unstale":"2024-03-24T12:13:42.032114+0000","last_undegraded":"2024-03-24T12:13:42.032114+0000","last_fullsized":"2024-03-24T12:13:42.032114+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:12.784866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:12.784866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697049+0000","last_change":"2024-03-24T12:13:22.031556+0000","last_active":"2024-03-24T12:13:39.697049+0000","last_peered":"2024-03-24T12:13:39.697049+0000","last_clean":"2024-03-24T12:13:39.697049+0000","last_became_active":"2024-03-24T12:13:22.030820+0000","last_became_peered":"2024-03-24T12:13:22.030820+0000","last_unstale":"2024-03-24T12:13:39.697049+0000","last_undegraded":"2024-03-24T12:13:39.697049+0000","last_fullsized":"2024-03-24T12:13:39.697049+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:55.952950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:55.952950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:02.860728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":907,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.692851+0000","last_change":"2024-03-24T12:12:57.411454+0000","last_active":"2024-03-24T12:13:39.692851+0000","last_peered":"2024-03-24T12:13:39.692851+0000","last_clean":"2024-03-24T12:13:39.692851+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:13:39.692851+0000","last_undegraded":"2024-03-24T12:13:39.692851+0000","last_fullsized":"2024-03-24T12:13:39.692851+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:57.411342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:12:57.411342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:08.802903+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"450'27","reported_seq":927,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.701068+0000","last_change":"2024-03-24T12:10:22.696939+0000","last_active":"2024-03-24T12:13:39.701068+0000","last_peered":"2024-03-24T12:13:39.701068+0000","last_clean":"2024-03-24T12:13:39.701068+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:13:39.701068+0000","last_undegraded":"2024-03-24T12:13:39.701068+0000","last_fullsized":"2024-03-24T12:13:39.701068+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:10:22.696749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:10:22.696749+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":678,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.033175+0000","last_change":"2024-03-24T12:13:22.022525+0000","last_active":"2024-03-24T12:13:42.033175+0000","last_peered":"2024-03-24T12:13:42.033175+0000","last_clean":"2024-03-24T12:13:42.033175+0000","last_became_active":"2024-03-24T12:13:22.022201+0000","last_became_peered":"2024-03-24T12:13:22.022201+0000","last_unstale":"2024-03-24T12:13:42.033175+0000","last_undegraded":"2024-03-24T12:13:42.033175+0000","last_fullsized":"2024-03-24T12:13:42.033175+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:13.011469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_clean_scrub_stamp":"2024-03-24T12:12:13.011469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":911,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.692434+0000","last_change":"2024-03-24T12:11:45.406146+0000","last_active":"2024-03-24T12:13:39.692434+0000","last_peered":"2024-03-24T12:13:39.692434+0000","last_clean":"2024-03-24T12:13:39.692434+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:13:39.692434+0000","last_undegraded":"2024-03-24T12:13:39.692434+0000","last_fullsized":"2024-03-24T12:13:39.692434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:45.406037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:11:45.406037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":876,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695187+0000","last_change":"2024-03-24T12:10:13.863177+0000","last_active":"2024-03-24T12:13:39.695187+0000","last_peered":"2024-03-24T12:13:39.695187+0000","last_clean":"2024-03-24T12:13:39.695187+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:13:39.695187+0000","last_undegraded":"2024-03-24T12:13:39.695187+0000","last_fullsized":"2024-03-24T12:13:39.695187+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:13.862457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:10:13.862457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502847+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"477'12450","reported_seq":17294,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.761646+0000","last_change":"2024-03-24T12:13:23.047338+0000","last_active":"2024-03-24T12:13:41.761646+0000","last_peered":"2024-03-24T12:13:41.761646+0000","last_clean":"2024-03-24T12:13:41.761646+0000","last_became_active":"2024-03-24T12:13:23.024696+0000","last_became_peered":"2024-03-24T12:13:23.024696+0000","last_unstale":"2024-03-24T12:13:41.761646+0000","last_undegraded":"2024-03-24T12:13:41.761646+0000","last_fullsized":"2024-03-24T12:13:41.761646+0000","mapping_epoch":466,"log_start":"441'6500","ondisk_log_start":"441'6500","created":457,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":699,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92045312,"num_objects":1405,"num_object_clones":0,"num_object_copies":4215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6845,"num_write_kb":263968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2341,"num_bytes_recovered":153419776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"450'35","reported_seq":951,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035785+0000","last_change":"2024-03-24T12:10:35.061280+0000","last_active":"2024-03-24T12:13:42.035785+0000","last_peered":"2024-03-24T12:13:42.035785+0000","last_clean":"2024-03-24T12:13:42.035785+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:13:42.035785+0000","last_undegraded":"2024-03-24T12:13:42.035785+0000","last_fullsized":"2024-03-24T12:13:42.035785+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-24T12:10:35.061151+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-03-24T11:59:08.692644+0000","last_clean_scrub_stamp":"2024-03-24T12:10:35.061151+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":748,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031014+0000","last_change":"2024-03-24T12:10:24.690361+0000","last_active":"2024-03-24T12:13:42.031014+0000","last_peered":"2024-03-24T12:13:42.031014+0000","last_clean":"2024-03-24T12:13:42.031014+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:13:42.031014+0000","last_undegraded":"2024-03-24T12:13:42.031014+0000","last_fullsized":"2024-03-24T12:13:42.031014+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:24.690274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:24.690274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"469'10","reported_seq":652,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697929+0000","last_change":"2024-03-24T12:13:22.025913+0000","last_active":"2024-03-24T12:13:39.697929+0000","last_peered":"2024-03-24T12:13:39.697929+0000","last_clean":"2024-03-24T12:13:39.697929+0000","last_became_active":"2024-03-24T12:13:22.023981+0000","last_became_peered":"2024-03-24T12:13:22.023981+0000","last_unstale":"2024-03-24T12:13:39.697929+0000","last_undegraded":"2024-03-24T12:13:39.697929+0000","last_fullsized":"2024-03-24T12:13:39.697929+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9","last_scrub_stamp":"2024-03-24T12:11:30.060478+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:11:30.060478+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":925,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.690490+0000","last_change":"2024-03-24T12:12:59.360415+0000","last_active":"2024-03-24T12:13:39.690490+0000","last_peered":"2024-03-24T12:13:39.690490+0000","last_clean":"2024-03-24T12:13:39.690490+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:13:39.690490+0000","last_undegraded":"2024-03-24T12:13:39.690490+0000","last_fullsized":"2024-03-24T12:13:39.690490+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:12:59.360287+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:00:07.070250+0000","last_clean_scrub_stamp":"2024-03-24T12:12:59.360287+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:17.172111+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"478'12465","reported_seq":12618,"reported_epoch":478,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-24T12:13:42.235347+0000","last_change":"2024-03-24T12:13:34.274335+0000","last_active":"2024-03-24T12:13:42.235347+0000","last_peered":"2024-03-24T12:13:42.235347+0000","last_clean":"2024-03-24T12:13:02.124975+0000","last_became_active":"2024-03-24T12:13:04.243485+0000","last_became_peered":"2024-03-24T12:13:04.243485+0000","last_unstale":"2024-03-24T12:13:42.235347+0000","last_undegraded":"2024-03-24T12:13:42.235347+0000","last_fullsized":"2024-03-24T12:13:42.235347+0000","mapping_epoch":460,"log_start":"443'6800","ondisk_log_start":"443'6800","created":457,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88047616,"num_objects":1344,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4032,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6889,"num_write_kb":263424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":1344}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"450'27","reported_seq":945,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032793+0000","last_change":"2024-03-24T12:13:22.021820+0000","last_active":"2024-03-24T12:13:42.032793+0000","last_peered":"2024-03-24T12:13:42.032793+0000","last_clean":"2024-03-24T12:13:42.032793+0000","last_became_active":"2024-03-24T12:13:22.021362+0000","last_became_peered":"2024-03-24T12:13:22.021362+0000","last_unstale":"2024-03-24T12:13:42.032793+0000","last_undegraded":"2024-03-24T12:13:42.032793+0000","last_fullsized":"2024-03-24T12:13:42.032793+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-03-24T12:12:19.829011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.829011+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:49.397589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":888,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.696930+0000","last_change":"2024-03-24T12:10:23.652133+0000","last_active":"2024-03-24T12:13:39.696930+0000","last_peered":"2024-03-24T12:13:39.696930+0000","last_clean":"2024-03-24T12:13:39.696930+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:13:39.696930+0000","last_undegraded":"2024-03-24T12:13:39.696930+0000","last_fullsized":"2024-03-24T12:13:39.696930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:10:23.652045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.652045+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":911,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.035368+0000","last_change":"2024-03-24T12:12:19.386952+0000","last_active":"2024-03-24T12:13:42.035368+0000","last_peered":"2024-03-24T12:13:42.035368+0000","last_clean":"2024-03-24T12:13:42.035368+0000","last_became_active":"2024-03-24T12:05:44.742639+0000","last_became_peered":"2024-03-24T12:05:44.742639+0000","last_unstale":"2024-03-24T12:13:42.035368+0000","last_undegraded":"2024-03-24T12:13:42.035368+0000","last_fullsized":"2024-03-24T12:13:42.035368+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:19.386800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:12:19.386800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":496,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032227+0000","last_change":"2024-03-24T12:11:15.598297+0000","last_active":"2024-03-24T12:13:42.032227+0000","last_peered":"2024-03-24T12:13:42.032227+0000","last_clean":"2024-03-24T12:13:42.032227+0000","last_became_active":"2024-03-24T12:08:36.166973+0000","last_became_peered":"2024-03-24T12:08:36.166973+0000","last_unstale":"2024-03-24T12:13:42.032227+0000","last_undegraded":"2024-03-24T12:13:42.032227+0000","last_fullsized":"2024-03-24T12:13:42.032227+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:15.598041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:11:15.598041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"478'12855","reported_seq":22208,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:44.334713+0000","last_change":"2024-03-24T12:13:28.333627+0000","last_active":"2024-03-24T12:13:44.334713+0000","last_peered":"2024-03-24T12:13:44.334713+0000","last_clean":"2024-03-24T12:13:44.334713+0000","last_became_active":"2024-03-24T12:13:28.317443+0000","last_became_peered":"2024-03-24T12:13:28.317443+0000","last_unstale":"2024-03-24T12:13:44.334713+0000","last_undegraded":"2024-03-24T12:13:44.334713+0000","last_fullsized":"2024-03-24T12:13:44.334713+0000","mapping_epoch":471,"log_start":"441'6700","ondisk_log_start":"441'6700","created":457,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":5,"last_scrub":"456'11365","last_scrub_stamp":"2024-03-24T12:12:59.557519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:59.557519+0000","objects_scrubbed":5558,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:45.336760+0000","scrub_duration":1688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83427328,"num_objects":1273,"num_object_clones":0,"num_object_copies":3819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":269568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2283,"num_bytes_recovered":149553152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"450'46","reported_seq":864,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695707+0000","last_change":"2024-03-24T12:11:35.124433+0000","last_active":"2024-03-24T12:13:39.695707+0000","last_peered":"2024-03-24T12:13:39.695707+0000","last_clean":"2024-03-24T12:13:39.695707+0000","last_became_active":"2024-03-24T12:05:44.741830+0000","last_became_peered":"2024-03-24T12:05:44.741830+0000","last_unstale":"2024-03-24T12:13:39.695707+0000","last_undegraded":"2024-03-24T12:13:39.695707+0000","last_fullsized":"2024-03-24T12:13:39.695707+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-24T12:11:35.124291+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:11:35.124291+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503236+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":34,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697516+0000","last_change":"2024-03-24T12:13:22.053063+0000","last_active":"2024-03-24T12:13:39.697516+0000","last_peered":"2024-03-24T12:13:39.697516+0000","last_clean":"2024-03-24T12:13:39.697516+0000","last_became_active":"2024-03-24T12:13:22.052715+0000","last_became_peered":"2024-03-24T12:13:22.052715+0000","last_unstale":"2024-03-24T12:13:39.697516+0000","last_undegraded":"2024-03-24T12:13:39.697516+0000","last_fullsized":"2024-03-24T12:13:39.697516+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:15.939499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:11:15.939499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":653,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.699306+0000","last_change":"2024-03-24T12:12:29.766921+0000","last_active":"2024-03-24T12:13:39.699306+0000","last_peered":"2024-03-24T12:13:39.699306+0000","last_clean":"2024-03-24T12:13:39.699306+0000","last_became_active":"2024-03-24T11:59:19.097373+0000","last_became_peered":"2024-03-24T11:59:19.097373+0000","last_unstale":"2024-03-24T12:13:39.699306+0000","last_undegraded":"2024-03-24T12:13:39.699306+0000","last_fullsized":"2024-03-24T12:13:39.699306+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:29.766811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:12:29.766811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:47.373912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":902,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698664+0000","last_change":"2024-03-24T12:13:03.823186+0000","last_active":"2024-03-24T12:13:39.698664+0000","last_peered":"2024-03-24T12:13:39.698664+0000","last_clean":"2024-03-24T12:13:39.698664+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:13:39.698664+0000","last_undegraded":"2024-03-24T12:13:39.698664+0000","last_fullsized":"2024-03-24T12:13:39.698664+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:03.823064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:13:03.823064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:12.437246+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"477'12478","reported_seq":17371,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.788599+0000","last_change":"2024-03-24T12:13:22.052392+0000","last_active":"2024-03-24T12:13:41.788599+0000","last_peered":"2024-03-24T12:13:41.788599+0000","last_clean":"2024-03-24T12:13:41.788599+0000","last_became_active":"2024-03-24T12:13:22.024525+0000","last_became_peered":"2024-03-24T12:13:22.024525+0000","last_unstale":"2024-03-24T12:13:41.788599+0000","last_undegraded":"2024-03-24T12:13:41.788599+0000","last_fullsized":"2024-03-24T12:13:41.788599+0000","mapping_epoch":465,"log_start":"442'6800","ondisk_log_start":"442'6800","created":457,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.290500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91160576,"num_objects":1391,"num_object_clones":0,"num_object_copies":4173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6861,"num_write_kb":264064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2370,"num_bytes_recovered":155320320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"450'29","reported_seq":955,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695369+0000","last_change":"2024-03-24T12:11:41.236799+0000","last_active":"2024-03-24T12:13:39.695369+0000","last_peered":"2024-03-24T12:13:39.695369+0000","last_clean":"2024-03-24T12:13:39.695369+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:13:39.695369+0000","last_undegraded":"2024-03-24T12:13:39.695369+0000","last_fullsized":"2024-03-24T12:13:39.695369+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-03-24T12:11:41.236628+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:11:41.236628+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503340+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":901,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.694856+0000","last_change":"2024-03-24T12:11:20.004402+0000","last_active":"2024-03-24T12:13:39.694856+0000","last_peered":"2024-03-24T12:13:39.694856+0000","last_clean":"2024-03-24T12:13:39.694856+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:13:39.694856+0000","last_undegraded":"2024-03-24T12:13:39.694856+0000","last_fullsized":"2024-03-24T12:13:39.694856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:11:20.004266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:11:20.004266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503170+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":896,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698693+0000","last_change":"2024-03-24T12:11:23.602693+0000","last_active":"2024-03-24T12:13:39.698693+0000","last_peered":"2024-03-24T12:13:39.698693+0000","last_clean":"2024-03-24T12:13:39.698693+0000","last_became_active":"2024-03-24T12:11:16.022628+0000","last_became_peered":"2024-03-24T12:11:16.022628+0000","last_unstale":"2024-03-24T12:13:39.698693+0000","last_undegraded":"2024-03-24T12:13:39.698693+0000","last_fullsized":"2024-03-24T12:13:39.698693+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:23.602435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:11:23.602435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":897,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.695090+0000","last_change":"2024-03-24T12:12:56.504765+0000","last_active":"2024-03-24T12:13:39.695090+0000","last_peered":"2024-03-24T12:13:39.695090+0000","last_clean":"2024-03-24T12:13:39.695090+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:13:39.695090+0000","last_undegraded":"2024-03-24T12:13:39.695090+0000","last_fullsized":"2024-03-24T12:13:39.695090+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:56.504643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:12:56.504643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:22.096467+0000","scrub_duration":346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"478'12561","reported_seq":12800,"reported_epoch":478,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-24T12:13:44.345117+0000","last_change":"2024-03-24T12:13:34.238353+0000","last_active":"2024-03-24T12:13:44.345117+0000","last_peered":"2024-03-24T12:13:44.345117+0000","last_clean":"2024-03-24T12:13:02.104123+0000","last_became_active":"2024-03-24T12:13:04.208976+0000","last_became_peered":"2024-03-24T12:13:04.208976+0000","last_unstale":"2024-03-24T12:13:44.345117+0000","last_undegraded":"2024-03-24T12:13:44.345117+0000","last_fullsized":"2024-03-24T12:13:44.345117+0000","mapping_epoch":460,"log_start":"442'6800","ondisk_log_start":"442'6800","created":457,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":5,"last_scrub":"448'7845","last_scrub_stamp":"2024-03-24T12:12:14.190779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:12:14.190779+0000","objects_scrubbed":7762,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85000192,"num_objects":1297,"num_object_clones":0,"num_object_copies":3891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2594,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6945,"num_write_kb":263744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":1297}],"blocked_by":[],"up_primary":4,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"450'21","reported_seq":930,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697416+0000","last_change":"2024-03-24T12:12:09.683485+0000","last_active":"2024-03-24T12:13:39.697416+0000","last_peered":"2024-03-24T12:13:39.697416+0000","last_clean":"2024-03-24T12:13:39.697416+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:13:39.697416+0000","last_undegraded":"2024-03-24T12:13:39.697416+0000","last_fullsized":"2024-03-24T12:13:39.697416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'15","last_scrub_stamp":"2024-03-24T12:12:09.683324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:12:09.683324+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075946+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":930,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.690138+0000","last_change":"2024-03-24T12:11:41.354046+0000","last_active":"2024-03-24T12:13:39.690138+0000","last_peered":"2024-03-24T12:13:39.690138+0000","last_clean":"2024-03-24T12:13:39.690138+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:13:39.690138+0000","last_undegraded":"2024-03-24T12:13:39.690138+0000","last_fullsized":"2024-03-24T12:13:39.690138+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:41.353830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:11:41.353830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":809,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.031847+0000","last_change":"2024-03-24T12:10:23.722476+0000","last_active":"2024-03-24T12:13:42.031847+0000","last_peered":"2024-03-24T12:13:42.031847+0000","last_clean":"2024-03-24T12:13:42.031847+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:13:42.031847+0000","last_undegraded":"2024-03-24T12:13:42.031847+0000","last_fullsized":"2024-03-24T12:13:42.031847+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:23.722321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:10:23.722321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":891,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.032661+0000","last_change":"2024-03-24T12:12:27.562418+0000","last_active":"2024-03-24T12:13:42.032661+0000","last_peered":"2024-03-24T12:13:42.032661+0000","last_clean":"2024-03-24T12:13:42.032661+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:13:42.032661+0000","last_undegraded":"2024-03-24T12:13:42.032661+0000","last_fullsized":"2024-03-24T12:13:42.032661+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:27.562289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:12:27.562289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"477'12658","reported_seq":17885,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:41.809404+0000","last_change":"2024-03-24T12:13:22.059030+0000","last_active":"2024-03-24T12:13:41.809404+0000","last_peered":"2024-03-24T12:13:41.809404+0000","last_clean":"2024-03-24T12:13:41.809404+0000","last_became_active":"2024-03-24T12:13:22.029246+0000","last_became_peered":"2024-03-24T12:13:22.029246+0000","last_unstale":"2024-03-24T12:13:41.809404+0000","last_undegraded":"2024-03-24T12:13:41.809404+0000","last_fullsized":"2024-03-24T12:13:41.809404+0000","mapping_epoch":465,"log_start":"443'7000","ondisk_log_start":"443'7000","created":457,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"463'11824","last_scrub_stamp":"2024-03-24T12:13:16.602875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:13:16.602875+0000","objects_scrubbed":2247,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:24.628431+0000","scrub_duration":679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":1408,"num_object_clones":0,"num_object_copies":4224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":839,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"450'11","reported_seq":914,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.698169+0000","last_change":"2024-03-24T12:11:36.552488+0000","last_active":"2024-03-24T12:13:39.698169+0000","last_peered":"2024-03-24T12:13:39.698169+0000","last_clean":"2024-03-24T12:13:39.698169+0000","last_became_active":"2024-03-24T11:52:17.354699+0000","last_became_peered":"2024-03-24T11:52:17.354699+0000","last_unstale":"2024-03-24T12:13:39.698169+0000","last_undegraded":"2024-03-24T12:13:39.698169+0000","last_fullsized":"2024-03-24T12:13:39.698169+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9","last_scrub_stamp":"2024-03-24T12:11:36.552333+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:11:36.552333+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075859+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":926,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.690834+0000","last_change":"2024-03-24T12:11:13.237903+0000","last_active":"2024-03-24T12:13:39.690834+0000","last_peered":"2024-03-24T12:13:39.690834+0000","last_clean":"2024-03-24T12:13:39.690834+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:13:39.690834+0000","last_undegraded":"2024-03-24T12:13:39.690834+0000","last_fullsized":"2024-03-24T12:13:39.690834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:13.237829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:11:13.237829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":926,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.692635+0000","last_change":"2024-03-24T12:13:04.242037+0000","last_active":"2024-03-24T12:13:39.692635+0000","last_peered":"2024-03-24T12:13:39.692635+0000","last_clean":"2024-03-24T12:13:39.692635+0000","last_became_active":"2024-03-24T11:52:19.384011+0000","last_became_peered":"2024-03-24T11:52:19.384011+0000","last_unstale":"2024-03-24T12:13:39.692635+0000","last_undegraded":"2024-03-24T12:13:39.692635+0000","last_fullsized":"2024-03-24T12:13:39.692635+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:04.241918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.011992+0000","last_clean_scrub_stamp":"2024-03-24T12:13:04.241918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:13.508463+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":823,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.030837+0000","last_change":"2024-03-24T12:13:02.684962+0000","last_active":"2024-03-24T12:13:42.030837+0000","last_peered":"2024-03-24T12:13:42.030837+0000","last_clean":"2024-03-24T12:13:42.030837+0000","last_became_active":"2024-03-24T12:08:36.164166+0000","last_became_peered":"2024-03-24T12:08:36.164166+0000","last_unstale":"2024-03-24T12:13:42.030837+0000","last_undegraded":"2024-03-24T12:13:42.030837+0000","last_fullsized":"2024-03-24T12:13:42.030837+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:02.684869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.684869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:03.486873+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"478'12490","reported_seq":15124,"reported_epoch":478,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T12:13:42.266963+0000","last_change":"2024-03-24T12:13:34.302748+0000","last_active":"2024-03-24T12:13:42.266963+0000","last_peered":"2024-03-24T12:13:42.266963+0000","last_clean":"2024-03-24T12:13:02.095903+0000","last_became_active":"2024-03-24T12:13:04.268801+0000","last_became_peered":"2024-03-24T12:13:04.268801+0000","last_unstale":"2024-03-24T12:13:42.266963+0000","last_undegraded":"2024-03-24T12:13:42.266963+0000","last_fullsized":"2024-03-24T12:13:42.266963+0000","mapping_epoch":460,"log_start":"441'6300","ondisk_log_start":"441'6300","created":457,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":6190,"log_dups_size":0,"ondisk_log_size":6190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89817088,"num_objects":1371,"num_object_clones":0,"num_object_copies":4113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1248,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6902,"num_write_kb":264704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":828,"num_bytes_recovered":54263808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":1371}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"450'25","reported_seq":934,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.696683+0000","last_change":"2024-03-24T12:11:44.652906+0000","last_active":"2024-03-24T12:13:39.696683+0000","last_peered":"2024-03-24T12:13:39.696683+0000","last_clean":"2024-03-24T12:13:39.696683+0000","last_became_active":"2024-03-24T11:52:17.350640+0000","last_became_peered":"2024-03-24T11:52:17.350640+0000","last_unstale":"2024-03-24T12:13:39.696683+0000","last_undegraded":"2024-03-24T12:13:39.696683+0000","last_fullsized":"2024-03-24T12:13:39.696683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-24T12:11:44.652731+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:11:44.652731+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:41.075916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":906,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.696492+0000","last_change":"2024-03-24T12:12:57.187652+0000","last_active":"2024-03-24T12:13:39.696492+0000","last_peered":"2024-03-24T12:13:39.696492+0000","last_clean":"2024-03-24T12:13:39.696492+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:13:39.696492+0000","last_undegraded":"2024-03-24T12:13:39.696492+0000","last_fullsized":"2024-03-24T12:13:39.696492+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:57.187540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:06.381099+0000","last_clean_scrub_stamp":"2024-03-24T12:12:57.187540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:05.471402+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":322,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.030982+0000","last_change":"2024-03-24T12:13:22.016335+0000","last_active":"2024-03-24T12:13:42.030982+0000","last_peered":"2024-03-24T12:13:42.030982+0000","last_clean":"2024-03-24T12:13:42.030982+0000","last_became_active":"2024-03-24T12:13:22.015998+0000","last_became_peered":"2024-03-24T12:13:22.015998+0000","last_unstale":"2024-03-24T12:13:42.030982+0000","last_undegraded":"2024-03-24T12:13:42.030982+0000","last_fullsized":"2024-03-24T12:13:42.030982+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:22.837495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:12:22.837495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:50.332357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":897,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.697311+0000","last_change":"2024-03-24T12:13:02.096236+0000","last_active":"2024-03-24T12:13:39.697311+0000","last_peered":"2024-03-24T12:13:39.697311+0000","last_clean":"2024-03-24T12:13:39.697311+0000","last_became_active":"2024-03-24T12:08:36.165334+0000","last_became_peered":"2024-03-24T12:08:36.165334+0000","last_unstale":"2024-03-24T12:13:39.697311+0000","last_undegraded":"2024-03-24T12:13:39.697311+0000","last_fullsized":"2024-03-24T12:13:39.697311+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:13:02.096047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:13:02.096047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:22.825181+0000","scrub_duration":308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"478'12345","reported_seq":12496,"reported_epoch":478,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-24T12:13:43.873493+0000","last_change":"2024-03-24T12:13:34.313048+0000","last_active":"2024-03-24T12:13:43.873493+0000","last_peered":"2024-03-24T12:13:43.873493+0000","last_clean":"2024-03-24T12:13:02.135574+0000","last_became_active":"2024-03-24T12:13:04.276138+0000","last_became_peered":"2024-03-24T12:13:04.276138+0000","last_unstale":"2024-03-24T12:13:43.873493+0000","last_undegraded":"2024-03-24T12:13:43.873493+0000","last_fullsized":"2024-03-24T12:13:43.873493+0000","mapping_epoch":460,"log_start":"441'6000","ondisk_log_start":"441'6000","created":457,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:10:41.539388+0000","last_clean_scrub_stamp":"2024-03-24T12:10:41.539388+0000","objects_scrubbed":0,"log_size":6345,"log_dups_size":0,"ondisk_log_size":6345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88899584,"num_objects":1357,"num_object_clones":0,"num_object_copies":4071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4071,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6837,"num_write_kb":262176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1357}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"450'46","reported_seq":982,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.701170+0000","last_change":"2024-03-24T12:08:47.101220+0000","last_active":"2024-03-24T12:13:39.701170+0000","last_peered":"2024-03-24T12:13:39.701170+0000","last_clean":"2024-03-24T12:13:39.701170+0000","last_became_active":"2024-03-24T12:08:36.154117+0000","last_became_peered":"2024-03-24T12:08:36.154117+0000","last_unstale":"2024-03-24T12:13:39.701170+0000","last_undegraded":"2024-03-24T12:13:39.701170+0000","last_fullsized":"2024-03-24T12:13:39.701170+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-03-24T12:08:47.101082+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:08:47.101082+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.502657+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":905,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-24T12:13:42.034346+0000","last_change":"2024-03-24T12:12:12.477784+0000","last_active":"2024-03-24T12:13:42.034346+0000","last_peered":"2024-03-24T12:13:42.034346+0000","last_clean":"2024-03-24T12:13:42.034346+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:13:42.034346+0000","last_undegraded":"2024-03-24T12:13:42.034346+0000","last_fullsized":"2024-03-24T12:13:42.034346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:12.477654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:12:12.477654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":366,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.694495+0000","last_change":"2024-03-24T12:12:27.315018+0000","last_active":"2024-03-24T12:13:39.694495+0000","last_peered":"2024-03-24T12:13:39.694495+0000","last_clean":"2024-03-24T12:13:39.694495+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:13:39.694495+0000","last_undegraded":"2024-03-24T12:13:39.694495+0000","last_fullsized":"2024-03-24T12:13:39.694495+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:12:27.314894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:12:27.314894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:52.284429+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":883,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-24T12:13:39.694517+0000","last_change":"2024-03-24T12:11:44.456204+0000","last_active":"2024-03-24T12:13:39.694517+0000","last_peered":"2024-03-24T12:13:39.694517+0000","last_clean":"2024-03-24T12:13:39.694517+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:13:39.694517+0000","last_undegraded":"2024-03-24T12:13:39.694517+0000","last_fullsized":"2024-03-24T12:13:39.694517+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:11:44.455779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:11:44.455779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:44.503307+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":2859565080,"num_objects":43637,"num_object_clones":0,"num_object_copies":130911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11945,"num_objects_unfound":0,"num_objects_dirty":43637,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151033,"num_write_kb":5278433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10103,"num_bytes_recovered":661979136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9282465792,"data_stored":9282453576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148497,"ondisk_log_size":148497,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":706653,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":58,"num_write":35,"num_write_kb":476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":676533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":390,"ondisk_log_size":390,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":477,"num_read_kb":472,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1547,"num_read_kb":1304,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":1079,"ondisk_log_size":1079,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005317,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2283244,"kb_used_data":1421856,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":91461908,"statfs":{"total":95995035648,"available":93656993792,"internally_reserved":0,"allocated":1455980544,"data_stored":1455486522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":882047434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332364,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2105088,"kb_used_data":1289524,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91640064,"statfs":{"total":95995035648,"available":93839425536,"internally_reserved":0,"allocated":1320472576,"data_stored":1319958384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":835123654},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575140,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2685192,"kb_used_data":1761336,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":91059960,"statfs":{"total":95995035648,"available":93245399040,"internally_reserved":0,"allocated":1803608064,"data_stored":1803096111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1941824,"kb_used_data":1026292,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":91803328,"statfs":{"total":95995035648,"available":94006607872,"internally_reserved":0,"allocated":1050923008,"data_stored":1050415096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411616,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1054300,"kb_used_data":431696,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92690852,"statfs":{"total":95995035648,"available":94915432448,"internally_reserved":0,"allocated":442056704,"data_stored":441585561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117274,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2205956,"kb_used_data":1282108,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":91539196,"statfs":{"total":95995035648,"available":93736136704,"internally_reserved":0,"allocated":1312878592,"data_stored":1312356689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":946010567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921045,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2017720,"kb_used_data":1184112,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":91727432,"statfs":{"total":95995035648,"available":93928890368,"internally_reserved":0,"allocated":1212530688,"data_stored":1212019738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":853604805},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455551,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1159240,"kb_used_data":779708,"kb_used_omap":7,"kb_used_meta":379512,"kb_avail":92585912,"statfs":{"total":95995035648,"available":94807973888,"internally_reserved":0,"allocated":798420992,"data_stored":797905733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":388620400},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1037697024,"data_stored":1037697024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1798180864,"data_stored":1798176792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":439484416,"data_stored":439484416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1297743872,"data_stored":1297743872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1186861056,"data_stored":1186856984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":775028736,"data_stored":775028736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1312428032,"data_stored":1312423960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1435041792,"data_stored":1435041792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:13:54.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 26 2024-03-24T12:13:54.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:54.732 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:54.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:54.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:13:54.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:54.875 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:54.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:54.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:55.013 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:55.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:55.114 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:55.150 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:55.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:55.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:55.290 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:55.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:55.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:55.428 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:55.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:55.477 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:55 smithi064 ceph-mon[30223]: pgmap v1033: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 1.39k op/s; 8729/100029 objects misplaced (8.726%); 4.8 MiB/s, 77 objects/s recovering 2024-03-24T12:13:55.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:55 smithi064 ceph-mon[42243]: pgmap v1033: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 1.39k op/s; 8729/100029 objects misplaced (8.726%); 4.8 MiB/s, 77 objects/s recovering 2024-03-24T12:13:55.529 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:55.555 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:55.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:55 smithi193 ceph-mon[33593]: pgmap v1033: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 1.39k op/s; 8729/100029 objects misplaced (8.726%); 4.8 MiB/s, 77 objects/s recovering 2024-03-24T12:13:55.632 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:55.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:55.669 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:55.689 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:13:55.711 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:55.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:55.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:55.859 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:55.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:55.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:55.999 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:55.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:56.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:56.143 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:56.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:56.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:56.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:56.280 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:56.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:56.381 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:56.415 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:56.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:56.517 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:56.553 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:56.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:56.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:56.696 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:56.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:56.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:56.839 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:56.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:56.940 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:56.980 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:56.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:57.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:57.123 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:57.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:57.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:57.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:57.261 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:57.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:57.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:57 smithi064 ceph-mon[30223]: pgmap v1034: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 815 op/s; 7813/96786 objects misplaced (8.072%); 743 KiB/s, 11 objects/s recovering 2024-03-24T12:13:57.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:57 smithi064 ceph-mon[42243]: pgmap v1034: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 815 op/s; 7813/96786 objects misplaced (8.072%); 743 KiB/s, 11 objects/s recovering 2024-03-24T12:13:57.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:57.400 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:57.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:57.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:57.538 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:57.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:57.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:13:57.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:57 smithi193 ceph-mon[33593]: pgmap v1034: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 815 op/s; 7813/96786 objects misplaced (8.072%); 743 KiB/s, 11 objects/s recovering 2024-03-24T12:13:57.621 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:13:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:13:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:57.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:57.679 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:57.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:57.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:57.816 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:57.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:57.917 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:57.955 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:57.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:58.057 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:58.093 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:58.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:58.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:58.232 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:58.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:58.283 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:58.284 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:58.333 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:13:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:58.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:13:58 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:13:58.369 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:58.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:13:58.470 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:58.508 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:58.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:13:58.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:58.645 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:58.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:58.746 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:13:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:58.783 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:58.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:13:58.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:58.933 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:58.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:59.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:13:59.073 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:59.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:59.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:13:59.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:13:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:59.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:59.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:13:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:59 smithi064 ceph-mon[30223]: pgmap v1035: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 1.28k op/s; 7255/81876 objects misplaced (8.861%); 699 KiB/s, 10 objects/s recovering 2024-03-24T12:13:59.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:13:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:59 smithi064 ceph-mon[42243]: pgmap v1035: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 1.28k op/s; 7255/81876 objects misplaced (8.861%); 699 KiB/s, 10 objects/s recovering 2024-03-24T12:13:59.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:13:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:59.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:59.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:59.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:13:59.491 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:59.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:13:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:59 smithi193 ceph-mon[33593]: pgmap v1035: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 1.28k op/s; 7255/81876 objects misplaced (8.861%); 699 KiB/s, 10 objects/s recovering 2024-03-24T12:13:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:13:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:59.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:13:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:59.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:13:59.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:13:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:13:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:59.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:13:59.892 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:13:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:13:59.928 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:13:59.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:00.030 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:00.070 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:00.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:00.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:00.220 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:00.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:00.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:00.367 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:00.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:00.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:00.514 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:00.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:00.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:00.651 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:00.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:00.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:00.789 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:00.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:00.890 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:00.924 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:00.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:01.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:01.040 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:14:01.040 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:14:01.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:243115 11.e e482/417 11:7f53428d:::benchmark_data_smithi064_206665_object110982:head v 482'15035, mlcod=482'15032)", 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:14:01.033176+0000", 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.003417144, 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034473500000000001, 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:01.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.033176+0000", 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.033176+0000", 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.033177+0000", 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7699999999999992e-07 2024-03-24T12:14:01.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.033186+0000", 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6090000000000001e-06 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.033195+0000", 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4440000000000008e-06 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.033201+0000", 2024-03-24T12:14:01.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6329999999999999e-06 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.033296+0000", 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4752000000000004e-05 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.033316+0000", 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9874e-05 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:01.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:01.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:01.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:243119 11.4 e482/457 11:2231bde0:::benchmark_data_smithi064_206665_object110986:head v 482'13377, mlcod=482'13375)", 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:14:01.034696+0000", 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0018975960000000001, 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019960249999999998, 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.034696+0000", 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.034696+0000", 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.034697+0000", 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.308e-06 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.034705+0000", 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7479999999999993e-06 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.034715+0000", 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5990000000000008e-06 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.034720+0000", 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2140000000000004e-06 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.035152+0000", 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043166800000000001 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.035174+0000", 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2362e-05 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:243120 11.7 e482/457)", 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:14:01.036444+0000", 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00014923099999999999, 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030571400000000001, 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-03-24T12:14:01.049 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.036444+0000", 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.036444+0000", 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.036446+0000", 2024-03-24T12:14:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-06 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.036454+0000", 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8450000000000003e-06 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.036464+0000", 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0019e-05 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:14:01.036471+0000", 2024-03-24T12:14:01.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.097e-06 2024-03-24T12:14:01.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:01.052 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:01.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:01.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:01.052 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-03-24T12:14:01.052 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 3 2024-03-24T12:14:01.052 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:14:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:01.057 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:01.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:01.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:01.199 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:01.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:01.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:01 smithi064 ceph-mon[30223]: pgmap v1036: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.16k op/s; 6240/78192 objects misplaced (7.980%); 595 KiB/s, 9 objects/s recovering 2024-03-24T12:14:01.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:01 smithi064 ceph-mon[42243]: pgmap v1036: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.16k op/s; 6240/78192 objects misplaced (7.980%); 595 KiB/s, 9 objects/s recovering 2024-03-24T12:14:01.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:01.339 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:01.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:01.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:01.475 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:01.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:01.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:01 smithi193 ceph-mon[33593]: pgmap v1036: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.16k op/s; 6240/78192 objects misplaced (7.980%); 595 KiB/s, 9 objects/s recovering 2024-03-24T12:14:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:01.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:01.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:01.750 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:01.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:01.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:01.881 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:01.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:01.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:01.983 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:02.020 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:02.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:02.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:02.163 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:02.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:02.166 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 11 pgp_num to 26 2024-03-24T12:14:02.166 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:02 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3857396571' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-24T12:14:02.166 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:02 smithi064 ceph-mon[30223]: from='client.36047 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-24T12:14:02.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:02 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3857396571' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-24T12:14:02.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:02 smithi064 ceph-mon[42243]: from='client.36047 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-24T12:14:02.264 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:02.300 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:02.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:02.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:02.439 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:02.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:02.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:02 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3857396571' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-24T12:14:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:02 smithi193 ceph-mon[33593]: from='client.36047 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-24T12:14:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:02.589 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:02.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:02.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:14:02.690 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:02.730 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:02.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:02.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:02.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:03.005 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:03.040 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:03.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:03.142 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:03.177 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:03.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:03.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:03.179 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:14:03.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[30223]: pgmap v1037: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 890 B/s rd, 0 B/s wr, 1.01k op/s; 6240/78192 objects misplaced (7.980%); 517 KiB/s, 8 objects/s recovering 2024-03-24T12:14:03.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[30223]: from='client.36047 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-24T12:14:03.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[30223]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T12:14:03.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/906435862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:03.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[30223]: from='client.26017 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:03.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[42243]: pgmap v1037: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 890 B/s rd, 0 B/s wr, 1.01k op/s; 6240/78192 objects misplaced (7.980%); 517 KiB/s, 8 objects/s recovering 2024-03-24T12:14:03.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[42243]: from='client.36047 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-24T12:14:03.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[42243]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T12:14:03.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/906435862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:03.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:03 smithi064 ceph-mon[42243]: from='client.26017 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:03.279 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:03.322 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:03.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:03.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:03.425 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:03.464 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:03.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:03.565 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:03 smithi193 ceph-mon[33593]: pgmap v1037: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 890 B/s rd, 0 B/s wr, 1.01k op/s; 6240/78192 objects misplaced (7.980%); 517 KiB/s, 8 objects/s recovering 2024-03-24T12:14:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:03 smithi193 ceph-mon[33593]: from='client.36047 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-24T12:14:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:03 smithi193 ceph-mon[33593]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T12:14:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:03 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/906435862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:03 smithi193 ceph-mon[33593]: from='client.26017 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:03.592 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:03.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:03.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:03.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:03.738 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:03.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:03.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:03.888 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:03.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:03.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:04.039 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:04.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:04.142 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:04.174 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:04.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:04.276 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:04 smithi064 ceph-mon[30223]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:14:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:04 smithi064 ceph-mon[30223]: from='client.26017 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:14:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:04 smithi064 ceph-mon[30223]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T12:14:04.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:04 smithi064 ceph-mon[42243]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:14:04.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:04 smithi064 ceph-mon[42243]: from='client.26017 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:14:04.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:04 smithi064 ceph-mon[42243]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T12:14:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:04.311 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:04.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:04.412 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:04.449 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:04.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:04.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:04 smithi193 ceph-mon[33593]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:14:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:04 smithi193 ceph-mon[33593]: from='client.26017 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:14:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:04 smithi193 ceph-mon[33593]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T12:14:04.591 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:04.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:04.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:04.727 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:04.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:04.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:04.871 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:04.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:04.972 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:05.020 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:05.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:05.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:05.163 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:05.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:05.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:05.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:05 smithi064 ceph-mon[30223]: pgmap v1040: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s; 5670/62478 objects misplaced (9.075%) 2024-03-24T12:14:05.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:05 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T12:14:05.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:05 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T12:14:05.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:05 smithi064 ceph-mon[42243]: pgmap v1040: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s; 5670/62478 objects misplaced (9.075%) 2024-03-24T12:14:05.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:05 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T12:14:05.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:05 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T12:14:05.303 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:05.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:05 smithi193 ceph-mon[33593]: pgmap v1040: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s; 5670/62478 objects misplaced (9.075%) 2024-03-24T12:14:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:05 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T12:14:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:05 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T12:14:05.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:05.445 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:05.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:05.548 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:05.584 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:05.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:05.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:05.722 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:05.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:05.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:05.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:05.969 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:06.010 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:06.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:06.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:14:06.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:06.147 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:06.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:06.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:06.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:06 smithi064 ceph-mon[30223]: 1.7 scrub starts 2024-03-24T12:14:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:06 smithi064 ceph-mon[30223]: 1.7 scrub ok 2024-03-24T12:14:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:06 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T12:14:06.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:06 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T12:14:06.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:06 smithi064 ceph-mon[42243]: 1.7 scrub starts 2024-03-24T12:14:06.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:06 smithi064 ceph-mon[42243]: 1.7 scrub ok 2024-03-24T12:14:06.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:06 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T12:14:06.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:06 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T12:14:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:06.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:06.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:06 smithi193 ceph-mon[33593]: 1.7 scrub starts 2024-03-24T12:14:06.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:06 smithi193 ceph-mon[33593]: 1.7 scrub ok 2024-03-24T12:14:06.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:06 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T12:14:06.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:06 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T12:14:06.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:06.465 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:06.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:06.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:06.611 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:06.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:06.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:06.756 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:06.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:06.857 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:06.888 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:06.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:06.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:14:06.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:14:06.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:14:06.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:14:06.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:14:06.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:07.014 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:14:07.021 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:07.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:07.063 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:14:07.114 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:14:07.122 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:07.125 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:14:07.125 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:14:07.125 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:14:07.125 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:14:07.126 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:14:07.126 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:14:07.126 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:14:07.126 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:14:07.155 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:07.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:07.257 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:07.288 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:07.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:07.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:07.414 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:07.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:07.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:07.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:14:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:07.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[30223]: pgmap v1041: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 962 op/s; 4641/58797 objects misplaced (7.893%); 96 KiB/s, 1 objects/s recovering 2024-03-24T12:14:07.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:14:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:14:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:14:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3234093927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[42243]: pgmap v1041: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 962 op/s; 4641/58797 objects misplaced (7.893%); 96 KiB/s, 1 objects/s recovering 2024-03-24T12:14:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:14:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:14:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:14:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:07 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3234093927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:07.553 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:07.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:07.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:07 smithi193 ceph-mon[33593]: pgmap v1041: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 962 op/s; 4641/58797 objects misplaced (7.893%); 96 KiB/s, 1 objects/s recovering 2024-03-24T12:14:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:07 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:14:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:07 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:14:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:07 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:14:07.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:07 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3234093927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:07.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:07.697 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:07.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:07.798 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:07.840 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:07.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:07.941 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:07.987 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:07.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:08.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:08.130 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:08.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:08.232 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:08.257 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:08.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:08.321 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:14:08.321 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:14:08.321 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-24T12:14:08.322 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T12:14:08.322 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T12:14:08.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-24T12:14:08.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:14:08 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:14:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:08 smithi193 ceph-mon[33593]: 11.0 scrub starts 2024-03-24T12:14:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:08 smithi193 ceph-mon[33593]: 11.0 scrub ok 2024-03-24T12:14:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:08 smithi193 ceph-mon[33593]: 11.12 scrub starts 2024-03-24T12:14:08.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:08 smithi193 ceph-mon[33593]: 11.12 scrub ok 2024-03-24T12:14:08.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:08.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:08.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:08.538 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:08.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:08.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:08.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:08 smithi064 ceph-mon[30223]: 11.0 scrub starts 2024-03-24T12:14:08.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:08 smithi064 ceph-mon[30223]: 11.0 scrub ok 2024-03-24T12:14:08.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:08 smithi064 ceph-mon[30223]: 11.12 scrub starts 2024-03-24T12:14:08.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:08 smithi064 ceph-mon[30223]: 11.12 scrub ok 2024-03-24T12:14:08.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:08 smithi064 ceph-mon[42243]: 11.0 scrub starts 2024-03-24T12:14:08.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:08 smithi064 ceph-mon[42243]: 11.0 scrub ok 2024-03-24T12:14:08.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:08 smithi064 ceph-mon[42243]: 11.12 scrub starts 2024-03-24T12:14:08.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:08 smithi064 ceph-mon[42243]: 11.12 scrub ok 2024-03-24T12:14:08.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:08.683 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:08.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:08.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:08.824 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:08.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:08.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:08.959 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:08.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:09.060 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:09.105 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:09.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:09.206 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:09.238 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:09.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:09.339 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:09.377 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:09.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:09.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:09.510 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:09.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:09.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[30223]: pgmap v1042: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 895 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.44k op/s; 3345/43533 objects misplaced (7.684%); 2.1 MiB/s, 34 objects/s recovering 2024-03-24T12:14:09.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[30223]: 11.11 scrub starts 2024-03-24T12:14:09.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[30223]: 11.11 scrub ok 2024-03-24T12:14:09.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:09.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[30223]: 11.17 scrub starts 2024-03-24T12:14:09.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[30223]: 11.17 scrub ok 2024-03-24T12:14:09.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[42243]: pgmap v1042: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 895 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.44k op/s; 3345/43533 objects misplaced (7.684%); 2.1 MiB/s, 34 objects/s recovering 2024-03-24T12:14:09.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[42243]: 11.11 scrub starts 2024-03-24T12:14:09.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[42243]: 11.11 scrub ok 2024-03-24T12:14:09.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:09.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[42243]: 11.17 scrub starts 2024-03-24T12:14:09.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:09 smithi064 ceph-mon[42243]: 11.17 scrub ok 2024-03-24T12:14:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:09 smithi193 ceph-mon[33593]: pgmap v1042: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 895 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.44k op/s; 3345/43533 objects misplaced (7.684%); 2.1 MiB/s, 34 objects/s recovering 2024-03-24T12:14:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:09 smithi193 ceph-mon[33593]: 11.11 scrub starts 2024-03-24T12:14:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:09 smithi193 ceph-mon[33593]: 11.11 scrub ok 2024-03-24T12:14:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:09 smithi193 ceph-mon[33593]: 11.17 scrub starts 2024-03-24T12:14:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:09 smithi193 ceph-mon[33593]: 11.17 scrub ok 2024-03-24T12:14:09.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:09.642 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:09.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:09.744 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:09.768 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:09.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:09.870 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:09.894 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:09.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:09.922 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:10.002 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:10.025 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:10.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:10.127 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:10.152 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:10.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:10.253 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:10.285 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:10.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:10.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:10.418 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:10.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:10.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[42243]: 3.2 scrub starts 2024-03-24T12:14:10.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[42243]: 3.2 scrub ok 2024-03-24T12:14:10.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[42243]: 6.17 scrub starts 2024-03-24T12:14:10.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[42243]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T12:14:10.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[42243]: 6.17 scrub ok 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[42243]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[30223]: 3.2 scrub starts 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[30223]: 3.2 scrub ok 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[30223]: 6.17 scrub starts 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[30223]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[30223]: 6.17 scrub ok 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T12:14:10.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:10 smithi064 ceph-mon[30223]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T12:14:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:10.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:10.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:10 smithi193 ceph-mon[33593]: 3.2 scrub starts 2024-03-24T12:14:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:10 smithi193 ceph-mon[33593]: 3.2 scrub ok 2024-03-24T12:14:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:10 smithi193 ceph-mon[33593]: 6.17 scrub starts 2024-03-24T12:14:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:10 smithi193 ceph-mon[33593]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T12:14:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:10 smithi193 ceph-mon[33593]: 6.17 scrub ok 2024-03-24T12:14:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:10 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T12:14:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:10 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T12:14:10.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:10 smithi193 ceph-mon[33593]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T12:14:10.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:10.674 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:10.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:10.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:10.799 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:10.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:10.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:10.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:10.927 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:10.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:11.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:11.052 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:11.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:11.134 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:14:11.134 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:14:11.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:11.179 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:11.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:11.281 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[30223]: pgmap v1044: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 164 active+clean; 837 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.08k op/s; 2811/40740 objects misplaced (6.900%); 2.3 MiB/s, 36 objects/s recovering 2024-03-24T12:14:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[30223]: 1.5 scrub starts 2024-03-24T12:14:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[30223]: 1.5 scrub ok 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[30223]: 1.8 scrub starts 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[30223]: 1.8 scrub ok 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[30223]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[42243]: pgmap v1044: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 164 active+clean; 837 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.08k op/s; 2811/40740 objects misplaced (6.900%); 2.3 MiB/s, 36 objects/s recovering 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[42243]: 1.5 scrub starts 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[42243]: 1.5 scrub ok 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[42243]: 1.8 scrub starts 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[42243]: 1.8 scrub ok 2024-03-24T12:14:11.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:11 smithi064 ceph-mon[42243]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T12:14:11.319 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:11.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:11.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:11.459 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:11.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:11.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:11 smithi193 ceph-mon[33593]: pgmap v1044: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 164 active+clean; 837 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.08k op/s; 2811/40740 objects misplaced (6.900%); 2.3 MiB/s, 36 objects/s recovering 2024-03-24T12:14:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:11 smithi193 ceph-mon[33593]: 1.5 scrub starts 2024-03-24T12:14:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:11 smithi193 ceph-mon[33593]: 1.5 scrub ok 2024-03-24T12:14:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:11 smithi193 ceph-mon[33593]: 1.8 scrub starts 2024-03-24T12:14:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:11 smithi193 ceph-mon[33593]: 1.8 scrub ok 2024-03-24T12:14:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:11 smithi193 ceph-mon[33593]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T12:14:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:11.605 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:11.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:11.707 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:11.733 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:11.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:11.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:11.875 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:11.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:11.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:12.017 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:12.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:12.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:12.165 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:12.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:12.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:12.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:12.302 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:12.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:12.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:12.435 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:12.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:12.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:14:12.506 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:14:12.506 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: 6.a scrub starts 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: 6.a scrub ok 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[30223]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:14:12.507 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T12:14:12.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T12:14:12.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: 6.a scrub starts 2024-03-24T12:14:12.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: 6.a scrub ok 2024-03-24T12:14:12.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:14:12.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:14:12.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:14:12.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:12 smithi064 ceph-mon[42243]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T12:14:12.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:12.583 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:12.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: 6.a scrub starts 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: 6.a scrub ok 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:14:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:12 smithi193 ceph-mon[33593]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T12:14:12.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:12.715 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:12.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:12.816 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:12.851 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:12.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:12.953 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:12.987 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:12.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:13.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:13.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:13.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:13.263 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:13.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:13.365 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:13.400 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:13.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:13.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:14:13.502 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: 11.10 scrub starts 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: pgmap v1047: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 164 active+clean; 837 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2811/40740 objects misplaced (6.900%); 2.7 MiB/s, 43 objects/s recovering 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: 11.10 scrub ok 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: 4.19 scrub starts 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: 4.19 scrub ok 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: 11.15 scrub starts 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: 11.15 scrub ok 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:14:13.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: 11.10 scrub starts 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: pgmap v1047: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 164 active+clean; 837 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2811/40740 objects misplaced (6.900%); 2.7 MiB/s, 43 objects/s recovering 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: 11.10 scrub ok 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: 4.19 scrub starts 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: 4.19 scrub ok 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: 11.15 scrub starts 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: 11.15 scrub ok 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:14:13.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:13 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:13.538 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:13.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: 11.10 scrub starts 2024-03-24T12:14:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: pgmap v1047: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 164 active+clean; 837 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2811/40740 objects misplaced (6.900%); 2.7 MiB/s, 43 objects/s recovering 2024-03-24T12:14:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: 11.10 scrub ok 2024-03-24T12:14:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: 4.19 scrub starts 2024-03-24T12:14:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: 4.19 scrub ok 2024-03-24T12:14:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: 11.15 scrub starts 2024-03-24T12:14:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: 11.15 scrub ok 2024-03-24T12:14:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:14:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:14:13.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:13 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:13.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:13.695 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:13.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:13.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:13.844 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:13.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:13.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:14.005 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:14.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:14.108 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:14.144 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:14.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:14.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:14.283 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:14.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:14.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:14.424 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:14.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:14.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:14 smithi064 ceph-mon[30223]: 11.13 scrub starts 2024-03-24T12:14:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:14 smithi064 ceph-mon[30223]: 11.13 scrub ok 2024-03-24T12:14:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:14 smithi064 ceph-mon[42243]: 11.13 scrub starts 2024-03-24T12:14:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:14 smithi064 ceph-mon[42243]: 11.13 scrub ok 2024-03-24T12:14:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:14.567 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:14.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:14 smithi193 ceph-mon[33593]: 11.13 scrub starts 2024-03-24T12:14:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:14 smithi193 ceph-mon[33593]: 11.13 scrub ok 2024-03-24T12:14:14.668 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:14.711 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:14.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:14.812 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:14.848 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:14.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:14.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:14.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:15.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:15.132 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:15.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:15.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:15.269 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:15.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:15.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:15.401 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:15.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:15.502 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:15.542 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:15.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[30223]: 1.6 scrub starts 2024-03-24T12:14:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[30223]: 1.6 scrub ok 2024-03-24T12:14:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[30223]: pgmap v1049: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 627 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1803/30645 objects misplaced (5.884%) 2024-03-24T12:14:15.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[30223]: 5.16 scrub starts 2024-03-24T12:14:15.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[30223]: 5.16 scrub ok 2024-03-24T12:14:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[42243]: 1.6 scrub starts 2024-03-24T12:14:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[42243]: 1.6 scrub ok 2024-03-24T12:14:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[42243]: pgmap v1049: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 627 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1803/30645 objects misplaced (5.884%) 2024-03-24T12:14:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[42243]: 5.16 scrub starts 2024-03-24T12:14:15.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:15 smithi064 ceph-mon[42243]: 5.16 scrub ok 2024-03-24T12:14:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:15 smithi193 ceph-mon[33593]: 1.6 scrub starts 2024-03-24T12:14:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:15 smithi193 ceph-mon[33593]: 1.6 scrub ok 2024-03-24T12:14:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:15 smithi193 ceph-mon[33593]: pgmap v1049: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 627 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1803/30645 objects misplaced (5.884%) 2024-03-24T12:14:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:15 smithi193 ceph-mon[33593]: 5.16 scrub starts 2024-03-24T12:14:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:15 smithi193 ceph-mon[33593]: 5.16 scrub ok 2024-03-24T12:14:15.643 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:15.678 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:15.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:15.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:15.817 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:15.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:15.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:15.958 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:15.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:16.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:16.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:16.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:16.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:16.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:16.375 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:16.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:16.476 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:16.508 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:16.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[30223]: 11.19 scrub starts 2024-03-24T12:14:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[30223]: 11.19 scrub ok 2024-03-24T12:14:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:14:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:14:16.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[30223]: 1.a scrub starts 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[30223]: 1.a scrub ok 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[42243]: 11.19 scrub starts 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[42243]: 11.19 scrub ok 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[42243]: 1.a scrub starts 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[42243]: 1.a scrub ok 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:14:16.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:16 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:14:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:16 smithi193 ceph-mon[33593]: 11.19 scrub starts 2024-03-24T12:14:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:16 smithi193 ceph-mon[33593]: 11.19 scrub ok 2024-03-24T12:14:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:16 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:14:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:16 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:14:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:16 smithi193 ceph-mon[33593]: 1.a scrub starts 2024-03-24T12:14:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:16 smithi193 ceph-mon[33593]: 1.a scrub ok 2024-03-24T12:14:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:16 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:14:16.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:16 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:14:16.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:16.652 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:16.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:16.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:16.790 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:16.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:16.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:16.892 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:16.926 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:16.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:17.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:17.073 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:17.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:17.175 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:17.209 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:17.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:17.310 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:17.345 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:17.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:17.387 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:14:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T12:14:17.387 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:14:17.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:17.485 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:17.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:17.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:14:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:17.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[30223]: pgmap v1050: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 574 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 807/28119 objects misplaced (2.870%) 2024-03-24T12:14:17.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[30223]: 4.1 scrub starts 2024-03-24T12:14:17.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[30223]: 4.1 scrub ok 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[30223]: 4.7 scrub starts 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[30223]: 4.7 scrub ok 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[42243]: pgmap v1050: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 574 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 807/28119 objects misplaced (2.870%) 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[42243]: 4.1 scrub starts 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[42243]: 4.1 scrub ok 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[42243]: 4.7 scrub starts 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[42243]: 4.7 scrub ok 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T12:14:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:17 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T12:14:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:17 smithi193 ceph-mon[33593]: pgmap v1050: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 574 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 807/28119 objects misplaced (2.870%) 2024-03-24T12:14:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:17 smithi193 ceph-mon[33593]: 4.1 scrub starts 2024-03-24T12:14:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:17 smithi193 ceph-mon[33593]: 4.1 scrub ok 2024-03-24T12:14:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:17 smithi193 ceph-mon[33593]: 4.7 scrub starts 2024-03-24T12:14:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:17 smithi193 ceph-mon[33593]: 4.7 scrub ok 2024-03-24T12:14:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:17 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T12:14:17.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:17 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T12:14:17.586 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:17.616 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:17.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:17.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:17.759 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:17.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:17.861 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:17.894 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:17.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:17.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:18.035 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:18.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:18.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:18.174 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:18.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:18.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:18.313 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:18.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:18.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:14:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T12:14:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T12:14:18.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-mon[33593]: 2.18 scrub starts 2024-03-24T12:14:18.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-mon[33593]: 2.18 scrub ok 2024-03-24T12:14:18.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-mon[33593]: 11.c scrub starts 2024-03-24T12:14:18.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-mon[33593]: 11.c scrub ok 2024-03-24T12:14:18.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:14:18.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:18 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:14:18.414 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:18.448 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:18.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:18.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T12:14:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T12:14:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[30223]: 2.18 scrub starts 2024-03-24T12:14:18.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[30223]: 2.18 scrub ok 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[30223]: 11.c scrub starts 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[30223]: 11.c scrub ok 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[42243]: 2.18 scrub starts 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[42243]: 2.18 scrub ok 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[42243]: 11.c scrub starts 2024-03-24T12:14:18.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[42243]: 11.c scrub ok 2024-03-24T12:14:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:14:18.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:18 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:14:18.587 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:18.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:18.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:18.756 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:18.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:18.858 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:18.894 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:18.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:18.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:19.055 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:19.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:19.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:19.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:19.190 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:19.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:19.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:19.303 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Removed 132130 objects 2024-03-24T12:14:19.303 INFO:tasks.radosbench.radosbench.0.smithi064.stdout:Clean up completed and total clean up time :118.964 2024-03-24T12:14:19.318 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:19.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:19.420 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:19.444 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:19.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:19.545 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[30223]: pgmap v1051: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 166 active+clean; 237 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 807/11955 objects misplaced (6.750%) 2024-03-24T12:14:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[30223]: 11.d scrub starts 2024-03-24T12:14:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[30223]: 11.d scrub ok 2024-03-24T12:14:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T12:14:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T12:14:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[42243]: pgmap v1051: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 166 active+clean; 237 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 807/11955 objects misplaced (6.750%) 2024-03-24T12:14:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[42243]: 11.d scrub starts 2024-03-24T12:14:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[42243]: 11.d scrub ok 2024-03-24T12:14:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T12:14:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:19 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T12:14:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:19.569 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:19.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:19 smithi193 ceph-mon[33593]: pgmap v1051: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 166 active+clean; 237 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 807/11955 objects misplaced (6.750%) 2024-03-24T12:14:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:19 smithi193 ceph-mon[33593]: 11.d scrub starts 2024-03-24T12:14:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:19 smithi193 ceph-mon[33593]: 11.d scrub ok 2024-03-24T12:14:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:19 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T12:14:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:19 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T12:14:19.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:19.696 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:19.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:19.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:19.822 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:19.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:19.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:19.949 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:19.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:20.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:20.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:20.177 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:20.203 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:20.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:20.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:20.320 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:20.333 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:20.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:20.334 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:20.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:20.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:20.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[30223]: 11.e scrub starts 2024-03-24T12:14:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[30223]: 11.e scrub ok 2024-03-24T12:14:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T12:14:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T12:14:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[30223]: 1.9 scrub starts 2024-03-24T12:14:20.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[30223]: 1.9 scrub ok 2024-03-24T12:14:20.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[42243]: 11.e scrub starts 2024-03-24T12:14:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[42243]: 11.e scrub ok 2024-03-24T12:14:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T12:14:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T12:14:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[42243]: 1.9 scrub starts 2024-03-24T12:14:20.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:20 smithi064 ceph-mon[42243]: 1.9 scrub ok 2024-03-24T12:14:20.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:20 smithi193 ceph-mon[33593]: 11.e scrub starts 2024-03-24T12:14:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:20 smithi193 ceph-mon[33593]: 11.e scrub ok 2024-03-24T12:14:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:20 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T12:14:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:20 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T12:14:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:20 smithi193 ceph-mon[33593]: 1.9 scrub starts 2024-03-24T12:14:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:20 smithi193 ceph-mon[33593]: 1.9 scrub ok 2024-03-24T12:14:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:20.588 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:20.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:20.689 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:20.714 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:20.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:20.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:20.841 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:20.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:20.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:20.969 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:20.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:21.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:21.096 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:21.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:21.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:21.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:21.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:21.272 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:14:21.325 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:21.351 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:21.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:21.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:21.478 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:21.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:21.496 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:21 smithi064 ceph-mon[30223]: pgmap v1052: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 154 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 84/7962 objects misplaced (1.055%) 2024-03-24T12:14:21.496 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:21 smithi064 ceph-mon[30223]: 2.16 scrub starts 2024-03-24T12:14:21.496 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:21 smithi064 ceph-mon[30223]: 2.16 scrub ok 2024-03-24T12:14:21.496 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:21 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/126977182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:21.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:21 smithi064 ceph-mon[42243]: pgmap v1052: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 154 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 84/7962 objects misplaced (1.055%) 2024-03-24T12:14:21.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:21 smithi064 ceph-mon[42243]: 2.16 scrub starts 2024-03-24T12:14:21.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:21 smithi064 ceph-mon[42243]: 2.16 scrub ok 2024-03-24T12:14:21.497 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:21 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/126977182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:21.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:21 smithi193 ceph-mon[33593]: pgmap v1052: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 154 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 84/7962 objects misplaced (1.055%) 2024-03-24T12:14:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:21 smithi193 ceph-mon[33593]: 2.16 scrub starts 2024-03-24T12:14:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:21 smithi193 ceph-mon[33593]: 2.16 scrub ok 2024-03-24T12:14:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:21 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/126977182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:21.603 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:21.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:21.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:21.729 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:21.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:21.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:21.853 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:21.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:21.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:21.983 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:21.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:22.085 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:22.112 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:22.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:22.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:22.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:22.240 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:22.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:22.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:22.369 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:22.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:22.470 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:22.497 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:22.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T12:14:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T12:14:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:14:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/126977182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:14:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[30223]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T12:14:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[30223]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T12:14:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T12:14:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T12:14:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:14:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/126977182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:14:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[42243]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T12:14:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:22 smithi064 ceph-mon[42243]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T12:14:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:22 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T12:14:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:22 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T12:14:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:22 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:14:22.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:22 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/126977182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:14:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:22 smithi193 ceph-mon[33593]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T12:14:22.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:22 smithi193 ceph-mon[33593]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T12:14:22.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:22.624 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:22.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:22.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:22.749 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:22.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:22.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:22.874 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:22.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:22.977 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:23.004 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:23.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:23.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:23.130 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:23.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:23.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:23.261 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:23.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:23.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:23.388 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:23.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:23.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:23.512 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:23.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:23.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[30223]: pgmap v1054: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 154 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 84/7962 objects misplaced (1.055%) 2024-03-24T12:14:23.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[30223]: 5.1b scrub starts 2024-03-24T12:14:23.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[30223]: 5.1b scrub ok 2024-03-24T12:14:23.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:14:23.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:14:23.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T12:14:23.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T12:14:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[42243]: pgmap v1054: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 154 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 84/7962 objects misplaced (1.055%) 2024-03-24T12:14:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[42243]: 5.1b scrub starts 2024-03-24T12:14:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[42243]: 5.1b scrub ok 2024-03-24T12:14:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:14:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:14:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T12:14:23.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:23 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T12:14:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:23 smithi193 ceph-mon[33593]: pgmap v1054: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 154 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 84/7962 objects misplaced (1.055%) 2024-03-24T12:14:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:23 smithi193 ceph-mon[33593]: 5.1b scrub starts 2024-03-24T12:14:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:23 smithi193 ceph-mon[33593]: 5.1b scrub ok 2024-03-24T12:14:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:23 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:14:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:23 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:14:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:23 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T12:14:23.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:23 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T12:14:23.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:23.637 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:23.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:23.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:23.744 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-24T12:14:23.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-24T12:14:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:23.765 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:23.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:23.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:23.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:23.993 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:24.022 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:24.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:24.123 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:24.151 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:24.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:24.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:24.253 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:24.281 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:24.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:24.342 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:14:24.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:14:24.381 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:14:24.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:24.401 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:14:24.412 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:24.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:24.421 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:14:24.442 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:14:24.463 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:14:24.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:24 smithi064 ceph-mon[30223]: 4.1e deep-scrub starts 2024-03-24T12:14:24.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:24 smithi064 ceph-mon[30223]: 4.1e deep-scrub ok 2024-03-24T12:14:24.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:24 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T12:14:24.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:24 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T12:14:24.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:24 smithi064 ceph-mon[42243]: 4.1e deep-scrub starts 2024-03-24T12:14:24.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:24 smithi064 ceph-mon[42243]: 4.1e deep-scrub ok 2024-03-24T12:14:24.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:24 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T12:14:24.464 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:24 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T12:14:24.483 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:14:24.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:14:24.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:14:24.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:14:24.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:14:24.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:14:24.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:14:24.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:14:24.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:14:24.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:24.541 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:24.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:24.556 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:14:24.556 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-03-24T12:14:24.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-03-24T12:14:24.556 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-03-24T12:14:24.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:141226 10.2 10:411c0e54:::benchmark_data_smithi064_174908_object141225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:14:24.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:06.900287+0000", 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 377.65549353, 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15928548100000001, 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141226 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900287+0000", 2024-03-24T12:14:24.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900287+0000", 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900289+0000", 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.339e-06 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900322+0000", 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2171000000000001e-05 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900337+0000", 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5877000000000001e-05 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.900347+0000", 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1050000000000001e-06 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.947992+0000", 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047645310000000003 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.948150+0000", 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015810299999999999 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.948361+0000", 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021125899999999999 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.954702+0000", 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063403089999999997 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:06.955498+0000", 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079653500000000002 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059413+0000", 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10391539499999999 2024-03-24T12:14:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059485+0000", 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1997999999999997e-05 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.059573+0000", 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.708e-05 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:142303 10.2 10:46c397f2:::benchmark_data_smithi064_174908_object142302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:07.820736+0000", 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 376.73504416100002, 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15797519400000001, 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 142303 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820736+0000", 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820736+0000", 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820739+0000", 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4169999999999999e-06 2024-03-24T12:14:24.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820765+0000", 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6361000000000001e-05 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820781+0000", 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.558e-05 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.820791+0000", 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7599999999999997e-06 2024-03-24T12:14:24.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.853298+0000", 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032507340000000003 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.853631+0000", 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033304500000000002 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.854099+0000", 2024-03-24T12:14:24.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000468269 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.858231+0000", 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041313319999999997 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.861699+0000", 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034681320000000001 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.978359+0000", 2024-03-24T12:14:24.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116660629 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.978531+0000", 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017170099999999999 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:07.978712+0000", 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018062800000000001 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:143342 10.2 10:46640381:::benchmark_data_smithi064_174908_object143341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:14:24.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:08.758084+0000", 2024-03-24T12:14:24.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.79769668300003, 2024-03-24T12:14:24.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15685858599999999, 2024-03-24T12:14:24.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:14:24.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:14:24.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143342 2024-03-24T12:14:24.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758084+0000", 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758084+0000", 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758086+0000", 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.181e-06 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758113+0000", 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6704e-05 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758126+0000", 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3035e-05 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.758136+0000", 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-03-24T12:14:24.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.805263+0000", 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047127304000000002 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.805429+0000", 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016577200000000001 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.805602+0000", 2024-03-24T12:14:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017333100000000001 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.813177+0000", 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075751949999999998 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.813218+0000", 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0893000000000003e-05 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.914242+0000", 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101023847 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.914412+0000", 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016971 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:08.914942+0000", 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053054199999999995 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:147067 10.2 10:4040c259:::benchmark_data_smithi064_174908_object2650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:12.081571+0000", 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 372.47420976000001, 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.160370979, 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147067 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081571+0000", 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081571+0000", 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081573+0000", 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8199999999999999e-06 2024-03-24T12:14:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081577+0000", 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5589999999999999e-06 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081586+0000", 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0969999999999995e-06 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.081594+0000", 2024-03-24T12:14:24.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.644e-06 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082124+0000", 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053023299999999996 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082237+0000", 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112945 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082367+0000", 2024-03-24T12:14:24.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130149 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.090336+0000", 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079686170000000008 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.091486+0000", 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011497689999999999 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.241833+0000", 2024-03-24T12:14:24.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.150347017 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.241893+0000", 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0747e-05 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.241942+0000", 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8381999999999999e-05 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:147070 10.2 10:45ca98de:::benchmark_data_smithi064_174908_object2653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:12.082601+0000", 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 372.47317925599998, 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.159632149, 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147070 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082601+0000", 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082601+0000", 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082603+0000", 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.31e-06 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082607+0000", 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5870000000000003e-06 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082618+0000", 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0579e-05 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082625+0000", 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5129999999999999e-06 2024-03-24T12:14:24.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.084948+0000", 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023222579999999998 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.085044+0000", 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6760000000000002e-05 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.085119+0000", 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4929e-05 2024-03-24T12:14:24.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.090506+0000", 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053868950000000001 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.091542+0000", 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001036326 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242074+0000", 2024-03-24T12:14:24.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.150531058 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242136+0000", 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2284000000000002e-05 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242233+0000", 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7650000000000002e-05 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:147072 10.2 10:4cd5e450:::benchmark_data_smithi064_174908_object2655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:08:12.082637+0000", 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 372.47314397899999, 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15983945199999999, 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147072 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082637+0000", 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082637+0000", 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082637+0000", 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4799999999999998e-07 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082640+0000", 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3039999999999999e-06 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082645+0000", 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3299999999999998e-06 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.082650+0000", 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1580000000000004e-06 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.087176+0000", 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004526408 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.087286+0000", 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010958700000000001 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.087401+0000", 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115451 2024-03-24T12:14:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.090512+0000", 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031100939999999999 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.091629+0000", 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001117844 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242362+0000", 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.150732586 2024-03-24T12:14:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242414+0000", 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2225999999999998e-05 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:08:12.242476+0000", 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1816000000000006e-05 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:283971 10.2 10:4a19583e:::benchmark_data_smithi064_174908_object139554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:11.429374+0000", 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 253.126406798, 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.185421002, 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 283971 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429374+0000", 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429374+0000", 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429377+0000", 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1980000000000001e-06 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429384+0000", 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3549999999999999e-06 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429400+0000", 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6112000000000001e-05 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429410+0000", 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3850000000000008e-06 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429484+0000", 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3849000000000004e-05 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429629+0000", 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014504000000000001 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429750+0000", 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121698 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614645+0000", 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18489487700000001 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614703+0000", 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8159999999999999e-05 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614715+0000", 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1252e-05 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614752+0000", 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7679000000000003e-05 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614795+0000", 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2397000000000001e-05 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.15714.0:283972 10.2 10:47bf4e9c:::benchmark_data_smithi064_174908_object139555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-24T12:14:24.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:11.429789+0000", 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 253.12599132700001, 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18520739, 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.15714", 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/208076448", 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 283972 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429789+0000", 2024-03-24T12:14:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429789+0000", 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429791+0000", 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.691e-06 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429796+0000", 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2490000000000001e-06 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429808+0000", 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1728e-05 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.429816+0000", 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3690000000000007e-06 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.432627+0000", 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028106009999999998 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611971+0000", 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17934376599999999 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.612103+0000", 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132075 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614841+0000", 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027382499999999998 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614907+0000", 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5557999999999998e-05 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614920+0000", 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2979000000000001e-05 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614960+0000", 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0302000000000001e-05 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614997+0000", 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6822000000000001e-05 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.15714.0:283971 10.2 e387/320 ondisk, result = 0)", 2024-03-24T12:14:24.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:11.433904+0000", 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 253.12187671800001, 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18075080800000001, 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433904+0000", 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433904+0000", 2024-03-24T12:14:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433905+0000", 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.584e-06 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433911+0000", 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4700000000000001e-06 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611801+0000", 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17789015399999999 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611818+0000", 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7056000000000001e-05 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614630+0000", 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028122820000000001 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614638+0000", 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9719999999999992e-06 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.614655+0000", 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6290000000000002e-05 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.15714.0:283977 10.0 e387/320 10:0061538f:::benchmark_data_smithi064_174908_object139560:head v 387'17936, mlcod=387'17934)", 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:10:11.433964+0000", 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 253.12181612399999, 2024-03-24T12:14:24.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18580016399999999, 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433964+0000", 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433964+0000", 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433966+0000", 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.051e-06 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.433972+0000", 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5479999999999998e-06 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611807+0000", 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17783502200000001 2024-03-24T12:14:24.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611818+0000", 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1031e-05 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611888+0000", 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9573999999999995e-05 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.611911+0000", 2024-03-24T12:14:24.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3061000000000002e-05 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.619732+0000", 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078214849999999995 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:10:11.619765+0000", 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2391999999999998e-05 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:195393 11.c e464/417 11:39c35ec8:::benchmark_data_smithi064_206665_object63260:head v 464'11972, mlcod=464'11969)", 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:18.662391+0000", 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.893389084999995, 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.163175183, 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.662391+0000", 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.662391+0000", 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.662393+0000", 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.251e-06 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.662399+0000", 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5579999999999997e-06 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.662409+0000", 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8719999999999997e-06 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.662416+0000", 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1729999999999998e-06 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.666770+0000", 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043540970000000003 2024-03-24T12:14:24.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.666791+0000", 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0313e-05 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.825530+0000", 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15873875600000001 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.825567+0000", 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7163000000000003e-05 2024-03-24T12:14:24.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:195394 11.c e464/417 11:3b1e715a:::benchmark_data_smithi064_206665_object63261:head v 464'11973, mlcod=464'11969)", 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:18.666942+0000", 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.888838770999996, 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15868498, 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.666942+0000", 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.666942+0000", 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.666942+0000", 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1600000000000001e-07 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.666948+0000", 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3190000000000002e-06 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.666961+0000", 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3529e-05 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.666966+0000", 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1290000000000004e-06 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678618+0000", 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011651796000000001 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678639+0000", 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1217e-05 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.825601+0000", 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14696139799999999 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.825627+0000", 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5976e-05 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:195404 11.c e464/417 11:3f956682:::benchmark_data_smithi064_206665_object63271:head v 464'11974, mlcod=464'11969)", 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:18.672952+0000", 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.882828402000001, 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17183659900000001, 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.672952+0000", 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.672952+0000", 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.672953+0000", 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1710000000000001e-06 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.624 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:24 smithi193 ceph-mon[33593]: 4.1e deep-scrub starts 2024-03-24T12:14:24.624 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:24 smithi193 ceph-mon[33593]: 4.1e deep-scrub ok 2024-03-24T12:14:24.624 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:24 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T12:14:24.625 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:24 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T12:14:24.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.672961+0000", 2024-03-24T12:14:24.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6660000000000008e-06 2024-03-24T12:14:24.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.672971+0000", 2024-03-24T12:14:24.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9639999999999998e-06 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.672978+0000", 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2699999999999999e-06 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.684068+0000", 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011089609 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.684087+0000", 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.889e-05 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.844745+0000", 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16065842999999999 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.844789+0000", 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599e-05 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:195405 11.c e464/417 11:3d5f96f9:::benchmark_data_smithi064_206665_object63272:head v 464'11975, mlcod=464'11969)", 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:18.678257+0000", 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.877523135999994, 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17203490199999999, 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678257+0000", 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678257+0000", 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678268+0000", 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0363000000000001e-05 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678296+0000", 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8056e-05 2024-03-24T12:14:24.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678305+0000", 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5880000000000004e-06 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678318+0000", 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2781e-05 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.811005+0000", 2024-03-24T12:14:24.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13268721 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.811026+0000", 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0264000000000001e-05 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.850190+0000", 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039163800999999998 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.850292+0000", 2024-03-24T12:14:24.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102839 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:195414 11.14 e464/462 11:2c9e7cc6:::benchmark_data_smithi064_206665_object63281:head v 464'11899, mlcod=464'11896)", 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:18.678539+0000", 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.877241338999994, 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.166293831, 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678539+0000", 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678539+0000", 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678540+0000", 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.186e-06 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678549+0000", 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4999999999999999e-06 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678557+0000", 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3040000000000005e-06 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.678564+0000", 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7290000000000003e-06 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.809819+0000", 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.131254869 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.809839+0000", 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0001999999999999e-05 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.844794+0000", 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034954768999999997 2024-03-24T12:14:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.844833+0000", 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9471999999999997e-05 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:195424 11.c e464/417 11:34a90518:::benchmark_data_smithi064_206665_object63291:head v 464'11976, mlcod=464'11970)", 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:18.685060+0000", 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.870721078000003, 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16526328900000001, 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.685060+0000", 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.685060+0000", 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.685061+0000", 2024-03-24T12:14:24.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1799999999999999e-06 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.685078+0000", 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7394e-05 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.801881+0000", 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116802617 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.801892+0000", 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1146e-05 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.825657+0000", 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023765340999999999 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.825675+0000", 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8210999999999999e-05 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.850298+0000", 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024622890000000001 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:18.850323+0000", 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4510000000000001e-05 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:196608 11.c e464/417 11:3b348c67:::benchmark_data_smithi064_206665_object64475:head v 464'12048, mlcod=464'12042)", 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:20.028743+0000", 2024-03-24T12:14:24.640 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.527037551999996, 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16074334600000001, 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.028743+0000", 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.028743+0000", 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.028744+0000", 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2330000000000001e-06 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.028751+0000", 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.686e-06 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.029131+0000", 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038007200000000002 2024-03-24T12:14:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.029139+0000", 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6709999999999999e-06 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.033792+0000", 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046532079999999998 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.033894+0000", 2024-03-24T12:14:24.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101656 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.189284+0000", 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.155390897 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.189486+0000", 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020192300000000001 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:196620 11.c e464/417 11:36797e7d:::benchmark_data_smithi064_206665_object64487:head v 464'12050, mlcod=464'12044)", 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:20.041108+0000", 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.514672449000003, 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.188864796, 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.041108+0000", 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.041108+0000", 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.041109+0000", 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2079999999999999e-06 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.041116+0000", 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.041125+0000", 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6290000000000005e-06 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.041132+0000", 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.533e-06 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.180982+0000", 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13984996099999999 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.181004+0000", 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2118999999999999e-05 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.229880+0000", 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048875649 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.229973+0000", 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3208000000000006e-05 2024-03-24T12:14:24.648 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.16023.0:196623 11.c e464/417 11:3feb8894:::benchmark_data_smithi064_206665_object64490:head v 464'12051, mlcod=464'12045)", 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:20.045078+0000", 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.510702921000004, 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18504226600000001, 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.045078+0000", 2024-03-24T12:14:24.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.045078+0000", 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.045079+0000", 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.19e-06 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.045085+0000", 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6279999999999999e-06 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.180667+0000", 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13558113599999999 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.180682+0000", 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5642999999999999e-05 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.189716+0000", 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090334160000000007 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.189735+0000", 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9488e-05 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.230057+0000", 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040322206999999999 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:20.230120+0000", 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2558000000000007e-05 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.16023.0:200487 11.10 11:0cf455db:::benchmark_data_smithi064_206665_object68354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-03-24T12:13:25.957446+0000", 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.598334149999999, 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39620245999999998, 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:14:24.653 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.16023", 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/943970664", 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 200487 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.957446+0000", 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.957446+0000", 2024-03-24T12:14:24.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.957448+0000", 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.957453+0000", 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.904e-06 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-03-24T12:14:24.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.957465+0000", 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1678999999999999e-05 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.957471+0000", 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4590000000000003e-06 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.957528+0000", 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7048e-05 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-03-24T12:14:24.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.957723+0000", 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019515999999999999 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.958026+0000", 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030296899999999999 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.959514+0000", 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014871789999999999 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.959628+0000", 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114121 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.960019+0000", 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039176800000000002 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.960064+0000", 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4121e-05 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:25.960178+0000", 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114335 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:26.353356+0000", 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39317796700000002 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:26.353395+0000", 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9205999999999997e-05 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-03-24T12:14:24.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-03-24T12:14:24.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-03-24T12:13:26.353649+0000", 2024-03-24T12:14:24.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025380699999999999 2024-03-24T12:14:24.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-03-24T12:14:24.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-24T12:14:24.660 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:14:24.660 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:24.661 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-24T12:14:24.661 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:14:24.661 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.28192880578159474 2024-03-24T12:14:24.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 1 0.28192880578159474 2024-03-24T12:14:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:24.689 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:24.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:24.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:24.815 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:24.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:24.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:24.946 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:24.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:25.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:25.071 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:25.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:25.172 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:25.201 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:25.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:25.303 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:25.331 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:25.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:25.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:25.461 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:25.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[30223]: pgmap v1056: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.4 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 84/780 objects misplaced (10.769%) 2024-03-24T12:14:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[30223]: 6.2 scrub starts 2024-03-24T12:14:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[30223]: 6.2 scrub ok 2024-03-24T12:14:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2051997647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:25.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[30223]: 4.4 scrub starts 2024-03-24T12:14:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[30223]: 4.4 scrub ok 2024-03-24T12:14:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[42243]: pgmap v1056: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.4 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 84/780 objects misplaced (10.769%) 2024-03-24T12:14:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[42243]: 6.2 scrub starts 2024-03-24T12:14:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[42243]: 6.2 scrub ok 2024-03-24T12:14:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2051997647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[42243]: 4.4 scrub starts 2024-03-24T12:14:25.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:25 smithi064 ceph-mon[42243]: 4.4 scrub ok 2024-03-24T12:14:25.563 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:25 smithi193 ceph-mon[33593]: pgmap v1056: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.4 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 84/780 objects misplaced (10.769%) 2024-03-24T12:14:25.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:25 smithi193 ceph-mon[33593]: 6.2 scrub starts 2024-03-24T12:14:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:25 smithi193 ceph-mon[33593]: 6.2 scrub ok 2024-03-24T12:14:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:25 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2051997647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:25 smithi193 ceph-mon[33593]: 4.4 scrub starts 2024-03-24T12:14:25.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:25 smithi193 ceph-mon[33593]: 4.4 scrub ok 2024-03-24T12:14:25.591 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:25.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:25.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:14:25.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:25.721 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:25.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:25.822 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:25.850 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:25.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:25.952 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:25.978 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:25.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:26.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:26.110 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:26.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:26.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:14:26.211 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:26.230 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:26.231 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:26.232 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:26.237 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:26.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:26.338 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:26.368 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:26.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:26.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:26 smithi064 ceph-mon[30223]: 2.19 scrub starts 2024-03-24T12:14:26.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:26 smithi064 ceph-mon[30223]: 2.19 scrub ok 2024-03-24T12:14:26.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:26 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T12:14:26.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:26 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T12:14:26.418 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:26 smithi064 ceph-mon[42243]: 2.19 scrub starts 2024-03-24T12:14:26.418 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:26 smithi064 ceph-mon[42243]: 2.19 scrub ok 2024-03-24T12:14:26.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:26 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T12:14:26.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:26 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T12:14:26.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:26.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:26 smithi193 ceph-mon[33593]: 2.19 scrub starts 2024-03-24T12:14:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:26 smithi193 ceph-mon[33593]: 2.19 scrub ok 2024-03-24T12:14:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:26 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T12:14:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:26 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T12:14:26.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:26.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:26.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:26.754 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:26.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:26.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:26.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:26.883 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:26.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:26.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:27.010 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:27.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:27.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:27.136 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:27.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:27.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:27.260 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:27.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:27.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:27.387 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:27.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:27.408 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:14:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:27.408 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: pgmap v1057: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 1.5 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 6.16 scrub starts 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 6.16 scrub ok 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 6.11 scrub starts 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 6.11 scrub ok 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 11.f scrub starts 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 2.12 deep-scrub starts 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 11.f scrub ok 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[30223]: 2.12 deep-scrub ok 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: pgmap v1057: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 1.5 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:27.409 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 6.16 scrub starts 2024-03-24T12:14:27.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 6.16 scrub ok 2024-03-24T12:14:27.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 6.11 scrub starts 2024-03-24T12:14:27.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 6.11 scrub ok 2024-03-24T12:14:27.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T12:14:27.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T12:14:27.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 11.f scrub starts 2024-03-24T12:14:27.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 2.12 deep-scrub starts 2024-03-24T12:14:27.411 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 11.f scrub ok 2024-03-24T12:14:27.411 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:27 smithi064 ceph-mon[42243]: 2.12 deep-scrub ok 2024-03-24T12:14:27.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:14:27.489 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:27.516 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:27.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:27.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: pgmap v1057: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 1.5 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 6.16 scrub starts 2024-03-24T12:14:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 6.16 scrub ok 2024-03-24T12:14:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 6.11 scrub starts 2024-03-24T12:14:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 6.11 scrub ok 2024-03-24T12:14:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T12:14:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T12:14:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 11.f scrub starts 2024-03-24T12:14:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 2.12 deep-scrub starts 2024-03-24T12:14:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 11.f scrub ok 2024-03-24T12:14:27.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:27 smithi193 ceph-mon[33593]: 2.12 deep-scrub ok 2024-03-24T12:14:27.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:27.642 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:27.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:27.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:27.768 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:27.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:27.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:27.894 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:27.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:27.995 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:28.024 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:28.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:28.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:28.152 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:28.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:28.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:28.278 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:28.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:28.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:14:28.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:28.404 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:28.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:28.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:28.529 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:28.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:28.630 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:28.660 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:28.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:28.761 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:28.790 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:28.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:14:28.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:14:28.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-mon[33593]: 6.d scrub starts 2024-03-24T12:14:28.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-mon[33593]: 6.d scrub ok 2024-03-24T12:14:28.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:28.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:28.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-mon[33593]: pgmap v1058: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 305 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:28.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:28 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:28.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:28.920 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:28.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:29.000 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:29.021 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:29.050 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:29.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:29.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:14:29.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:14:29.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[30223]: 6.d scrub starts 2024-03-24T12:14:29.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[30223]: 6.d scrub ok 2024-03-24T12:14:29.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[30223]: pgmap v1058: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 305 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[42243]: 6.d scrub starts 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[42243]: 6.d scrub ok 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[42243]: pgmap v1058: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 305 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:29.052 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:28 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:29.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:29.178 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:29.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:29.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:29.304 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:29.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:29.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:29.430 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:29.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:29.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:29.555 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:29.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:29.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:29.681 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:29.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:29.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:29.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[42243]: 5.8 scrub starts 2024-03-24T12:14:29.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[42243]: 5.8 scrub ok 2024-03-24T12:14:29.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[42243]: 4.0 scrub starts 2024-03-24T12:14:29.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[42243]: 4.0 scrub ok 2024-03-24T12:14:29.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:14:29.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[30223]: 5.8 scrub starts 2024-03-24T12:14:29.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[30223]: 5.8 scrub ok 2024-03-24T12:14:29.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[30223]: 4.0 scrub starts 2024-03-24T12:14:29.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[30223]: 4.0 scrub ok 2024-03-24T12:14:29.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:29 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:14:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:29.811 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:29.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:29 smithi193 ceph-mon[33593]: 5.8 scrub starts 2024-03-24T12:14:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:29 smithi193 ceph-mon[33593]: 5.8 scrub ok 2024-03-24T12:14:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:29 smithi193 ceph-mon[33593]: 4.0 scrub starts 2024-03-24T12:14:29.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:29 smithi193 ceph-mon[33593]: 4.0 scrub ok 2024-03-24T12:14:29.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:29 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:14:29.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:29.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:30.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:30.061 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:30.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:30.162 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:30.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:30.187 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:30.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:30.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:30.315 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:30.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:30.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:30.441 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:30.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:30.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:30.566 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:30.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:30.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:30.692 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:30.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:30.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: 2.2 deep-scrub starts 2024-03-24T12:14:30.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: 2.2 deep-scrub ok 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: pgmap v1059: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 293 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3203626300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.28192880578159474}]: dispatch 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: 2.2 deep-scrub starts 2024-03-24T12:14:30.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: 2.2 deep-scrub ok 2024-03-24T12:14:30.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: pgmap v1059: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 293 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:30.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T12:14:30.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T12:14:30.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:14:30.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:14:30.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:30.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:30 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3203626300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.28192880578159474}]: dispatch 2024-03-24T12:14:30.794 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:30.818 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:30.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: 2.2 deep-scrub starts 2024-03-24T12:14:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: 2.2 deep-scrub ok 2024-03-24T12:14:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: pgmap v1059: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 293 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T12:14:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T12:14:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:14:30.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:14:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:30.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:30 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3203626300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.28192880578159474}]: dispatch 2024-03-24T12:14:30.920 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:30.945 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:30.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:30.957 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:30.995 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 0.281929 (482c) 2024-03-24T12:14:31.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:31.071 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:31.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:31.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:31.196 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:31.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:31.298 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:31.323 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:31.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:31.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:31.451 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:31.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:31.552 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:31.577 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:31.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:31.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:31.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:31.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:31.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:31.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T12:14:31.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T12:14:31.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:14:31.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:14:31.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: 6.13 scrub starts 2024-03-24T12:14:31.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3203626300' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.28192880578159474}]': finished 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: 6.13 scrub ok 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1963142700' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: 6.13 scrub starts 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3203626300' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.28192880578159474}]': finished 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: 6.13 scrub ok 2024-03-24T12:14:31.810 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1963142700' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:14:31.829 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:31.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T12:14:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T12:14:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:14:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:14:31.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: 6.13 scrub starts 2024-03-24T12:14:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3203626300' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.28192880578159474}]': finished 2024-03-24T12:14:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T12:14:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: 6.13 scrub ok 2024-03-24T12:14:31.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1963142700' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:14:31.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:14:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:31.956 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:31.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:14:31.990 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_4' removed 2024-03-24T12:14:32.057 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:32.102 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:32.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:32.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:32.247 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:32.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:32.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:32.373 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:32.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:32.476 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:32.500 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:32.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:32.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:14:32.626 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:32.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:14:32.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:32.752 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:32.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:14:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:14:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[30223]: pgmap v1061: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 29 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1963142700' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:14:32.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[30223]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T12:14:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:14:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:14:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[42243]: pgmap v1061: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 29 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1963142700' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:14:32.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:32 smithi064 ceph-mon[42243]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T12:14:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:32 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:14:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:32 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:14:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:32 smithi193 ceph-mon[33593]: pgmap v1061: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 29 op/s; 45/639 objects misplaced (7.042%) 2024-03-24T12:14:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:32 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1963142700' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:14:32.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:32 smithi193 ceph-mon[33593]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T12:14:32.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:14:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:32.879 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:32.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:14:32.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:14:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:33.004 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:33.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:14:33.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:14:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: docker: command not found 2024-03-24T12:14:33.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi064 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:14:33.231 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:14:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:33.257 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:33.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:14:33.358 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:14:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:33.382 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:33.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:14:33.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:14:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:14:33.509 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: docker: command not found 2024-03-24T12:14:33.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 1: 'sudo docker kill -s 1 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:14:33.602 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-24T12:14:33.612 INFO:tasks.thrashosds:joining thrashosds 2024-03-24T12:14:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:33 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:14:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:33 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:14:33.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:33 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T12:14:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:33 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T12:14:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:33 smithi193 ceph-mon[33593]: 4.14 scrub starts 2024-03-24T12:14:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:33 smithi193 ceph-mon[33593]: 4.14 scrub ok 2024-03-24T12:14:33.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:33 smithi193 ceph-mon[33593]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T12:14:33.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:14:33.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:14:33.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T12:14:33.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T12:14:33.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[30223]: 4.14 scrub starts 2024-03-24T12:14:33.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[30223]: 4.14 scrub ok 2024-03-24T12:14:33.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[30223]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T12:14:33.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:14:33.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:14:33.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T12:14:33.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T12:14:33.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[42243]: 4.14 scrub starts 2024-03-24T12:14:33.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[42243]: 4.14 scrub ok 2024-03-24T12:14:33.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:33 smithi064 ceph-mon[42243]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T12:14:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:34 smithi193 ceph-mon[33593]: 4.e scrub starts 2024-03-24T12:14:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:34 smithi193 ceph-mon[33593]: 4.e scrub ok 2024-03-24T12:14:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:34 smithi193 ceph-mon[33593]: pgmap v1064: 142 pgs: 23 peering, 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-24T12:14:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:34 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T12:14:34.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:34 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T12:14:34.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[30223]: 4.e scrub starts 2024-03-24T12:14:34.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[30223]: 4.e scrub ok 2024-03-24T12:14:34.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[30223]: pgmap v1064: 142 pgs: 23 peering, 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-24T12:14:34.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T12:14:34.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T12:14:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[42243]: 4.e scrub starts 2024-03-24T12:14:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[42243]: 4.e scrub ok 2024-03-24T12:14:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[42243]: pgmap v1064: 142 pgs: 23 peering, 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-24T12:14:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T12:14:34.843 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:34 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T12:14:35.589 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-03-24T12:14:35.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: 6.1d scrub starts 2024-03-24T12:14:35.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: 6.1d scrub ok 2024-03-24T12:14:35.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-24T12:14:35.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:14:35.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: 6.10 scrub starts 2024-03-24T12:14:35.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: 4.11 scrub starts 2024-03-24T12:14:35.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: 4.11 scrub ok 2024-03-24T12:14:35.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: 6.10 scrub ok 2024-03-24T12:14:35.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4000570301' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:14:35.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: 6.1d scrub starts 2024-03-24T12:14:35.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: 6.1d scrub ok 2024-03-24T12:14:35.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-24T12:14:35.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:14:35.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: 6.10 scrub starts 2024-03-24T12:14:35.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: 4.11 scrub starts 2024-03-24T12:14:35.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: 4.11 scrub ok 2024-03-24T12:14:35.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: 6.10 scrub ok 2024-03-24T12:14:35.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4000570301' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:14:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: 6.1d scrub starts 2024-03-24T12:14:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: 6.1d scrub ok 2024-03-24T12:14:35.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-24T12:14:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:14:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: 6.10 scrub starts 2024-03-24T12:14:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: 4.11 scrub starts 2024-03-24T12:14:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: 4.11 scrub ok 2024-03-24T12:14:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: 6.10 scrub ok 2024-03-24T12:14:35.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4000570301' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:14:36.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T12:14:36.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T12:14:36.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T12:14:36.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[30223]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:14:36.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4000570301' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:14:36.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[30223]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[30223]: pgmap v1066: 142 pgs: 23 peering, 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 178 B/s, 3 objects/s recovering 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[42243]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4000570301' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[42243]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:36 smithi064 ceph-mon[42243]: pgmap v1066: 142 pgs: 23 peering, 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 178 B/s, 3 objects/s recovering 2024-03-24T12:14:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:36 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T12:14:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:36 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T12:14:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:36 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T12:14:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:36 smithi193 ceph-mon[33593]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:14:36.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:36 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4000570301' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:14:36.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:36 smithi193 ceph-mon[33593]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T12:14:36.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:36 smithi193 ceph-mon[33593]: pgmap v1066: 142 pgs: 23 peering, 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 178 B/s, 3 objects/s recovering 2024-03-24T12:14:37.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:14:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:14:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:38.099 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:38 smithi064 ceph-mon[30223]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-24T12:14:38.099 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:38 smithi064 ceph-mon[42243]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-24T12:14:38.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:38 smithi193 ceph-mon[33593]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-24T12:14:38.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:14:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:14:38.338 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-24T12:14:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-03-24T12:14:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-03-24T12:14:38.339 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-24T12:14:38.597 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-03-24T12:14:38.597 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-03-24T12:14:38.598 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-03-24T12:14:38.598 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-03-24T12:14:38.618 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-03-24T12:14:38.638 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-03-24T12:14:38.659 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-03-24T12:14:38.679 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-03-24T12:14:38.681 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:14:38.681 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:14:38.681 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:14:38.681 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:14:38.681 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:14:38.681 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:14:38.681 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:14:38.681 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:14:39.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-24T12:14:39.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[42243]: pgmap v1067: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 317 B/s, 9 objects/s recovering 2024-03-24T12:14:39.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:39.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:14:39.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:14:39.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3887923789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:39.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:14:39.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:14:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[30223]: pgmap v1067: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 317 B/s, 9 objects/s recovering 2024-03-24T12:14:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:14:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:14:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3887923789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:14:39.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:39 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:14:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:39 smithi193 ceph-mon[33593]: pgmap v1067: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 317 B/s, 9 objects/s recovering 2024-03-24T12:14:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:39 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:39 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:14:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:39 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:14:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:39 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3887923789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:14:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:39 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:14:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:39 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:14:39.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:14:40.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:14:40.197 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:40.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:40 smithi064 ceph-mon[30223]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T12:14:40.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:40 smithi064 ceph-mon[42243]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T12:14:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:40 smithi193 ceph-mon[33593]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T12:14:41.054 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:41.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[42243]: pgmap v1069: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 311 B/s rd, 0 op/s; 273 B/s, 7 objects/s recovering 2024-03-24T12:14:41.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:14:41.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:14:41.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[42243]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T12:14:41.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:14:41.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:14:41.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[30223]: pgmap v1069: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 311 B/s rd, 0 op/s; 273 B/s, 7 objects/s recovering 2024-03-24T12:14:41.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:14:41.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:14:41.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[30223]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T12:14:41.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:14:41.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:41 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:14:41.531 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:41 smithi193 ceph-mon[33593]: pgmap v1069: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 311 B/s rd, 0 op/s; 273 B/s, 7 objects/s recovering 2024-03-24T12:14:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:41 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:14:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:41 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:14:41.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:41 smithi193 ceph-mon[33593]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T12:14:41.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:41 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:14:41.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:41 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:14:42.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:42 smithi064 ceph-mon[30223]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T12:14:42.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:42 smithi064 ceph-mon[42243]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T12:14:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:42 smithi193 ceph-mon[33593]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T12:14:43.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:43 smithi064 ceph-mon[30223]: pgmap v1072: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 179 B/s, 6 objects/s recovering 2024-03-24T12:14:43.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:43.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:43.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:43 smithi064 ceph-mon[42243]: pgmap v1072: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 179 B/s, 6 objects/s recovering 2024-03-24T12:14:43.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:43.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:43.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:43 smithi193 ceph-mon[33593]: pgmap v1072: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 179 B/s, 6 objects/s recovering 2024-03-24T12:14:43.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:43.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-03-24T12:14:44.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:44 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3428733009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:44.139 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:44 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3428733009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:44 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3428733009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:14:45.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: Cluster is now healthy 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3428733009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T12:14:45.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: Cluster is now healthy 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3428733009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T12:14:45.355 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:45 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T12:14:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-24T12:14:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: Cluster is now healthy 2024-03-24T12:14:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3428733009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:14:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T12:14:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T12:14:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T12:14:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T12:14:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:45 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T12:14:45.846 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a: {} 2024-03-24T12:14:45.847 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b: {} 2024-03-24T12:14:45.847 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c: {} 2024-03-24T12:14:45.847 INFO:teuthology.orchestra.run.smithi064.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:14:45.847 INFO:teuthology.orchestra.run.smithi064.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:14:45.847 INFO:teuthology.orchestra.run.smithi064.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:14:46.164 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:14:46.165 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:14:46.165 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T12:14:46.165 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T12:14:46.165 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T12:14:46.165 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T12:14:46.165 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T12:14:46.165 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T12:14:46.165 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3906678878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:14:46.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T12:14:46.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T12:14:46.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T12:14:46.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T12:14:46.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T12:14:46.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T12:14:46.166 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:46 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3906678878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:14:46.450 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:46 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T12:14:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:46 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T12:14:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:46 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T12:14:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:46 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T12:14:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:46 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T12:14:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:46 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T12:14:46.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:46 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3906678878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:14:47.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: 5.17 scrub starts 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: 5.17 scrub ok 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: 4.1b scrub starts 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: 4.1b scrub ok 2024-03-24T12:14:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: 5.17 scrub starts 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: 5.17 scrub ok 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: 4.1b scrub starts 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: 4.1b scrub ok 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:14:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:14:47.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-24T12:14:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:14:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:14:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: 5.17 scrub starts 2024-03-24T12:14:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: 5.17 scrub ok 2024-03-24T12:14:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T12:14:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: 4.1b scrub starts 2024-03-24T12:14:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: 4.1b scrub ok 2024-03-24T12:14:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:14:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:47 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:14:47.649 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:14:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:14:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:48.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:14:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:14:48.386 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:48.435 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:49.356 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:49 smithi064 ceph-mon[30223]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-24T12:14:49.356 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:49 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T12:14:49.356 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:49 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T12:14:49.357 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:49.357 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:49 smithi064 ceph-mon[42243]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-24T12:14:49.357 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:49 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T12:14:49.357 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:49 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T12:14:49.357 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:49 smithi193 ceph-mon[33593]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-24T12:14:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:49 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T12:14:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:49 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T12:14:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:51 smithi064 ceph-mon[30223]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:51 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:14:51.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:51 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:14:51.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:51 smithi064 ceph-mon[42243]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:51 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:14:51.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:51 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:14:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:51 smithi193 ceph-mon[33593]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:51 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:14:51.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:51 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:14:52.149 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-24T12:14:52.159 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-03-24T12:14:52.372 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T12:14:52.372 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[42243]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[42243]: 6.7 scrub starts 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[42243]: 6.7 scrub ok 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3733060560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[42243]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/992717332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[30223]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[30223]: 6.7 scrub starts 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[30223]: 6.7 scrub ok 2024-03-24T12:14:52.373 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3733060560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:52.374 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[30223]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:52.374 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:52 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/992717332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:52 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T12:14:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:52 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T12:14:52.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:52 smithi193 ceph-mon[33593]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T12:14:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:52 smithi193 ceph-mon[33593]: 6.7 scrub starts 2024-03-24T12:14:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:52 smithi193 ceph-mon[33593]: 6.7 scrub ok 2024-03-24T12:14:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:52 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3733060560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:52 smithi193 ceph-mon[33593]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:14:52.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:52 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/992717332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:53.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-24T12:14:53.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[30223]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-24T12:14:53.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[30223]: from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:14:53.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/992717332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:14:53.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[30223]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T12:14:53.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T12:14:53.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T12:14:53.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[42243]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-24T12:14:53.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[42243]: from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:14:53.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/992717332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:14:53.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[42243]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T12:14:53.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T12:14:53.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:53 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T12:14:53.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:53 smithi193 ceph-mon[33593]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-24T12:14:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:53 smithi193 ceph-mon[33593]: from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:14:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/992717332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:14:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:53 smithi193 ceph-mon[33593]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T12:14:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:53 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T12:14:53.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:53 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T12:14:54.060 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:14:54.060 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:14:54.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T12:14:54.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T12:14:54.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[30223]: 2.a scrub starts 2024-03-24T12:14:54.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[30223]: 2.a scrub ok 2024-03-24T12:14:54.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:14:54.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:14:54.394 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T12:14:54.394 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T12:14:54.395 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[42243]: 2.a scrub starts 2024-03-24T12:14:54.395 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[42243]: 2.a scrub ok 2024-03-24T12:14:54.395 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:14:54.395 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:54 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:14:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:54 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T12:14:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:54 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T12:14:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:54 smithi193 ceph-mon[33593]: 2.a scrub starts 2024-03-24T12:14:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:54 smithi193 ceph-mon[33593]: 2.a scrub ok 2024-03-24T12:14:54.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:54 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:14:54.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:54 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:14:54.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:14:54.714 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:55.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:55 smithi064 ceph-mon[30223]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-24T12:14:55.241 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:55 smithi064 ceph-mon[30223]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T12:14:55.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:55 smithi064 ceph-mon[42243]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-24T12:14:55.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:55 smithi064 ceph-mon[42243]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T12:14:55.580 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:14:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:55 smithi193 ceph-mon[33593]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-24T12:14:55.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:55 smithi193 ceph-mon[33593]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T12:14:56.280 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[42243]: 4.8 scrub starts 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[42243]: 4.8 scrub ok 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]: dispatch 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]: dispatch 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 2]}]: dispatch 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[30223]: 4.8 scrub starts 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[30223]: 4.8 scrub ok 2024-03-24T12:14:56.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T12:14:56.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T12:14:56.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]: dispatch 2024-03-24T12:14:56.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]: dispatch 2024-03-24T12:14:56.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 2]}]: dispatch 2024-03-24T12:14:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:56 smithi193 ceph-mon[33593]: 4.8 scrub starts 2024-03-24T12:14:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:56 smithi193 ceph-mon[33593]: 4.8 scrub ok 2024-03-24T12:14:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:56 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T12:14:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:56 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T12:14:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]: dispatch 2024-03-24T12:14:56.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]: dispatch 2024-03-24T12:14:56.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 2]}]: dispatch 2024-03-24T12:14:57.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:57.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: 6.0 scrub starts 2024-03-24T12:14:57.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: 6.0 scrub ok 2024-03-24T12:14:57.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T12:14:57.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T12:14:57.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]': finished 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 2]}]': finished 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: 5.7 scrub starts 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: 5.7 scrub ok 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: 6.1c scrub starts 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[30223]: 6.1c scrub ok 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: 6.0 scrub starts 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: 6.0 scrub ok 2024-03-24T12:14:57.382 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]': finished 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 2]}]': finished 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: 5.7 scrub starts 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: 5.7 scrub ok 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: 6.1c scrub starts 2024-03-24T12:14:57.383 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-mon[42243]: 6.1c scrub ok 2024-03-24T12:14:57.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: 6.0 scrub starts 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: 6.0 scrub ok 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]': finished 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 2]}]': finished 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: 5.7 scrub starts 2024-03-24T12:14:57.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: 5.7 scrub ok 2024-03-24T12:14:57.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: 6.1c scrub starts 2024-03-24T12:14:57.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:57 smithi193 ceph-mon[33593]: 6.1c scrub ok 2024-03-24T12:14:57.707 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:14:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:14:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:58.178 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[30223]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1872480584' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[30223]: from='client.36128 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[42243]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1872480584' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[42243]: from='client.36128 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:58.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:58.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:14:58 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:14:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:58 smithi193 ceph-mon[33593]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T12:14:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:58 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1872480584' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:58.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:58 smithi193 ceph-mon[33593]: from='client.36128 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:14:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:14:58.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:58.512 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:14:58.512 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:14:59.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-24T12:14:59.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[30223]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-24T12:14:59.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[30223]: from='client.36128 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[30223]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[42243]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T12:14:59.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[42243]: from='client.36128 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:14:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[42243]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T12:14:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T12:14:59.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:14:59 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T12:14:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:59 smithi193 ceph-mon[33593]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-24T12:14:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:59 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T12:14:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:59 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T12:14:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:59 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:59 smithi193 ceph-mon[33593]: from='client.36128 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:14:59.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:59 smithi193 ceph-mon[33593]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T12:14:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:59 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T12:14:59.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:14:59 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T12:14:59.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:14:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:00 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T12:15:00.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:00 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T12:15:00.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:00 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T12:15:00.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:00 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T12:15:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:00 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T12:15:00.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:00 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T12:15:00.636 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:01.489 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:01 smithi064 ceph-mon[30223]: pgmap v1089: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 40 B/s, 2 objects/s recovering 2024-03-24T12:15:01.490 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:01 smithi064 ceph-mon[42243]: pgmap v1089: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 40 B/s, 2 objects/s recovering 2024-03-24T12:15:01.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:01 smithi193 ceph-mon[33593]: pgmap v1089: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 40 B/s, 2 objects/s recovering 2024-03-24T12:15:02.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:02 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T12:15:02.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:02 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T12:15:02.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:02 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T12:15:02.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:02 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T12:15:02.461 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:02 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T12:15:02.461 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:02 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T12:15:02.461 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:02 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T12:15:02.461 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:02 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T12:15:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:02 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T12:15:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:02 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T12:15:02.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:02 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T12:15:02.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:02 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T12:15:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:03 smithi064 ceph-mon[30223]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-03-24T12:15:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:03 smithi064 ceph-mon[30223]: 5.1a deep-scrub starts 2024-03-24T12:15:03.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:03 smithi064 ceph-mon[30223]: 5.1a deep-scrub ok 2024-03-24T12:15:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:03 smithi064 ceph-mon[42243]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-03-24T12:15:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:03 smithi064 ceph-mon[42243]: 5.1a deep-scrub starts 2024-03-24T12:15:03.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:03 smithi064 ceph-mon[42243]: 5.1a deep-scrub ok 2024-03-24T12:15:03.339 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:03.339 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:03 smithi193 ceph-mon[33593]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-03-24T12:15:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:03 smithi193 ceph-mon[33593]: 5.1a deep-scrub starts 2024-03-24T12:15:03.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:03 smithi193 ceph-mon[33593]: 5.1a deep-scrub ok 2024-03-24T12:15:04.210 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-24T12:15:04.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:15:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T12:15:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T12:15:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[30223]: 5.1 deep-scrub starts 2024-03-24T12:15:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[30223]: 5.1 deep-scrub ok 2024-03-24T12:15:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1761272983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:15:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T12:15:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T12:15:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T12:15:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T12:15:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[42243]: 5.1 deep-scrub starts 2024-03-24T12:15:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[42243]: 5.1 deep-scrub ok 2024-03-24T12:15:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1761272983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:15:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T12:15:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:04 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T12:15:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:04 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T12:15:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:04 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T12:15:04.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:04 smithi193 ceph-mon[33593]: 5.1 deep-scrub starts 2024-03-24T12:15:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:04 smithi193 ceph-mon[33593]: 5.1 deep-scrub ok 2024-03-24T12:15:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:04 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1761272983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:15:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:04 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T12:15:04.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:04 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T12:15:04.668 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:05.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-24T12:15:05.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[30223]: pgmap v1091: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 549 B/s rd, 0 op/s; 29 B/s, 2 objects/s recovering 2024-03-24T12:15:05.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T12:15:05.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T12:15:05.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1761272983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:15:05.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[30223]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T12:15:05.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[30223]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T12:15:05.394 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[42243]: pgmap v1091: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 549 B/s rd, 0 op/s; 29 B/s, 2 objects/s recovering 2024-03-24T12:15:05.394 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T12:15:05.394 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T12:15:05.394 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1761272983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:15:05.394 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[42243]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T12:15:05.395 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:05 smithi064 ceph-mon[42243]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T12:15:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:05 smithi193 ceph-mon[33593]: pgmap v1091: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 549 B/s rd, 0 op/s; 29 B/s, 2 objects/s recovering 2024-03-24T12:15:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:05 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T12:15:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:05 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T12:15:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:05 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1761272983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:15:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:05 smithi193 ceph-mon[33593]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T12:15:05.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:05 smithi193 ceph-mon[33593]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T12:15:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:07.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:07 smithi064 ceph-mon[42243]: pgmap v1094: 142 pgs: 8 peering, 134 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 1 objects/s recovering 2024-03-24T12:15:07.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:07 smithi064 ceph-mon[30223]: pgmap v1094: 142 pgs: 8 peering, 134 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 1 objects/s recovering 2024-03-24T12:15:07.552 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:15:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:15:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:07.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:07 smithi193 ceph-mon[33593]: pgmap v1094: 142 pgs: 8 peering, 134 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 1 objects/s recovering 2024-03-24T12:15:07.689 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:07.689 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:08 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T12:15:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:08 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T12:15:08.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:08 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T12:15:08.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:08 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T12:15:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:08 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T12:15:08.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:08 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T12:15:08.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:15:08 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:15:08.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:15:09.114 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:09.229 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:09 smithi064 ceph-mon[30223]: pgmap v1095: 142 pgs: 8 peering, 134 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 148 B/s, 4 objects/s recovering 2024-03-24T12:15:09.230 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:09.230 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:09 smithi064 ceph-mon[30223]: 2.8 deep-scrub starts 2024-03-24T12:15:09.230 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:09 smithi064 ceph-mon[30223]: 2.8 deep-scrub ok 2024-03-24T12:15:09.230 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:09 smithi064 ceph-mon[42243]: pgmap v1095: 142 pgs: 8 peering, 134 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 148 B/s, 4 objects/s recovering 2024-03-24T12:15:09.230 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:09.230 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:09 smithi064 ceph-mon[42243]: 2.8 deep-scrub starts 2024-03-24T12:15:09.230 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:09 smithi064 ceph-mon[42243]: 2.8 deep-scrub ok 2024-03-24T12:15:09.528 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:09 smithi193 ceph-mon[33593]: pgmap v1095: 142 pgs: 8 peering, 134 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 148 B/s, 4 objects/s recovering 2024-03-24T12:15:09.528 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:09.528 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:09 smithi193 ceph-mon[33593]: 2.8 deep-scrub starts 2024-03-24T12:15:09.528 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:09 smithi193 ceph-mon[33593]: 2.8 deep-scrub ok 2024-03-24T12:15:10.241 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-24T12:15:10.513 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:10 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T12:15:10.513 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:10 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T12:15:10.513 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:10 smithi193 ceph-mon[33593]: 5.c deep-scrub starts 2024-03-24T12:15:10.513 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:10 smithi193 ceph-mon[33593]: 5.c deep-scrub ok 2024-03-24T12:15:10.513 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:10 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2393145972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:15:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T12:15:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T12:15:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[30223]: 5.c deep-scrub starts 2024-03-24T12:15:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[30223]: 5.c deep-scrub ok 2024-03-24T12:15:10.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2393145972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:15:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T12:15:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T12:15:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[42243]: 5.c deep-scrub starts 2024-03-24T12:15:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[42243]: 5.c deep-scrub ok 2024-03-24T12:15:10.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:10 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2393145972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:15:11.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-24T12:15:11.294 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:11.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[42243]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 196 B/s, 5 objects/s recovering 2024-03-24T12:15:11.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[42243]: 5.19 scrub starts 2024-03-24T12:15:11.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[42243]: 5.19 scrub ok 2024-03-24T12:15:11.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2393145972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:15:11.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[42243]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T12:15:11.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[30223]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 196 B/s, 5 objects/s recovering 2024-03-24T12:15:11.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[30223]: 5.19 scrub starts 2024-03-24T12:15:11.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[30223]: 5.19 scrub ok 2024-03-24T12:15:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2393145972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:15:11.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:11 smithi064 ceph-mon[30223]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T12:15:11.385 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:11.385 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:11 smithi193 ceph-mon[33593]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 196 B/s, 5 objects/s recovering 2024-03-24T12:15:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:11 smithi193 ceph-mon[33593]: 5.19 scrub starts 2024-03-24T12:15:11.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:11 smithi193 ceph-mon[33593]: 5.19 scrub ok 2024-03-24T12:15:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:11 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2393145972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:15:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:11 smithi193 ceph-mon[33593]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T12:15:12.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:15:12.262 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:12.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:12 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T12:15:12.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:12 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T12:15:12.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:12 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T12:15:12.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:12 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T12:15:12.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:12 smithi193 ceph-mon[33593]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T12:15:12.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:12 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:15:12.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:12 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:15:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T12:15:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T12:15:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T12:15:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T12:15:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[30223]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T12:15:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:15:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:15:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T12:15:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T12:15:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T12:15:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T12:15:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[42243]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T12:15:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:15:12.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:12 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:15:13.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:13 smithi193 ceph-mon[33593]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 245 B/s, 7 objects/s recovering 2024-03-24T12:15:13.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:13 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T12:15:13.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:13 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T12:15:13.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:13 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T12:15:13.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:13 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T12:15:13.372 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:13 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[30223]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 245 B/s, 7 objects/s recovering 2024-03-24T12:15:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T12:15:13.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[42243]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 245 B/s, 7 objects/s recovering 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T12:15:13.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:13 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:13.942 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:13.942 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:14.265 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-24T12:15:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:14 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1112334554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:14 smithi064 ceph-mon[30223]: from='client.36146 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:14 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1112334554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:14.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:14 smithi064 ceph-mon[42243]: from='client.36146 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:14 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1112334554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:14 smithi193 ceph-mon[33593]: from='client.36146 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:14.790 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:15:15.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-24T12:15:15.029 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:15.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[30223]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 84 B/s, 3 objects/s recovering 2024-03-24T12:15:15.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T12:15:15.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T12:15:15.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T12:15:15.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T12:15:15.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[30223]: from='client.36146 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:15:15.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[30223]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T12:15:15.285 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:15.401 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:15 smithi193 ceph-mon[33593]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 84 B/s, 3 objects/s recovering 2024-03-24T12:15:15.401 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:15 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T12:15:15.402 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:15 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T12:15:15.402 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:15 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T12:15:15.402 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:15 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T12:15:15.402 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:15 smithi193 ceph-mon[33593]: from='client.36146 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:15:15.402 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:15 smithi193 ceph-mon[33593]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T12:15:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[42243]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 84 B/s, 3 objects/s recovering 2024-03-24T12:15:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T12:15:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T12:15:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T12:15:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T12:15:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[42243]: from='client.36146 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:15:15.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:15 smithi064 ceph-mon[42243]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T12:15:16.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:16 smithi193 ceph-mon[33593]: 4.3 scrub starts 2024-03-24T12:15:16.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:16 smithi193 ceph-mon[33593]: 4.3 scrub ok 2024-03-24T12:15:16.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:16 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T12:15:16.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:16 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T12:15:16.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:16 smithi193 ceph-mon[33593]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T12:15:16.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:16 smithi193 ceph-mon[33593]: 2.5 deep-scrub starts 2024-03-24T12:15:16.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:16 smithi193 ceph-mon[33593]: 2.5 deep-scrub ok 2024-03-24T12:15:16.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[30223]: 4.3 scrub starts 2024-03-24T12:15:16.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[30223]: 4.3 scrub ok 2024-03-24T12:15:16.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T12:15:16.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T12:15:16.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[30223]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T12:15:16.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[30223]: 2.5 deep-scrub starts 2024-03-24T12:15:16.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[30223]: 2.5 deep-scrub ok 2024-03-24T12:15:16.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[42243]: 4.3 scrub starts 2024-03-24T12:15:16.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[42243]: 4.3 scrub ok 2024-03-24T12:15:16.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T12:15:16.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T12:15:16.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[42243]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T12:15:16.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[42243]: 2.5 deep-scrub starts 2024-03-24T12:15:16.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:16 smithi064 ceph-mon[42243]: 2.5 deep-scrub ok 2024-03-24T12:15:16.791 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:16.791 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:17.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:17 smithi193 ceph-mon[33593]: pgmap v1103: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-03-24T12:15:17.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:17 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T12:15:17.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:17 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T12:15:17.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:17 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T12:15:17.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:17 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T12:15:17.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:17 smithi193 ceph-mon[33593]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T12:15:17.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:17 smithi193 ceph-mon[33593]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 83'130 2024-03-24T12:15:17.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:17 smithi193 ceph-mon[33593]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T12:15:17.404 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:15:17.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:15:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:17.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[30223]: pgmap v1103: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[30223]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[30223]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 83'130 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[30223]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[42243]: pgmap v1103: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T12:15:17.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T12:15:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[42243]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T12:15:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[42243]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 83'130 2024-03-24T12:15:17.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:17 smithi064 ceph-mon[42243]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T12:15:17.645 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-24T12:15:18.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:15:18.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T12:15:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T12:15:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: 2.4 deep-scrub starts 2024-03-24T12:15:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: 2.4 deep-scrub ok 2024-03-24T12:15:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1241501745' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:15:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:15:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1241501745' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:15:18.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:18 smithi193 ceph-mon[33593]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T12:15:18.366 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: 2.4 deep-scrub starts 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: 2.4 deep-scrub ok 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1241501745' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1241501745' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[30223]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T12:15:18.367 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: 2.4 deep-scrub starts 2024-03-24T12:15:18.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: 2.4 deep-scrub ok 2024-03-24T12:15:18.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1241501745' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:15:18.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:15:18.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:15:18.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1241501745' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:15:18.368 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:18 smithi064 ceph-mon[42243]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T12:15:18.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-24T12:15:19.045 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:19.335 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:19.335 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:19 smithi193 ceph-mon[33593]: pgmap v1106: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 2 peering, 138 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 58 B/s, 4 objects/s recovering 2024-03-24T12:15:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:19.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:19 smithi193 ceph-mon[33593]: 2.e scrub starts 2024-03-24T12:15:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:19 smithi193 ceph-mon[33593]: 2.e scrub ok 2024-03-24T12:15:19.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:19 smithi193 ceph-mon[33593]: osdmap e515: 8 total, 8 up, 8 in 2024-03-24T12:15:19.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[30223]: pgmap v1106: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 2 peering, 138 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 58 B/s, 4 objects/s recovering 2024-03-24T12:15:19.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:19.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[30223]: 2.e scrub starts 2024-03-24T12:15:19.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[30223]: 2.e scrub ok 2024-03-24T12:15:19.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[30223]: osdmap e515: 8 total, 8 up, 8 in 2024-03-24T12:15:19.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[42243]: pgmap v1106: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 2 peering, 138 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 58 B/s, 4 objects/s recovering 2024-03-24T12:15:19.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:19.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[42243]: 2.e scrub starts 2024-03-24T12:15:19.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[42243]: 2.e scrub ok 2024-03-24T12:15:19.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:19 smithi064 ceph-mon[42243]: osdmap e515: 8 total, 8 up, 8 in 2024-03-24T12:15:19.961 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:15:20.200 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:20.442 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[30223]: 4.c scrub starts 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[30223]: 4.c scrub ok 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[42243]: 4.c scrub starts 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[42243]: 4.c scrub ok 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T12:15:20.443 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:20 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T12:15:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:20 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T12:15:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:20 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T12:15:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:20 smithi193 ceph-mon[33593]: 4.c scrub starts 2024-03-24T12:15:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:20 smithi193 ceph-mon[33593]: 4.c scrub ok 2024-03-24T12:15:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:20 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T12:15:20.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:20 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T12:15:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T12:15:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[30223]: pgmap v1109: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 244 B/s, 8 objects/s recovering 2024-03-24T12:15:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T12:15:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[30223]: 6.14 deep-scrub starts 2024-03-24T12:15:21.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[30223]: 6.14 deep-scrub ok 2024-03-24T12:15:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T12:15:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[42243]: pgmap v1109: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 244 B/s, 8 objects/s recovering 2024-03-24T12:15:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T12:15:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[42243]: 6.14 deep-scrub starts 2024-03-24T12:15:21.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:21 smithi064 ceph-mon[42243]: 6.14 deep-scrub ok 2024-03-24T12:15:21.426 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:21 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T12:15:21.426 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:21 smithi193 ceph-mon[33593]: pgmap v1109: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 244 B/s, 8 objects/s recovering 2024-03-24T12:15:21.426 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:21 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T12:15:21.426 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:21 smithi193 ceph-mon[33593]: 6.14 deep-scrub starts 2024-03-24T12:15:21.427 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:21 smithi193 ceph-mon[33593]: 6.14 deep-scrub ok 2024-03-24T12:15:21.962 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:21.962 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:22.297 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-24T12:15:22.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:22 smithi193 ceph-mon[33593]: 6.19 scrub starts 2024-03-24T12:15:22.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:22 smithi193 ceph-mon[33593]: 4.1d scrub starts 2024-03-24T12:15:22.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:22 smithi193 ceph-mon[33593]: 4.1d scrub ok 2024-03-24T12:15:22.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:22 smithi193 ceph-mon[33593]: 6.19 scrub ok 2024-03-24T12:15:22.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:22 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1434786350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:15:22.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:22 smithi193 ceph-mon[33593]: 5.12 scrub starts 2024-03-24T12:15:22.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:22 smithi193 ceph-mon[33593]: 5.12 scrub ok 2024-03-24T12:15:22.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:15:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[30223]: 6.19 scrub starts 2024-03-24T12:15:22.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[30223]: 4.1d scrub starts 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[30223]: 4.1d scrub ok 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[30223]: 6.19 scrub ok 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1434786350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[30223]: 5.12 scrub starts 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[30223]: 5.12 scrub ok 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[42243]: 6.19 scrub starts 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[42243]: 4.1d scrub starts 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[42243]: 4.1d scrub ok 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[42243]: 6.19 scrub ok 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1434786350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[42243]: 5.12 scrub starts 2024-03-24T12:15:22.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:22 smithi064 ceph-mon[42243]: 5.12 scrub ok 2024-03-24T12:15:22.781 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:23.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-24T12:15:23.279 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[30223]: 5.2 scrub starts 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[30223]: 5.2 scrub ok 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[30223]: pgmap v1110: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 382 B/s rd, 0 op/s; 183 B/s, 6 objects/s recovering 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1434786350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[30223]: osdmap e516: 8 total, 8 up, 8 in 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[42243]: 5.2 scrub starts 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[42243]: 5.2 scrub ok 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[42243]: pgmap v1110: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 382 B/s rd, 0 op/s; 183 B/s, 6 objects/s recovering 2024-03-24T12:15:23.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1434786350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:15:23.545 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:23 smithi064 ceph-mon[42243]: osdmap e516: 8 total, 8 up, 8 in 2024-03-24T12:15:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:23 smithi193 ceph-mon[33593]: 5.2 scrub starts 2024-03-24T12:15:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:23 smithi193 ceph-mon[33593]: 5.2 scrub ok 2024-03-24T12:15:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:23 smithi193 ceph-mon[33593]: pgmap v1110: 142 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 140 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 382 B/s rd, 0 op/s; 183 B/s, 6 objects/s recovering 2024-03-24T12:15:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:23 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1434786350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:15:23.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:23 smithi193 ceph-mon[33593]: osdmap e516: 8 total, 8 up, 8 in 2024-03-24T12:15:24.442 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:24.442 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T12:15:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T12:15:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[30223]: 6.b scrub starts 2024-03-24T12:15:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[30223]: 6.b scrub ok 2024-03-24T12:15:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[30223]: 3.0 scrub starts 2024-03-24T12:15:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[30223]: 3.0 scrub ok 2024-03-24T12:15:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T12:15:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T12:15:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[42243]: 6.b scrub starts 2024-03-24T12:15:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[42243]: 6.b scrub ok 2024-03-24T12:15:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[42243]: 3.0 scrub starts 2024-03-24T12:15:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:24 smithi064 ceph-mon[42243]: 3.0 scrub ok 2024-03-24T12:15:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:24 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T12:15:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:24 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T12:15:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:24 smithi193 ceph-mon[33593]: 6.b scrub starts 2024-03-24T12:15:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:24 smithi193 ceph-mon[33593]: 6.b scrub ok 2024-03-24T12:15:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:24 smithi193 ceph-mon[33593]: 3.0 scrub starts 2024-03-24T12:15:24.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:24 smithi193 ceph-mon[33593]: 3.0 scrub ok 2024-03-24T12:15:25.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:15:25.316 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:25 smithi193 ceph-mon[33593]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 124 B/s, 2 objects/s recovering 2024-03-24T12:15:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:25 smithi193 ceph-mon[33593]: 1.1 scrub starts 2024-03-24T12:15:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:25 smithi193 ceph-mon[33593]: 1.1 scrub ok 2024-03-24T12:15:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:25 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T12:15:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:25 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T12:15:25.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[42243]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 124 B/s, 2 objects/s recovering 2024-03-24T12:15:25.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[42243]: 1.1 scrub starts 2024-03-24T12:15:25.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[42243]: 1.1 scrub ok 2024-03-24T12:15:25.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T12:15:25.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T12:15:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[30223]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 124 B/s, 2 objects/s recovering 2024-03-24T12:15:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[30223]: 1.1 scrub starts 2024-03-24T12:15:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[30223]: 1.1 scrub ok 2024-03-24T12:15:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T12:15:25.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:25 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T12:15:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-24T12:15:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:26 smithi193 ceph-mon[33593]: 5.1e scrub starts 2024-03-24T12:15:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:26 smithi193 ceph-mon[33593]: 5.1e scrub ok 2024-03-24T12:15:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:26 smithi193 ceph-mon[33593]: 4.d scrub starts 2024-03-24T12:15:26.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:26 smithi193 ceph-mon[33593]: 4.d scrub ok 2024-03-24T12:15:26.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:26 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3603036180' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:15:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[30223]: 5.1e scrub starts 2024-03-24T12:15:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[30223]: 5.1e scrub ok 2024-03-24T12:15:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[30223]: 4.d scrub starts 2024-03-24T12:15:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[30223]: 4.d scrub ok 2024-03-24T12:15:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3603036180' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:15:26.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[42243]: 5.1e scrub starts 2024-03-24T12:15:26.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[42243]: 5.1e scrub ok 2024-03-24T12:15:26.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[42243]: 4.d scrub starts 2024-03-24T12:15:26.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[42243]: 4.d scrub ok 2024-03-24T12:15:26.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:26 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3603036180' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:15:27.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-24T12:15:27.286 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:27.287 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:27.374 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:27.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[30223]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 676 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:15:27.380 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[30223]: 2.b scrub starts 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[30223]: 2.b scrub ok 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3603036180' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[30223]: osdmap e517: 8 total, 8 up, 8 in 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[42243]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 676 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[42243]: 2.b scrub starts 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[42243]: 2.b scrub ok 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3603036180' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[42243]: osdmap e517: 8 total, 8 up, 8 in 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:15:27.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:15:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:27 smithi193 ceph-mon[33593]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 676 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:15:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:27 smithi193 ceph-mon[33593]: 2.b scrub starts 2024-03-24T12:15:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:27 smithi193 ceph-mon[33593]: 2.b scrub ok 2024-03-24T12:15:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:27 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3603036180' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:15:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:27 smithi193 ceph-mon[33593]: osdmap e517: 8 total, 8 up, 8 in 2024-03-24T12:15:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:27 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:15:27.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:27 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:15:27.639 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:15:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:15:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:28.028 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:15:28.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:15:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:15:28.279 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:28.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:28 smithi193 ceph-mon[33593]: 4.9 scrub starts 2024-03-24T12:15:28.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:28 smithi193 ceph-mon[33593]: 4.9 scrub ok 2024-03-24T12:15:28.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:28 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T12:15:28.525 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:28 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T12:15:28.525 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:28.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[30223]: 4.9 scrub starts 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[30223]: 4.9 scrub ok 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[42243]: 4.9 scrub starts 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[42243]: 4.9 scrub ok 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T12:15:28.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:29.443 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:29 smithi193 ceph-mon[33593]: 2.0 deep-scrub starts 2024-03-24T12:15:29.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:29 smithi193 ceph-mon[33593]: 2.0 deep-scrub ok 2024-03-24T12:15:29.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:29 smithi193 ceph-mon[33593]: pgmap v1115: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:15:29.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:29 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:29.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:29 smithi193 ceph-mon[33593]: 1.a scrub starts 2024-03-24T12:15:29.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:29 smithi193 ceph-mon[33593]: 1.a scrub ok 2024-03-24T12:15:29.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:29 smithi193 ceph-mon[33593]: 1.0 scrub starts 2024-03-24T12:15:29.444 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:29 smithi193 ceph-mon[33593]: 1.0 scrub ok 2024-03-24T12:15:29.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[30223]: 2.0 deep-scrub starts 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[30223]: 2.0 deep-scrub ok 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[30223]: pgmap v1115: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[30223]: 1.a scrub starts 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[30223]: 1.a scrub ok 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[30223]: 1.0 scrub starts 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[30223]: 1.0 scrub ok 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[42243]: 2.0 deep-scrub starts 2024-03-24T12:15:29.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[42243]: 2.0 deep-scrub ok 2024-03-24T12:15:29.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[42243]: pgmap v1115: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:15:29.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:29.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[42243]: 1.a scrub starts 2024-03-24T12:15:29.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[42243]: 1.a scrub ok 2024-03-24T12:15:29.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[42243]: 1.0 scrub starts 2024-03-24T12:15:29.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:29 smithi064 ceph-mon[42243]: 1.0 scrub ok 2024-03-24T12:15:30.021 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:30.021 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:30.329 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-03-24T12:15:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:30 smithi193 ceph-mon[33593]: 1.4 scrub starts 2024-03-24T12:15:30.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:30 smithi193 ceph-mon[33593]: 1.4 scrub ok 2024-03-24T12:15:30.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:30 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2785013148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:15:30.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:30 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:15:30.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:15:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:30 smithi064 ceph-mon[30223]: 1.4 scrub starts 2024-03-24T12:15:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:30 smithi064 ceph-mon[30223]: 1.4 scrub ok 2024-03-24T12:15:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:30 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2785013148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:15:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:30 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:15:30.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:30 smithi064 ceph-mon[42243]: 1.4 scrub starts 2024-03-24T12:15:30.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:30 smithi064 ceph-mon[42243]: 1.4 scrub ok 2024-03-24T12:15:30.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:30 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2785013148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:15:30.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:30 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:15:30.916 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:31.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-24T12:15:31.370 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:31.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T12:15:31.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:31.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T12:15:31.492 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: 2.14 scrub starts 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: 2.14 scrub ok 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2785013148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: osdmap e518: 8 total, 8 up, 8 in 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: 2.14 scrub starts 2024-03-24T12:15:31.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: 2.14 scrub ok 2024-03-24T12:15:31.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T12:15:31.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T12:15:31.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2785013148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:15:31.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: osdmap e518: 8 total, 8 up, 8 in 2024-03-24T12:15:31.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:15:31.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:15:31.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:31 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:15:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T12:15:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T12:15:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: 2.14 scrub starts 2024-03-24T12:15:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: 2.14 scrub ok 2024-03-24T12:15:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T12:15:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T12:15:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2785013148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:15:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: osdmap e518: 8 total, 8 up, 8 in 2024-03-24T12:15:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:15:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:15:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:31 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:15:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:32 smithi193 ceph-mon[33593]: 4.a scrub starts 2024-03-24T12:15:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:32 smithi193 ceph-mon[33593]: 4.a scrub ok 2024-03-24T12:15:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:32 smithi193 ceph-mon[33593]: 2.16 scrub starts 2024-03-24T12:15:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:32 smithi193 ceph-mon[33593]: 2.16 scrub ok 2024-03-24T12:15:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:32 smithi193 ceph-mon[33593]: 6.4 scrub starts 2024-03-24T12:15:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:32 smithi193 ceph-mon[33593]: 6.4 scrub ok 2024-03-24T12:15:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:32 smithi193 ceph-mon[33593]: osdmap e519: 8 total, 8 up, 8 in 2024-03-24T12:15:32.613 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:32.613 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:32.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[30223]: 4.a scrub starts 2024-03-24T12:15:32.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[30223]: 4.a scrub ok 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[30223]: 2.16 scrub starts 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[30223]: 2.16 scrub ok 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[30223]: 6.4 scrub starts 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[30223]: 6.4 scrub ok 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[30223]: osdmap e519: 8 total, 8 up, 8 in 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[42243]: 4.a scrub starts 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[42243]: 4.a scrub ok 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[42243]: 2.16 scrub starts 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[42243]: 2.16 scrub ok 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[42243]: 6.4 scrub starts 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[42243]: 6.4 scrub ok 2024-03-24T12:15:32.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:32 smithi064 ceph-mon[42243]: osdmap e519: 8 total, 8 up, 8 in 2024-03-24T12:15:33.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:15:33.479 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:33 smithi193 ceph-mon[33593]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:15:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:33 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T12:15:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:33 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T12:15:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:33 smithi193 ceph-mon[33593]: 5.10 scrub starts 2024-03-24T12:15:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:33 smithi193 ceph-mon[33593]: 5.10 scrub ok 2024-03-24T12:15:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[30223]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:15:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T12:15:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T12:15:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[30223]: 5.10 scrub starts 2024-03-24T12:15:33.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[30223]: 5.10 scrub ok 2024-03-24T12:15:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[42243]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:15:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T12:15:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T12:15:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[42243]: 5.10 scrub starts 2024-03-24T12:15:33.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:33 smithi064 ceph-mon[42243]: 5.10 scrub ok 2024-03-24T12:15:34.343 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-24T12:15:34.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:34 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4182036448' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:15:34.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:34 smithi193 ceph-mon[33593]: 2.15 scrub starts 2024-03-24T12:15:34.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:34 smithi193 ceph-mon[33593]: 2.15 scrub ok 2024-03-24T12:15:34.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:34 smithi193 ceph-mon[33593]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T12:15:34.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:34 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:15:34.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:34 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:15:34.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4182036448' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:15:34.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[30223]: 2.15 scrub starts 2024-03-24T12:15:34.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[30223]: 2.15 scrub ok 2024-03-24T12:15:34.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[30223]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T12:15:34.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:15:34.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:15:34.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4182036448' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:15:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[42243]: 2.15 scrub starts 2024-03-24T12:15:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[42243]: 2.15 scrub ok 2024-03-24T12:15:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[42243]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T12:15:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:15:34.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:34 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:15:35.115 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:35.115 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:35.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-24T12:15:35.498 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:35 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T12:15:35.512 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:35 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T12:15:35.513 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:35 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4182036448' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:15:35.513 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:35 smithi193 ceph-mon[33593]: osdmap e520: 8 total, 8 up, 8 in 2024-03-24T12:15:35.513 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:35 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T12:15:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:35 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T12:15:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:35 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4182036448' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:15:35.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:35 smithi064 ceph-mon[30223]: osdmap e520: 8 total, 8 up, 8 in 2024-03-24T12:15:35.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:35 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T12:15:35.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:35 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T12:15:35.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:35 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4182036448' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:15:35.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:35 smithi064 ceph-mon[42243]: osdmap e520: 8 total, 8 up, 8 in 2024-03-24T12:15:35.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T12:15:36.090 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:36.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:36 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T12:15:36.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:36 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T12:15:36.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:36 smithi193 ceph-mon[33593]: 6.3 scrub starts 2024-03-24T12:15:36.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:36 smithi193 ceph-mon[33593]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:36.477 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:36 smithi193 ceph-mon[33593]: 6.3 scrub ok 2024-03-24T12:15:36.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T12:15:36.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T12:15:36.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[30223]: 6.3 scrub starts 2024-03-24T12:15:36.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[30223]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:36.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[30223]: 6.3 scrub ok 2024-03-24T12:15:36.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T12:15:36.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T12:15:36.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[42243]: 6.3 scrub starts 2024-03-24T12:15:36.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[42243]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:36.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:36 smithi064 ceph-mon[42243]: 6.3 scrub ok 2024-03-24T12:15:37.581 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:37 smithi193 ceph-mon[33593]: 5.8 deep-scrub starts 2024-03-24T12:15:37.581 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:37 smithi193 ceph-mon[33593]: 5.8 deep-scrub ok 2024-03-24T12:15:37.581 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:37 smithi193 ceph-mon[33593]: osdmap e521: 8 total, 8 up, 8 in 2024-03-24T12:15:37.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:15:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:15:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:37 smithi064 ceph-mon[30223]: 5.8 deep-scrub starts 2024-03-24T12:15:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:37 smithi064 ceph-mon[30223]: 5.8 deep-scrub ok 2024-03-24T12:15:37.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:37 smithi064 ceph-mon[30223]: osdmap e521: 8 total, 8 up, 8 in 2024-03-24T12:15:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:37 smithi064 ceph-mon[42243]: 5.8 deep-scrub starts 2024-03-24T12:15:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:37 smithi064 ceph-mon[42243]: 5.8 deep-scrub ok 2024-03-24T12:15:37.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:37 smithi064 ceph-mon[42243]: osdmap e521: 8 total, 8 up, 8 in 2024-03-24T12:15:37.868 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:37.869 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:38.358 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-03-24T12:15:38.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:15:38.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T12:15:38.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T12:15:38.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: 5.d scrub starts 2024-03-24T12:15:38.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: 5.d scrub ok 2024-03-24T12:15:38.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: 4.12 scrub starts 2024-03-24T12:15:38.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: 4.12 scrub ok 2024-03-24T12:15:38.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:38.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2678582256' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:38.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: from='client.36182 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:38.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:38 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:38.515 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T12:15:38.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: 5.d scrub starts 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: 5.d scrub ok 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: 4.12 scrub starts 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: 4.12 scrub ok 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2678582256' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: from='client.36182 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:38.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: 5.d scrub starts 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: 5.d scrub ok 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: 4.12 scrub starts 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: 4.12 scrub ok 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2678582256' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: from='client.36182 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:38.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:38 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:38.756 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:15:39.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-24T12:15:39.436 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:39.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[30223]: 5.16 scrub starts 2024-03-24T12:15:39.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[30223]: 5.16 scrub ok 2024-03-24T12:15:39.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T12:15:39.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[30223]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:15:39.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T12:15:39.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[30223]: osdmap e522: 8 total, 8 up, 8 in 2024-03-24T12:15:39.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[42243]: 5.16 scrub starts 2024-03-24T12:15:39.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[42243]: 5.16 scrub ok 2024-03-24T12:15:39.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T12:15:39.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[42243]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:15:39.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T12:15:39.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:39 smithi064 ceph-mon[42243]: osdmap e522: 8 total, 8 up, 8 in 2024-03-24T12:15:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:39 smithi193 ceph-mon[33593]: 5.16 scrub starts 2024-03-24T12:15:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:39 smithi193 ceph-mon[33593]: 5.16 scrub ok 2024-03-24T12:15:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:39 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T12:15:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:39 smithi193 ceph-mon[33593]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:15:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:39 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T12:15:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:39 smithi193 ceph-mon[33593]: osdmap e522: 8 total, 8 up, 8 in 2024-03-24T12:15:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:40 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T12:15:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:40 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T12:15:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:40 smithi193 ceph-mon[33593]: 2.1c scrub starts 2024-03-24T12:15:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:40 smithi193 ceph-mon[33593]: 2.1c scrub ok 2024-03-24T12:15:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:40 smithi193 ceph-mon[33593]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:40.616 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-03-24T12:15:40.616 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:15:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T12:15:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T12:15:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[30223]: 2.1c scrub starts 2024-03-24T12:15:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[30223]: 2.1c scrub ok 2024-03-24T12:15:40.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[30223]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:40.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T12:15:40.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T12:15:40.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[42243]: 2.1c scrub starts 2024-03-24T12:15:40.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[42243]: 2.1c scrub ok 2024-03-24T12:15:40.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:40 smithi064 ceph-mon[42243]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:41 smithi064 ceph-mon[30223]: 6.a deep-scrub starts 2024-03-24T12:15:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:41 smithi064 ceph-mon[30223]: 6.e scrub starts 2024-03-24T12:15:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:41 smithi064 ceph-mon[30223]: 6.a deep-scrub ok 2024-03-24T12:15:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:41 smithi064 ceph-mon[30223]: 6.e scrub ok 2024-03-24T12:15:41.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:41 smithi064 ceph-mon[42243]: 6.a deep-scrub starts 2024-03-24T12:15:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:41 smithi064 ceph-mon[42243]: 6.e scrub starts 2024-03-24T12:15:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:41 smithi064 ceph-mon[42243]: 6.a deep-scrub ok 2024-03-24T12:15:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:41 smithi064 ceph-mon[42243]: 6.e scrub ok 2024-03-24T12:15:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:41 smithi193 ceph-mon[33593]: 6.a deep-scrub starts 2024-03-24T12:15:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:41 smithi193 ceph-mon[33593]: 6.e scrub starts 2024-03-24T12:15:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:41 smithi193 ceph-mon[33593]: 6.a deep-scrub ok 2024-03-24T12:15:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:41 smithi193 ceph-mon[33593]: 6.e scrub ok 2024-03-24T12:15:42.058 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-24T12:15:42.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[30223]: 5.b scrub starts 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[30223]: 5.b scrub ok 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1373130310' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[30223]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1373130310' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[30223]: osdmap e523: 8 total, 8 up, 8 in 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T12:15:42.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T12:15:42.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[42243]: 5.b scrub starts 2024-03-24T12:15:42.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[42243]: 5.b scrub ok 2024-03-24T12:15:42.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1373130310' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:42.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[42243]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-03-24T12:15:42.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1373130310' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:15:42.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:42 smithi064 ceph-mon[42243]: osdmap e523: 8 total, 8 up, 8 in 2024-03-24T12:15:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:42 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T12:15:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:42 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T12:15:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:42 smithi193 ceph-mon[33593]: 5.b scrub starts 2024-03-24T12:15:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:42 smithi193 ceph-mon[33593]: 5.b scrub ok 2024-03-24T12:15:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1373130310' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:15:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:42 smithi193 ceph-mon[33593]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-03-24T12:15:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:42 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1373130310' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:15:42.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:42 smithi193 ceph-mon[33593]: osdmap e523: 8 total, 8 up, 8 in 2024-03-24T12:15:42.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-24T12:15:43.116 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:43.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:43 smithi064 ceph-mon[30223]: 2.d scrub starts 2024-03-24T12:15:43.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:43 smithi064 ceph-mon[30223]: 2.d scrub ok 2024-03-24T12:15:43.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:43.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:43 smithi064 ceph-mon[30223]: osdmap e524: 8 total, 8 up, 8 in 2024-03-24T12:15:43.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:43 smithi064 ceph-mon[42243]: 2.d scrub starts 2024-03-24T12:15:43.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:43 smithi064 ceph-mon[42243]: 2.d scrub ok 2024-03-24T12:15:43.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:43.546 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:43 smithi064 ceph-mon[42243]: osdmap e524: 8 total, 8 up, 8 in 2024-03-24T12:15:43.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:43 smithi193 ceph-mon[33593]: 2.d scrub starts 2024-03-24T12:15:43.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:43 smithi193 ceph-mon[33593]: 2.d scrub ok 2024-03-24T12:15:43.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:43.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:43 smithi193 ceph-mon[33593]: osdmap e524: 8 total, 8 up, 8 in 2024-03-24T12:15:44.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[30223]: 4.1e scrub starts 2024-03-24T12:15:44.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[30223]: 4.1e scrub ok 2024-03-24T12:15:44.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[30223]: 1.6 scrub starts 2024-03-24T12:15:44.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[30223]: 1.6 scrub ok 2024-03-24T12:15:44.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[30223]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:15:44.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[42243]: 4.1e scrub starts 2024-03-24T12:15:44.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[42243]: 4.1e scrub ok 2024-03-24T12:15:44.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[42243]: 1.6 scrub starts 2024-03-24T12:15:44.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[42243]: 1.6 scrub ok 2024-03-24T12:15:44.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:44 smithi064 ceph-mon[42243]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:15:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:44 smithi193 ceph-mon[33593]: 4.1e scrub starts 2024-03-24T12:15:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:44 smithi193 ceph-mon[33593]: 4.1e scrub ok 2024-03-24T12:15:44.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:44 smithi193 ceph-mon[33593]: 1.6 scrub starts 2024-03-24T12:15:44.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:44 smithi193 ceph-mon[33593]: 1.6 scrub ok 2024-03-24T12:15:44.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:44 smithi193 ceph-mon[33593]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:15:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.7 to 1 (10000) 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[30223]: 2.12 scrub starts 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[30223]: 2.12 scrub ok 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[30223]: 6.8 scrub starts 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[30223]: 6.8 scrub ok 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/11050353' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[30223]: from='client.36194 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[42243]: 2.12 scrub starts 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[42243]: 2.12 scrub ok 2024-03-24T12:15:45.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[42243]: 6.8 scrub starts 2024-03-24T12:15:45.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[42243]: 6.8 scrub ok 2024-03-24T12:15:45.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/11050353' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:45.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:45 smithi064 ceph-mon[42243]: from='client.36194 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:45 smithi193 ceph-mon[33593]: 2.12 scrub starts 2024-03-24T12:15:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:45 smithi193 ceph-mon[33593]: 2.12 scrub ok 2024-03-24T12:15:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:45 smithi193 ceph-mon[33593]: 6.8 scrub starts 2024-03-24T12:15:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:45 smithi193 ceph-mon[33593]: 6.8 scrub ok 2024-03-24T12:15:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:45 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/11050353' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:45.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:45 smithi193 ceph-mon[33593]: from='client.36194 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:46.205 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-24T12:15:46.460 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:46 smithi064 ceph-mon[30223]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:15:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:46 smithi064 ceph-mon[30223]: osdmap e525: 8 total, 8 up, 8 in 2024-03-24T12:15:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:46 smithi064 ceph-mon[30223]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:46.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:46 smithi064 ceph-mon[42243]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:15:46.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:46 smithi064 ceph-mon[42243]: osdmap e525: 8 total, 8 up, 8 in 2024-03-24T12:15:46.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:46 smithi064 ceph-mon[42243]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:46.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:46 smithi193 ceph-mon[33593]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:15:46.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:46 smithi193 ceph-mon[33593]: osdmap e525: 8 total, 8 up, 8 in 2024-03-24T12:15:46.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:46 smithi193 ceph-mon[33593]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:47.716 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:47 smithi064 ceph-mon[30223]: 2.18 deep-scrub starts 2024-03-24T12:15:47.716 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:47 smithi064 ceph-mon[30223]: 2.18 deep-scrub ok 2024-03-24T12:15:47.716 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:47 smithi064 ceph-mon[30223]: osdmap e526: 8 total, 8 up, 8 in 2024-03-24T12:15:47.717 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:15:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:15:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:47.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:47 smithi064 ceph-mon[42243]: 2.18 deep-scrub starts 2024-03-24T12:15:47.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:47 smithi064 ceph-mon[42243]: 2.18 deep-scrub ok 2024-03-24T12:15:47.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:47 smithi064 ceph-mon[42243]: osdmap e526: 8 total, 8 up, 8 in 2024-03-24T12:15:47.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:47 smithi193 ceph-mon[33593]: 2.18 deep-scrub starts 2024-03-24T12:15:47.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:47 smithi193 ceph-mon[33593]: 2.18 deep-scrub ok 2024-03-24T12:15:47.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:47 smithi193 ceph-mon[33593]: osdmap e526: 8 total, 8 up, 8 in 2024-03-24T12:15:48.396 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:15:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:15:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:48 smithi064 ceph-mon[30223]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T12:15:48.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:48 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:48 smithi064 ceph-mon[42243]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T12:15:48.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:48 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:48.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:48 smithi193 ceph-mon[33593]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T12:15:48.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:48 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:49.409 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-24T12:15:49.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[30223]: 4.b scrub starts 2024-03-24T12:15:49.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[30223]: 4.b scrub ok 2024-03-24T12:15:49.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[30223]: 4.1 scrub starts 2024-03-24T12:15:49.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[30223]: 4.1 scrub ok 2024-03-24T12:15:49.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[30223]: 4.7 deep-scrub starts 2024-03-24T12:15:49.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[30223]: 4.7 deep-scrub ok 2024-03-24T12:15:49.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2284806420' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:49.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[30223]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:49.792 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[42243]: 4.b scrub starts 2024-03-24T12:15:49.792 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[42243]: 4.b scrub ok 2024-03-24T12:15:49.792 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[42243]: 4.1 scrub starts 2024-03-24T12:15:49.792 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[42243]: 4.1 scrub ok 2024-03-24T12:15:49.792 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[42243]: 4.7 deep-scrub starts 2024-03-24T12:15:49.792 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[42243]: 4.7 deep-scrub ok 2024-03-24T12:15:49.792 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2284806420' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:49.792 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:49 smithi064 ceph-mon[42243]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:49.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:49 smithi193 ceph-mon[33593]: 4.b scrub starts 2024-03-24T12:15:49.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:49 smithi193 ceph-mon[33593]: 4.b scrub ok 2024-03-24T12:15:49.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:49 smithi193 ceph-mon[33593]: 4.1 scrub starts 2024-03-24T12:15:49.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:49 smithi193 ceph-mon[33593]: 4.1 scrub ok 2024-03-24T12:15:49.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:49 smithi193 ceph-mon[33593]: 4.7 deep-scrub starts 2024-03-24T12:15:49.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:49 smithi193 ceph-mon[33593]: 4.7 deep-scrub ok 2024-03-24T12:15:49.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:49 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2284806420' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:49.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:49 smithi193 ceph-mon[33593]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:15:50.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-24T12:15:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:50.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: 5.1b scrub starts 2024-03-24T12:15:50.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: 5.1b scrub ok 2024-03-24T12:15:50.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:15:50.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: osdmap e527: 8 total, 8 up, 8 in 2024-03-24T12:15:50.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: 1.9 scrub starts 2024-03-24T12:15:50.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: 1.9 scrub ok 2024-03-24T12:15:50.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:15:50.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: 2.c scrub starts 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: 2.c scrub ok 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[42243]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: 5.1b scrub starts 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: 5.1b scrub ok 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: osdmap e527: 8 total, 8 up, 8 in 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: 1.9 scrub starts 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: 1.9 scrub ok 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:15:50.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: 2.c scrub starts 2024-03-24T12:15:50.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: 2.c scrub ok 2024-03-24T12:15:50.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:50 smithi064 ceph-mon[30223]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:15:50.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: 5.1b scrub starts 2024-03-24T12:15:50.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: 5.1b scrub ok 2024-03-24T12:15:50.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:15:50.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: osdmap e527: 8 total, 8 up, 8 in 2024-03-24T12:15:50.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: 1.9 scrub starts 2024-03-24T12:15:50.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: 1.9 scrub ok 2024-03-24T12:15:50.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:15:50.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:15:50.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: 2.c scrub starts 2024-03-24T12:15:50.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: 2.c scrub ok 2024-03-24T12:15:50.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:50 smithi193 ceph-mon[33593]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:15:51.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 6.10 deep-scrub starts 2024-03-24T12:15:51.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 6.10 deep-scrub ok 2024-03-24T12:15:51.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 5.15 scrub starts 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 5.15 scrub ok 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 2.1e deep-scrub starts 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 2.1e deep-scrub ok 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 4.f scrub starts 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 4.f scrub ok 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: osdmap e528: 8 total, 8 up, 8 in 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 1.7 scrub starts 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[42243]: 1.7 scrub ok 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 6.10 deep-scrub starts 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 6.10 deep-scrub ok 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 5.15 scrub starts 2024-03-24T12:15:51.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 5.15 scrub ok 2024-03-24T12:15:51.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 2.1e deep-scrub starts 2024-03-24T12:15:51.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 2.1e deep-scrub ok 2024-03-24T12:15:51.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 4.f scrub starts 2024-03-24T12:15:51.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 4.f scrub ok 2024-03-24T12:15:51.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: osdmap e528: 8 total, 8 up, 8 in 2024-03-24T12:15:51.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 1.7 scrub starts 2024-03-24T12:15:51.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:51 smithi064 ceph-mon[30223]: 1.7 scrub ok 2024-03-24T12:15:51.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 6.10 deep-scrub starts 2024-03-24T12:15:51.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 6.10 deep-scrub ok 2024-03-24T12:15:51.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 5.15 scrub starts 2024-03-24T12:15:51.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 5.15 scrub ok 2024-03-24T12:15:51.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 2.1e deep-scrub starts 2024-03-24T12:15:51.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 2.1e deep-scrub ok 2024-03-24T12:15:51.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 4.f scrub starts 2024-03-24T12:15:51.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 4.f scrub ok 2024-03-24T12:15:51.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: osdmap e528: 8 total, 8 up, 8 in 2024-03-24T12:15:51.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 1.7 scrub starts 2024-03-24T12:15:51.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:51 smithi193 ceph-mon[33593]: 1.7 scrub ok 2024-03-24T12:15:52.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:15:52.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:15:52.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[30223]: 6.2 deep-scrub starts 2024-03-24T12:15:52.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[30223]: 6.2 deep-scrub ok 2024-03-24T12:15:52.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[30223]: osdmap e529: 8 total, 8 up, 8 in 2024-03-24T12:15:52.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[30223]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:15:52.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[30223]: osdmap e530: 8 total, 8 up, 8 in 2024-03-24T12:15:52.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:15:52.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:15:52.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[42243]: 6.2 deep-scrub starts 2024-03-24T12:15:52.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[42243]: 6.2 deep-scrub ok 2024-03-24T12:15:52.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[42243]: osdmap e529: 8 total, 8 up, 8 in 2024-03-24T12:15:52.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[42243]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:15:52.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:52 smithi064 ceph-mon[42243]: osdmap e530: 8 total, 8 up, 8 in 2024-03-24T12:15:52.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:52 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:15:52.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:52 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:15:52.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:52 smithi193 ceph-mon[33593]: 6.2 deep-scrub starts 2024-03-24T12:15:52.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:52 smithi193 ceph-mon[33593]: 6.2 deep-scrub ok 2024-03-24T12:15:52.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:52 smithi193 ceph-mon[33593]: osdmap e529: 8 total, 8 up, 8 in 2024-03-24T12:15:52.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:52 smithi193 ceph-mon[33593]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:15:52.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:52 smithi193 ceph-mon[33593]: osdmap e530: 8 total, 8 up, 8 in 2024-03-24T12:15:53.439 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.5 to 1 (10000) 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: 6.1d scrub starts 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: 6.17 scrub starts 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: 6.1d scrub ok 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: 2.1a scrub starts 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: 2.1a scrub ok 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: 6.17 scrub ok 2024-03-24T12:15:53.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:53 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1673789358' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: 6.1d scrub starts 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: 6.17 scrub starts 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: 6.1d scrub ok 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: 2.1a scrub starts 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: 2.1a scrub ok 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: 6.17 scrub ok 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1673789358' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:53.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:15:53.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:15:53.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: 6.1d scrub starts 2024-03-24T12:15:53.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: 6.17 scrub starts 2024-03-24T12:15:53.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: 6.1d scrub ok 2024-03-24T12:15:53.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: 2.1a scrub starts 2024-03-24T12:15:53.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: 2.1a scrub ok 2024-03-24T12:15:53.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: 6.17 scrub ok 2024-03-24T12:15:53.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:53 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1673789358' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:54.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-24T12:15:54.338 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:54.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:54 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1673789358' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:15:54.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:54 smithi064 ceph-mon[30223]: osdmap e531: 8 total, 8 up, 8 in 2024-03-24T12:15:54.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:54 smithi064 ceph-mon[30223]: pgmap v1142: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:54.752 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:54 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1673789358' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:15:54.752 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:54 smithi064 ceph-mon[42243]: osdmap e531: 8 total, 8 up, 8 in 2024-03-24T12:15:54.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:54 smithi064 ceph-mon[42243]: pgmap v1142: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:54.754 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:54 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1673789358' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:15:54.754 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:54 smithi193 ceph-mon[33593]: osdmap e531: 8 total, 8 up, 8 in 2024-03-24T12:15:54.754 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:54 smithi193 ceph-mon[33593]: pgmap v1142: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:55.781 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:55 smithi064 ceph-mon[30223]: 4.6 scrub starts 2024-03-24T12:15:55.781 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:55 smithi064 ceph-mon[30223]: 4.6 scrub ok 2024-03-24T12:15:55.781 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:55 smithi064 ceph-mon[42243]: 4.6 scrub starts 2024-03-24T12:15:55.781 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:55 smithi064 ceph-mon[42243]: 4.6 scrub ok 2024-03-24T12:15:55.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:55 smithi193 ceph-mon[33593]: 4.6 scrub starts 2024-03-24T12:15:55.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:55 smithi193 ceph-mon[33593]: 4.6 scrub ok 2024-03-24T12:15:56.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:56 smithi064 ceph-mon[30223]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-24T12:15:56.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:56 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-24T12:15:56.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:56 smithi064 ceph-mon[42243]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-24T12:15:56.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:56 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-24T12:15:56.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:56 smithi193 ceph-mon[33593]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-24T12:15:56.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:56 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-24T12:15:57.468 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-24T12:15:57.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:15:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:57.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[30223]: 6.1f scrub starts 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[30223]: 6.1f scrub ok 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[30223]: osdmap e532: 8 total, 8 up, 8 in 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3291129836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[30223]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[42243]: 6.1f scrub starts 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[42243]: 6.1f scrub ok 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[42243]: osdmap e532: 8 total, 8 up, 8 in 2024-03-24T12:15:57.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3291129836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:57.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:57 smithi064 ceph-mon[42243]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:57 smithi193 ceph-mon[33593]: 6.1f scrub starts 2024-03-24T12:15:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:57 smithi193 ceph-mon[33593]: 6.1f scrub ok 2024-03-24T12:15:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:57 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-24T12:15:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:57 smithi193 ceph-mon[33593]: osdmap e532: 8 total, 8 up, 8 in 2024-03-24T12:15:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:57 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3291129836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:57.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:57 smithi193 ceph-mon[33593]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:15:58.183 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-24T12:15:58.183 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-24T12:15:58.183 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-24T12:15:58.183 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-24T12:15:58.183 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T12:15:58.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T12:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:15:58.467 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:15:58.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: 1.8 scrub starts 2024-03-24T12:15:58.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: 1.8 scrub ok 2024-03-24T12:15:58.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: 2.9 scrub starts 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: 2.9 scrub ok 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: osdmap e533: 8 total, 8 up, 8 in 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: pgmap v1146: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: 1.8 scrub starts 2024-03-24T12:15:58.470 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: 1.8 scrub ok 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: 2.9 scrub starts 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: 2.9 scrub ok 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: osdmap e533: 8 total, 8 up, 8 in 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: pgmap v1146: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:58.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:58 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:58.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: 1.8 scrub starts 2024-03-24T12:15:58.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: 1.8 scrub ok 2024-03-24T12:15:58.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: 2.9 scrub starts 2024-03-24T12:15:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: 2.9 scrub ok 2024-03-24T12:15:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:15:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: osdmap e533: 8 total, 8 up, 8 in 2024-03-24T12:15:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:15:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:15:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: pgmap v1146: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-24T12:15:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:58.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:58 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:59.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[30223]: 4.19 scrub starts 2024-03-24T12:15:59.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[30223]: 4.19 scrub ok 2024-03-24T12:15:59.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[30223]: 1.5 scrub starts 2024-03-24T12:15:59.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[30223]: 1.5 scrub ok 2024-03-24T12:15:59.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[30223]: osdmap e534: 8 total, 8 up, 8 in 2024-03-24T12:15:59.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[30223]: 6.16 deep-scrub starts 2024-03-24T12:15:59.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[30223]: 6.16 deep-scrub ok 2024-03-24T12:15:59.791 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[42243]: 4.19 scrub starts 2024-03-24T12:15:59.791 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[42243]: 4.19 scrub ok 2024-03-24T12:15:59.791 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[42243]: 1.5 scrub starts 2024-03-24T12:15:59.791 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[42243]: 1.5 scrub ok 2024-03-24T12:15:59.791 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[42243]: osdmap e534: 8 total, 8 up, 8 in 2024-03-24T12:15:59.791 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[42243]: 6.16 deep-scrub starts 2024-03-24T12:15:59.791 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:15:59 smithi064 ceph-mon[42243]: 6.16 deep-scrub ok 2024-03-24T12:15:59.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:59 smithi193 ceph-mon[33593]: 4.19 scrub starts 2024-03-24T12:15:59.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:59 smithi193 ceph-mon[33593]: 4.19 scrub ok 2024-03-24T12:15:59.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:59 smithi193 ceph-mon[33593]: 1.5 scrub starts 2024-03-24T12:15:59.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:59 smithi193 ceph-mon[33593]: 1.5 scrub ok 2024-03-24T12:15:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:59 smithi193 ceph-mon[33593]: osdmap e534: 8 total, 8 up, 8 in 2024-03-24T12:15:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:59 smithi193 ceph-mon[33593]: 6.16 deep-scrub starts 2024-03-24T12:15:59.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:15:59 smithi193 ceph-mon[33593]: 6.16 deep-scrub ok 2024-03-24T12:16:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:16:00.589 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":536,"fsid":"06f23456-e9d4-11ee-95cd-87774f69a715","created":"2024-03-24T11:46:40.657065+0000","modified":"2024-03-24T12:16:00.481932+0000","last_up_change":"2024-03-24T12:08:35.127870+0000","last_in_change":"2024-03-24T11:51:43.667453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:49:56.369674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":426,"last_epoch_started":425,"last_epoch_clean":425,"source_version":"83'130","target_version":"83'130"},"last_change":"452","last_force_op_resend":"0","last_force_op_resend_prenautilus":"426","last_force_op_resend_preluminous":"424","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T11:52:13.871974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-24T11:52:14.117018+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:52:15.453744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:52:17.611004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:52:19.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc14a64f-9e6c-41f1-901e-c49fabffb53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2675992571}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2675992571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2675992571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2675992571}]},"public_addr":"172.21.15.64:6801/2675992571","cluster_addr":"172.21.15.64:6802/2675992571","heartbeat_back_addr":"172.21.15.64:6804/2675992571","heartbeat_front_addr":"172.21.15.64:6803/2675992571","state":["exists","up"]},{"osd":1,"uuid":"74587332-f0b2-4905-9888-7bdb604cc0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":841667271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":841667271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":841667271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":841667271}]},"public_addr":"172.21.15.64:6805/841667271","cluster_addr":"172.21.15.64:6806/841667271","heartbeat_back_addr":"172.21.15.64:6808/841667271","heartbeat_front_addr":"172.21.15.64:6807/841667271","state":["exists","up"]},{"osd":2,"uuid":"a0c0ed85-9f33-4198-a1f2-447eb17c364b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":435147390}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":435147390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":435147390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":435147390}]},"public_addr":"172.21.15.64:6809/435147390","cluster_addr":"172.21.15.64:6810/435147390","heartbeat_back_addr":"172.21.15.64:6812/435147390","heartbeat_front_addr":"172.21.15.64:6811/435147390","state":["exists","up"]},{"osd":3,"uuid":"1396f66c-3e0c-4a01-b0a7-6f763989a479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3355318038}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3355318038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3355318038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3355318038}]},"public_addr":"172.21.15.64:6813/3355318038","cluster_addr":"172.21.15.64:6814/3355318038","heartbeat_back_addr":"172.21.15.64:6816/3355318038","heartbeat_front_addr":"172.21.15.64:6815/3355318038","state":["exists","up"]},{"osd":4,"uuid":"50d65d8b-3e90-46a4-b19d-50c3015af40b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":513,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3851339176}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3851339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3851339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3851339176}]},"public_addr":"172.21.15.193:6800/3851339176","cluster_addr":"172.21.15.193:6801/3851339176","heartbeat_back_addr":"172.21.15.193:6803/3851339176","heartbeat_front_addr":"172.21.15.193:6802/3851339176","state":["exists","up"]},{"osd":5,"uuid":"50c5d50e-7e68-487c-9f68-1abb8c927ff6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":41,"last_clean_end":317,"up_from":353,"up_thru":510,"down_at":318,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":1531860860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":1531860860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":1531860860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":1531860860}]},"public_addr":"172.21.15.193:6804/1531860860","cluster_addr":"172.21.15.193:6805/1531860860","heartbeat_back_addr":"172.21.15.193:6807/1531860860","heartbeat_front_addr":"172.21.15.193:6806/1531860860","state":["exists","up"]},{"osd":6,"uuid":"7e74fe39-5707-4c90-ad38-1083cd7a6996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":523,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":2408237278}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":2408237278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":2408237278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":2408237278}]},"public_addr":"172.21.15.193:6808/2408237278","cluster_addr":"172.21.15.193:6809/2408237278","heartbeat_back_addr":"172.21.15.193:6811/2408237278","heartbeat_front_addr":"172.21.15.193:6810/2408237278","state":["exists","up"]},{"osd":7,"uuid":"e568df64-cc9a-43f8-a432-8628d1da1015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":513,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":2649989609}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":2649989609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":2649989609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":2649989609}]},"public_addr":"172.21.15.193:6812/2649989609","cluster_addr":"172.21.15.193:6813/2649989609","heartbeat_back_addr":"172.21.15.193:6815/2649989609","heartbeat_front_addr":"172.21.15.193:6814/2649989609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:16:00.117166+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:57.530335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:14:45.973630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:50.735805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:51.939007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:49.494053+0000","dead_epoch":317},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:14:50.522630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:59.038772+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[0,1,7]}],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":7,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":0}]},{"pgid":"4.14","mappings":[{"from":3,"to":5}]},{"pgid":"4.19","mappings":[{"from":3,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"5.6","mappings":[{"from":7,"to":3}]},{"pgid":"5.13","mappings":[{"from":1,"to":2}]},{"pgid":"5.18","mappings":[{"from":4,"to":3}]},{"pgid":"5.1c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6800/3538503390":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/1277626303":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/3407451558":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/3753895773":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/842059688":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/1610565583":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/2730170969":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/3257265142":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/1459540658":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/629947599":"2024-03-25T11:47:08.661692+0000","172.21.15.64:6800/520335082":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/3351953651":"2024-03-25T11:52:55.117233+0000","172.21.15.64:6800/4027721305":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/4028269665":"2024-03-25T11:47:36.035658+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T12:16:00.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:16:00.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:16:00.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: 4.4 scrub starts 2024-03-24T12:16:00.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: 4.4 scrub ok 2024-03-24T12:16:00.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: 5.1c scrub starts 2024-03-24T12:16:00.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: 5.1c scrub ok 2024-03-24T12:16:00.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: osdmap e535: 8 total, 8 up, 8 in 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: 5.3 scrub starts 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: 5.3 scrub ok 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: pgmap v1149: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: 4.4 scrub starts 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: 4.4 scrub ok 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: 5.1c scrub starts 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: 5.1c scrub ok 2024-03-24T12:16:00.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: osdmap e535: 8 total, 8 up, 8 in 2024-03-24T12:16:00.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: 5.3 scrub starts 2024-03-24T12:16:00.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: 5.3 scrub ok 2024-03-24T12:16:00.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: pgmap v1149: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:00.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:16:00.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:00 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: 4.4 scrub starts 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: 4.4 scrub ok 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: 5.1c scrub starts 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: 5.1c scrub ok 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: osdmap e535: 8 total, 8 up, 8 in 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: 5.3 scrub starts 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: 5.3 scrub ok 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: pgmap v1149: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:00.837 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:16:00.838 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:00 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:16:01.412 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T12:16:01.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T12:16:01.662 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:01.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[30223]: 6.d scrub starts 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[30223]: 4.2 scrub starts 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[30223]: 4.2 scrub ok 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[30223]: 6.d scrub ok 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[30223]: osdmap e536: 8 total, 8 up, 8 in 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1824765585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[30223]: 3.2 scrub starts 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[30223]: 3.2 scrub ok 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[42243]: 6.d scrub starts 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[42243]: 4.2 scrub starts 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[42243]: 4.2 scrub ok 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[42243]: 6.d scrub ok 2024-03-24T12:16:01.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[42243]: osdmap e536: 8 total, 8 up, 8 in 2024-03-24T12:16:01.785 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1824765585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:16:01.785 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[42243]: 3.2 scrub starts 2024-03-24T12:16:01.785 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:01 smithi064 ceph-mon[42243]: 3.2 scrub ok 2024-03-24T12:16:01.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:01 smithi193 ceph-mon[33593]: 6.d scrub starts 2024-03-24T12:16:01.835 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:01 smithi193 ceph-mon[33593]: 4.2 scrub starts 2024-03-24T12:16:01.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:01 smithi193 ceph-mon[33593]: 4.2 scrub ok 2024-03-24T12:16:01.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:01 smithi193 ceph-mon[33593]: 6.d scrub ok 2024-03-24T12:16:01.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:01 smithi193 ceph-mon[33593]: osdmap e536: 8 total, 8 up, 8 in 2024-03-24T12:16:01.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:01 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1824765585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:16:01.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:01 smithi193 ceph-mon[33593]: 3.2 scrub starts 2024-03-24T12:16:01.836 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:01 smithi193 ceph-mon[33593]: 3.2 scrub ok 2024-03-24T12:16:03.229 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:03 smithi064 ceph-mon[30223]: pgmap v1151: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:03.230 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:03 smithi064 ceph-mon[30223]: 4.14 scrub starts 2024-03-24T12:16:03.230 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:03 smithi064 ceph-mon[30223]: 4.14 scrub ok 2024-03-24T12:16:03.230 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:03 smithi064 ceph-mon[42243]: pgmap v1151: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:03.230 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:03 smithi064 ceph-mon[42243]: 4.14 scrub starts 2024-03-24T12:16:03.230 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:03 smithi064 ceph-mon[42243]: 4.14 scrub ok 2024-03-24T12:16:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:03 smithi193 ceph-mon[33593]: pgmap v1151: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:03 smithi193 ceph-mon[33593]: 4.14 scrub starts 2024-03-24T12:16:03.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:03 smithi193 ceph-mon[33593]: 4.14 scrub ok 2024-03-24T12:16:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:16:03.707 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":536,"fsid":"06f23456-e9d4-11ee-95cd-87774f69a715","created":"2024-03-24T11:46:40.657065+0000","modified":"2024-03-24T12:16:00.481932+0000","last_up_change":"2024-03-24T12:08:35.127870+0000","last_in_change":"2024-03-24T11:51:43.667453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:49:56.369674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":426,"last_epoch_started":425,"last_epoch_clean":425,"source_version":"83'130","target_version":"83'130"},"last_change":"452","last_force_op_resend":"0","last_force_op_resend_prenautilus":"426","last_force_op_resend_preluminous":"424","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T11:52:13.871974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-24T11:52:14.117018+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:52:15.453744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:52:17.611004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:52:19.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc14a64f-9e6c-41f1-901e-c49fabffb53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2675992571}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2675992571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2675992571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2675992571}]},"public_addr":"172.21.15.64:6801/2675992571","cluster_addr":"172.21.15.64:6802/2675992571","heartbeat_back_addr":"172.21.15.64:6804/2675992571","heartbeat_front_addr":"172.21.15.64:6803/2675992571","state":["exists","up"]},{"osd":1,"uuid":"74587332-f0b2-4905-9888-7bdb604cc0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":841667271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":841667271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":841667271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":841667271}]},"public_addr":"172.21.15.64:6805/841667271","cluster_addr":"172.21.15.64:6806/841667271","heartbeat_back_addr":"172.21.15.64:6808/841667271","heartbeat_front_addr":"172.21.15.64:6807/841667271","state":["exists","up"]},{"osd":2,"uuid":"a0c0ed85-9f33-4198-a1f2-447eb17c364b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":435147390}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":435147390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":435147390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":435147390}]},"public_addr":"172.21.15.64:6809/435147390","cluster_addr":"172.21.15.64:6810/435147390","heartbeat_back_addr":"172.21.15.64:6812/435147390","heartbeat_front_addr":"172.21.15.64:6811/435147390","state":["exists","up"]},{"osd":3,"uuid":"1396f66c-3e0c-4a01-b0a7-6f763989a479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3355318038}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3355318038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3355318038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3355318038}]},"public_addr":"172.21.15.64:6813/3355318038","cluster_addr":"172.21.15.64:6814/3355318038","heartbeat_back_addr":"172.21.15.64:6816/3355318038","heartbeat_front_addr":"172.21.15.64:6815/3355318038","state":["exists","up"]},{"osd":4,"uuid":"50d65d8b-3e90-46a4-b19d-50c3015af40b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":513,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3851339176}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3851339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3851339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3851339176}]},"public_addr":"172.21.15.193:6800/3851339176","cluster_addr":"172.21.15.193:6801/3851339176","heartbeat_back_addr":"172.21.15.193:6803/3851339176","heartbeat_front_addr":"172.21.15.193:6802/3851339176","state":["exists","up"]},{"osd":5,"uuid":"50c5d50e-7e68-487c-9f68-1abb8c927ff6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":41,"last_clean_end":317,"up_from":353,"up_thru":510,"down_at":318,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":1531860860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":1531860860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":1531860860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":1531860860}]},"public_addr":"172.21.15.193:6804/1531860860","cluster_addr":"172.21.15.193:6805/1531860860","heartbeat_back_addr":"172.21.15.193:6807/1531860860","heartbeat_front_addr":"172.21.15.193:6806/1531860860","state":["exists","up"]},{"osd":6,"uuid":"7e74fe39-5707-4c90-ad38-1083cd7a6996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":523,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":2408237278}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":2408237278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":2408237278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":2408237278}]},"public_addr":"172.21.15.193:6808/2408237278","cluster_addr":"172.21.15.193:6809/2408237278","heartbeat_back_addr":"172.21.15.193:6811/2408237278","heartbeat_front_addr":"172.21.15.193:6810/2408237278","state":["exists","up"]},{"osd":7,"uuid":"e568df64-cc9a-43f8-a432-8628d1da1015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":513,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":2649989609}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":2649989609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":2649989609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":2649989609}]},"public_addr":"172.21.15.193:6812/2649989609","cluster_addr":"172.21.15.193:6813/2649989609","heartbeat_back_addr":"172.21.15.193:6815/2649989609","heartbeat_front_addr":"172.21.15.193:6814/2649989609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:16:00.117166+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:57.530335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:14:45.973630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:50.735805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:51.939007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:49.494053+0000","dead_epoch":317},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:14:50.522630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:15:59.038772+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[0,1,7]}],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":7,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":0}]},{"pgid":"4.14","mappings":[{"from":3,"to":5}]},{"pgid":"4.19","mappings":[{"from":3,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"5.6","mappings":[{"from":7,"to":3}]},{"pgid":"5.13","mappings":[{"from":1,"to":2}]},{"pgid":"5.18","mappings":[{"from":4,"to":3}]},{"pgid":"5.1c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6800/3538503390":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/1277626303":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/3407451558":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/3753895773":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/842059688":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/1610565583":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/2730170969":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/3257265142":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/1459540658":"2024-03-25T11:52:55.117233+0000","172.21.15.64:0/629947599":"2024-03-25T11:47:08.661692+0000","172.21.15.64:6800/520335082":"2024-03-25T11:47:08.661692+0000","172.21.15.64:0/3351953651":"2024-03-25T11:52:55.117233+0000","172.21.15.64:6800/4027721305":"2024-03-25T11:47:36.035658+0000","172.21.15.64:0/4028269665":"2024-03-25T11:47:36.035658+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T12:16:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:04 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:16:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:04 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:16:04.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:04 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1223122318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:16:04.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:04 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:16:04.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:04 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:16:04.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:04 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1223122318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:16:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:04 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:16:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:04 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:16:04.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:04 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1223122318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:16:04.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T12:16:04.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T12:16:04.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T12:16:04.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T12:16:04.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T12:16:04.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T12:16:04.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T12:16:04.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T12:16:04.993 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:04.994 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:04.994 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:04.997 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:04.997 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:04.999 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:04.999 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:05.003 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:05 smithi193 ceph-mon[33593]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2024-03-24T12:16:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:05 smithi193 ceph-mon[33593]: osdmap e537: 8 total, 8 up, 8 in 2024-03-24T12:16:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:05 smithi193 ceph-mon[33593]: 4.0 scrub starts 2024-03-24T12:16:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:05 smithi193 ceph-mon[33593]: 4.0 scrub ok 2024-03-24T12:16:05.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:05 smithi193 ceph-mon[33593]: 4.5 scrub starts 2024-03-24T12:16:05.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:05 smithi193 ceph-mon[33593]: 4.5 scrub ok 2024-03-24T12:16:05.433 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[42243]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[42243]: osdmap e537: 8 total, 8 up, 8 in 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[42243]: 4.0 scrub starts 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[42243]: 4.0 scrub ok 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[42243]: 4.5 scrub starts 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[42243]: 4.5 scrub ok 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[30223]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[30223]: osdmap e537: 8 total, 8 up, 8 in 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[30223]: 4.0 scrub starts 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[30223]: 4.0 scrub ok 2024-03-24T12:16:05.434 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[30223]: 4.5 scrub starts 2024-03-24T12:16:05.435 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:05 smithi064 ceph-mon[30223]: 4.5 scrub ok 2024-03-24T12:16:06.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:06 smithi193 ceph-mon[33593]: 2.10 scrub starts 2024-03-24T12:16:06.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:06 smithi193 ceph-mon[33593]: 2.10 scrub ok 2024-03-24T12:16:06.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:06 smithi064 ceph-mon[30223]: 2.10 scrub starts 2024-03-24T12:16:06.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:06 smithi064 ceph-mon[30223]: 2.10 scrub ok 2024-03-24T12:16:06.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:06 smithi064 ceph-mon[42243]: 2.10 scrub starts 2024-03-24T12:16:06.381 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:06 smithi064 ceph-mon[42243]: 2.10 scrub ok 2024-03-24T12:16:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[30223]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 op/s; 17 B/s, 1 objects/s recovering 2024-03-24T12:16:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[30223]: 6.c scrub starts 2024-03-24T12:16:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[30223]: 6.c scrub ok 2024-03-24T12:16:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[30223]: 4.e scrub starts 2024-03-24T12:16:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[30223]: 4.e scrub ok 2024-03-24T12:16:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[30223]: 2.2 deep-scrub starts 2024-03-24T12:16:07.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[30223]: 2.2 deep-scrub ok 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[30223]: osdmap e538: 8 total, 8 up, 8 in 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[42243]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 op/s; 17 B/s, 1 objects/s recovering 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[42243]: 6.c scrub starts 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[42243]: 6.c scrub ok 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[42243]: 4.e scrub starts 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[42243]: 4.e scrub ok 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[42243]: 2.2 deep-scrub starts 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[42243]: 2.2 deep-scrub ok 2024-03-24T12:16:07.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-mon[42243]: osdmap e538: 8 total, 8 up, 8 in 2024-03-24T12:16:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:07 smithi193 ceph-mon[33593]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 op/s; 17 B/s, 1 objects/s recovering 2024-03-24T12:16:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:07 smithi193 ceph-mon[33593]: 6.c scrub starts 2024-03-24T12:16:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:07 smithi193 ceph-mon[33593]: 6.c scrub ok 2024-03-24T12:16:07.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:07 smithi193 ceph-mon[33593]: 4.e scrub starts 2024-03-24T12:16:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:07 smithi193 ceph-mon[33593]: 4.e scrub ok 2024-03-24T12:16:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:07 smithi193 ceph-mon[33593]: 2.2 deep-scrub starts 2024-03-24T12:16:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:07 smithi193 ceph-mon[33593]: 2.2 deep-scrub ok 2024-03-24T12:16:07.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:07 smithi193 ceph-mon[33593]: osdmap e538: 8 total, 8 up, 8 in 2024-03-24T12:16:07.633 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:16:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:16:08.206 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:08 smithi064 ceph-mon[30223]: 6.12 scrub starts 2024-03-24T12:16:08.207 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:08 smithi064 ceph-mon[30223]: 6.12 scrub ok 2024-03-24T12:16:08.207 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:08 smithi064 ceph-mon[42243]: 6.12 scrub starts 2024-03-24T12:16:08.207 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:08 smithi064 ceph-mon[42243]: 6.12 scrub ok 2024-03-24T12:16:08.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:16:08 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:16:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:08 smithi193 ceph-mon[33593]: 6.12 scrub starts 2024-03-24T12:16:08.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:08 smithi193 ceph-mon[33593]: 6.12 scrub ok 2024-03-24T12:16:09.227 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:09 smithi064 ceph-mon[30223]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 15 B/s, 0 objects/s recovering 2024-03-24T12:16:09.228 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:09 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:09.228 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:09 smithi064 ceph-mon[30223]: 6.1 scrub starts 2024-03-24T12:16:09.228 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:09 smithi064 ceph-mon[30223]: 6.1 scrub ok 2024-03-24T12:16:09.228 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:09 smithi064 ceph-mon[42243]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 15 B/s, 0 objects/s recovering 2024-03-24T12:16:09.228 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:09 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:09.228 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:09 smithi064 ceph-mon[42243]: 6.1 scrub starts 2024-03-24T12:16:09.228 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:09 smithi064 ceph-mon[42243]: 6.1 scrub ok 2024-03-24T12:16:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:09 smithi193 ceph-mon[33593]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 15 B/s, 0 objects/s recovering 2024-03-24T12:16:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:09 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:09.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:09 smithi193 ceph-mon[33593]: 6.1 scrub starts 2024-03-24T12:16:09.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:09 smithi193 ceph-mon[33593]: 6.1 scrub ok 2024-03-24T12:16:10.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:10 smithi064 ceph-mon[30223]: 4.15 scrub starts 2024-03-24T12:16:10.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:10 smithi064 ceph-mon[30223]: 4.15 scrub ok 2024-03-24T12:16:10.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:10 smithi064 ceph-mon[30223]: 4.13 scrub starts 2024-03-24T12:16:10.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:10 smithi064 ceph-mon[30223]: 4.13 scrub ok 2024-03-24T12:16:10.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:10 smithi064 ceph-mon[42243]: 4.15 scrub starts 2024-03-24T12:16:10.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:10 smithi064 ceph-mon[42243]: 4.15 scrub ok 2024-03-24T12:16:10.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:10 smithi064 ceph-mon[42243]: 4.13 scrub starts 2024-03-24T12:16:10.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:10 smithi064 ceph-mon[42243]: 4.13 scrub ok 2024-03-24T12:16:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:10 smithi193 ceph-mon[33593]: 4.15 scrub starts 2024-03-24T12:16:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:10 smithi193 ceph-mon[33593]: 4.15 scrub ok 2024-03-24T12:16:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:10 smithi193 ceph-mon[33593]: 4.13 scrub starts 2024-03-24T12:16:10.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:10 smithi193 ceph-mon[33593]: 4.13 scrub ok 2024-03-24T12:16:10.911 INFO:teuthology.orchestra.run.smithi064.stdout:261993005347 2024-03-24T12:16:10.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T12:16:11.169 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[30223]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[30223]: 5.7 scrub starts 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[30223]: 5.7 scrub ok 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[30223]: 4.1b scrub starts 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[30223]: 4.1b scrub ok 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[42243]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[42243]: 5.7 scrub starts 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[42243]: 5.7 scrub ok 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[42243]: 4.1b scrub starts 2024-03-24T12:16:11.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:11 smithi064 ceph-mon[42243]: 4.1b scrub ok 2024-03-24T12:16:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:11 smithi193 ceph-mon[33593]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-24T12:16:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:11 smithi193 ceph-mon[33593]: 5.7 scrub starts 2024-03-24T12:16:11.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:11 smithi193 ceph-mon[33593]: 5.7 scrub ok 2024-03-24T12:16:11.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:11 smithi193 ceph-mon[33593]: 4.1b scrub starts 2024-03-24T12:16:11.587 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:11 smithi193 ceph-mon[33593]: 4.1b scrub ok 2024-03-24T12:16:12.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:12 smithi064 ceph-mon[42243]: 6.15 scrub starts 2024-03-24T12:16:12.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:12 smithi064 ceph-mon[42243]: 6.15 scrub ok 2024-03-24T12:16:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:12 smithi064 ceph-mon[30223]: 6.15 scrub starts 2024-03-24T12:16:12.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:12 smithi064 ceph-mon[30223]: 6.15 scrub ok 2024-03-24T12:16:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:12 smithi193 ceph-mon[33593]: 6.15 scrub starts 2024-03-24T12:16:12.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:12 smithi193 ceph-mon[33593]: 6.15 scrub ok 2024-03-24T12:16:13.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:13 smithi064 ceph-mon[42243]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:16:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:13 smithi064 ceph-mon[42243]: 6.13 scrub starts 2024-03-24T12:16:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:13 smithi064 ceph-mon[42243]: 6.13 scrub ok 2024-03-24T12:16:13.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:13 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:13.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:13 smithi064 ceph-mon[30223]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:16:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:13 smithi064 ceph-mon[30223]: 6.13 scrub starts 2024-03-24T12:16:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:13 smithi064 ceph-mon[30223]: 6.13 scrub ok 2024-03-24T12:16:13.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:13 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:13 smithi193 ceph-mon[33593]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:16:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:13 smithi193 ceph-mon[33593]: 6.13 scrub starts 2024-03-24T12:16:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:13 smithi193 ceph-mon[33593]: 6.13 scrub ok 2024-03-24T12:16:13.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:13 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:14.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[30223]: 4.16 scrub starts 2024-03-24T12:16:14.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[30223]: 4.16 scrub ok 2024-03-24T12:16:14.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[30223]: 4.17 scrub starts 2024-03-24T12:16:14.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[30223]: 4.17 scrub ok 2024-03-24T12:16:14.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[30223]: 5.9 scrub starts 2024-03-24T12:16:14.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[30223]: 5.9 scrub ok 2024-03-24T12:16:14.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[42243]: 4.16 scrub starts 2024-03-24T12:16:14.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[42243]: 4.16 scrub ok 2024-03-24T12:16:14.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[42243]: 4.17 scrub starts 2024-03-24T12:16:14.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[42243]: 4.17 scrub ok 2024-03-24T12:16:14.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[42243]: 5.9 scrub starts 2024-03-24T12:16:14.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:14 smithi064 ceph-mon[42243]: 5.9 scrub ok 2024-03-24T12:16:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:14 smithi193 ceph-mon[33593]: 4.16 scrub starts 2024-03-24T12:16:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:14 smithi193 ceph-mon[33593]: 4.16 scrub ok 2024-03-24T12:16:14.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:14 smithi193 ceph-mon[33593]: 4.17 scrub starts 2024-03-24T12:16:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:14 smithi193 ceph-mon[33593]: 4.17 scrub ok 2024-03-24T12:16:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:14 smithi193 ceph-mon[33593]: 5.9 scrub starts 2024-03-24T12:16:14.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:14 smithi193 ceph-mon[33593]: 5.9 scrub ok 2024-03-24T12:16:15.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:15 smithi064 ceph-mon[30223]: pgmap v1159: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:15.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:15 smithi064 ceph-mon[30223]: 6.7 scrub starts 2024-03-24T12:16:15.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:15 smithi064 ceph-mon[30223]: 6.7 scrub ok 2024-03-24T12:16:15.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:15 smithi064 ceph-mon[42243]: pgmap v1159: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:15.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:15 smithi064 ceph-mon[42243]: 6.7 scrub starts 2024-03-24T12:16:15.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:15 smithi064 ceph-mon[42243]: 6.7 scrub ok 2024-03-24T12:16:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:15 smithi193 ceph-mon[33593]: pgmap v1159: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:15 smithi193 ceph-mon[33593]: 6.7 scrub starts 2024-03-24T12:16:15.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:15 smithi193 ceph-mon[33593]: 6.7 scrub ok 2024-03-24T12:16:16.727 INFO:teuthology.orchestra.run.smithi064.stdout:55834575171 2024-03-24T12:16:16.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T12:16:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[30223]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[30223]: 5.17 scrub starts 2024-03-24T12:16:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[30223]: 5.17 scrub ok 2024-03-24T12:16:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[30223]: 4.18 scrub starts 2024-03-24T12:16:17.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[30223]: 4.18 scrub ok 2024-03-24T12:16:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[42243]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[42243]: 5.17 scrub starts 2024-03-24T12:16:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[42243]: 5.17 scrub ok 2024-03-24T12:16:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[42243]: 4.18 scrub starts 2024-03-24T12:16:17.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-mon[42243]: 4.18 scrub ok 2024-03-24T12:16:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:17 smithi193 ceph-mon[33593]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:17 smithi193 ceph-mon[33593]: 5.17 scrub starts 2024-03-24T12:16:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:17 smithi193 ceph-mon[33593]: 5.17 scrub ok 2024-03-24T12:16:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:17 smithi193 ceph-mon[33593]: 4.18 scrub starts 2024-03-24T12:16:17.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:17 smithi193 ceph-mon[33593]: 4.18 scrub ok 2024-03-24T12:16:17.801 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:17 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:16:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:16:17.873 INFO:teuthology.orchestra.run.smithi064.stdout:141733921076 2024-03-24T12:16:17.874 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T12:16:18.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:18 smithi064 ceph-mon[42243]: 2.a scrub starts 2024-03-24T12:16:18.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:18 smithi064 ceph-mon[42243]: 2.a scrub ok 2024-03-24T12:16:18.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:18 smithi064 ceph-mon[42243]: purged_snaps scrub starts 2024-03-24T12:16:18.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:18 smithi064 ceph-mon[42243]: purged_snaps scrub ok 2024-03-24T12:16:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:18 smithi064 ceph-mon[30223]: 2.a scrub starts 2024-03-24T12:16:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:18 smithi064 ceph-mon[30223]: 2.a scrub ok 2024-03-24T12:16:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:18 smithi064 ceph-mon[30223]: purged_snaps scrub starts 2024-03-24T12:16:18.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:18 smithi064 ceph-mon[30223]: purged_snaps scrub ok 2024-03-24T12:16:18.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:16:18 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:16:18.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:18 smithi193 ceph-mon[33593]: 2.a scrub starts 2024-03-24T12:16:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:18 smithi193 ceph-mon[33593]: 2.a scrub ok 2024-03-24T12:16:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:18 smithi193 ceph-mon[33593]: purged_snaps scrub starts 2024-03-24T12:16:18.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:18 smithi193 ceph-mon[33593]: purged_snaps scrub ok 2024-03-24T12:16:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:34359738696 2024-03-24T12:16:18.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T12:16:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:19.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: 2.19 scrub starts 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: 2.19 scrub ok 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: osdmap e539: 8 total, 8 up, 8 in 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: 5.c scrub starts 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: 5.c scrub ok 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: 4.1c scrub starts 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[30223]: 4.1c scrub ok 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: 2.19 scrub starts 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: 2.19 scrub ok 2024-03-24T12:16:19.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: osdmap e539: 8 total, 8 up, 8 in 2024-03-24T12:16:19.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:19.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: 5.c scrub starts 2024-03-24T12:16:19.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: 5.c scrub ok 2024-03-24T12:16:19.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: 4.1c scrub starts 2024-03-24T12:16:19.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:19 smithi064 ceph-mon[42243]: 4.1c scrub ok 2024-03-24T12:16:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: 2.19 scrub starts 2024-03-24T12:16:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: 2.19 scrub ok 2024-03-24T12:16:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: osdmap e539: 8 total, 8 up, 8 in 2024-03-24T12:16:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: 5.c scrub starts 2024-03-24T12:16:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: 5.c scrub ok 2024-03-24T12:16:19.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: 4.1c scrub starts 2024-03-24T12:16:19.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:19 smithi193 ceph-mon[33593]: 4.1c scrub ok 2024-03-24T12:16:20.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:20 smithi064 ceph-mon[30223]: 2.11 scrub starts 2024-03-24T12:16:20.243 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:20 smithi064 ceph-mon[30223]: 2.11 scrub ok 2024-03-24T12:16:20.244 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:20 smithi064 ceph-mon[42243]: 2.11 scrub starts 2024-03-24T12:16:20.244 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:20 smithi064 ceph-mon[42243]: 2.11 scrub ok 2024-03-24T12:16:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:20 smithi193 ceph-mon[33593]: 2.11 scrub starts 2024-03-24T12:16:20.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:20 smithi193 ceph-mon[33593]: 2.11 scrub ok 2024-03-24T12:16:20.848 INFO:teuthology.orchestra.run.smithi064.stdout:219043332395 2024-03-24T12:16:20.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T12:16:21.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[30223]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:21.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[30223]: 6.11 scrub starts 2024-03-24T12:16:21.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[30223]: 6.11 scrub ok 2024-03-24T12:16:21.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[30223]: 5.a scrub starts 2024-03-24T12:16:21.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[30223]: 5.a scrub ok 2024-03-24T12:16:21.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[42243]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:21.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[42243]: 6.11 scrub starts 2024-03-24T12:16:21.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[42243]: 6.11 scrub ok 2024-03-24T12:16:21.428 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[42243]: 5.a scrub starts 2024-03-24T12:16:21.428 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:21 smithi064 ceph-mon[42243]: 5.a scrub ok 2024-03-24T12:16:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:21 smithi193 ceph-mon[33593]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:21.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:21 smithi193 ceph-mon[33593]: 6.11 scrub starts 2024-03-24T12:16:21.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:21 smithi193 ceph-mon[33593]: 6.11 scrub ok 2024-03-24T12:16:21.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:21 smithi193 ceph-mon[33593]: 5.a scrub starts 2024-03-24T12:16:21.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:21 smithi193 ceph-mon[33593]: 5.a scrub ok 2024-03-24T12:16:21.772 INFO:teuthology.orchestra.run.smithi064.stdout:1516123455583 2024-03-24T12:16:21.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T12:16:22.683 INFO:teuthology.orchestra.run.smithi064.stdout:115964117306 2024-03-24T12:16:22.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T12:16:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:22.909 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[42243]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[42243]: 4.11 scrub starts 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[42243]: 4.11 scrub ok 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[42243]: osdmap e540: 8 total, 8 up, 8 in 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[42243]: 5.1f scrub starts 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[42243]: 5.1f scrub ok 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[42243]: 2.8 scrub starts 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[42243]: 2.8 scrub ok 2024-03-24T12:16:23.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[30223]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:23.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[30223]: 4.11 scrub starts 2024-03-24T12:16:23.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[30223]: 4.11 scrub ok 2024-03-24T12:16:23.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[30223]: osdmap e540: 8 total, 8 up, 8 in 2024-03-24T12:16:23.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[30223]: 5.1f scrub starts 2024-03-24T12:16:23.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[30223]: 5.1f scrub ok 2024-03-24T12:16:23.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[30223]: 2.8 scrub starts 2024-03-24T12:16:23.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:23 smithi064 ceph-mon[30223]: 2.8 scrub ok 2024-03-24T12:16:23.304 INFO:teuthology.orchestra.run.smithi064.stdout:77309411648 2024-03-24T12:16:23.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T12:16:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:23 smithi193 ceph-mon[33593]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:23 smithi193 ceph-mon[33593]: 4.11 scrub starts 2024-03-24T12:16:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:23 smithi193 ceph-mon[33593]: 4.11 scrub ok 2024-03-24T12:16:23.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:23 smithi193 ceph-mon[33593]: osdmap e540: 8 total, 8 up, 8 in 2024-03-24T12:16:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:23 smithi193 ceph-mon[33593]: 5.1f scrub starts 2024-03-24T12:16:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:23 smithi193 ceph-mon[33593]: 5.1f scrub ok 2024-03-24T12:16:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:23 smithi193 ceph-mon[33593]: 2.8 scrub starts 2024-03-24T12:16:23.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:23 smithi193 ceph-mon[33593]: 2.8 scrub ok 2024-03-24T12:16:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:23.618 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:25.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:25.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: 3.1 scrub starts 2024-03-24T12:16:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: 3.1 scrub ok 2024-03-24T12:16:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: 2.1b scrub starts 2024-03-24T12:16:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: 2.1b scrub ok 2024-03-24T12:16:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: 2.3 scrub starts 2024-03-24T12:16:25.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: 2.3 scrub ok 2024-03-24T12:16:25.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: 2.17 scrub starts 2024-03-24T12:16:25.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[42243]: 2.17 scrub ok 2024-03-24T12:16:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: 3.1 scrub starts 2024-03-24T12:16:25.303 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: 3.1 scrub ok 2024-03-24T12:16:25.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: 2.1b scrub starts 2024-03-24T12:16:25.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: 2.1b scrub ok 2024-03-24T12:16:25.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: 2.3 scrub starts 2024-03-24T12:16:25.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: 2.3 scrub ok 2024-03-24T12:16:25.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: 2.17 scrub starts 2024-03-24T12:16:25.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:25 smithi064 ceph-mon[30223]: 2.17 scrub ok 2024-03-24T12:16:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:25.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: 3.1 scrub starts 2024-03-24T12:16:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: 3.1 scrub ok 2024-03-24T12:16:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: 2.1b scrub starts 2024-03-24T12:16:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: 2.1b scrub ok 2024-03-24T12:16:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: 2.3 scrub starts 2024-03-24T12:16:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: 2.3 scrub ok 2024-03-24T12:16:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: 2.17 scrub starts 2024-03-24T12:16:25.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:25 smithi193 ceph-mon[33593]: 2.17 scrub ok 2024-03-24T12:16:25.589 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:25.600 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:26.326 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:26 smithi064 ceph-mon[30223]: 4.8 deep-scrub starts 2024-03-24T12:16:26.326 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:26 smithi064 ceph-mon[30223]: 4.8 deep-scrub ok 2024-03-24T12:16:26.326 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:26 smithi064 ceph-mon[30223]: 2.7 scrub starts 2024-03-24T12:16:26.326 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:26 smithi064 ceph-mon[30223]: 2.7 scrub ok 2024-03-24T12:16:26.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:26 smithi064 ceph-mon[42243]: 4.8 deep-scrub starts 2024-03-24T12:16:26.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:26 smithi064 ceph-mon[42243]: 4.8 deep-scrub ok 2024-03-24T12:16:26.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:26 smithi064 ceph-mon[42243]: 2.7 scrub starts 2024-03-24T12:16:26.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:26 smithi064 ceph-mon[42243]: 2.7 scrub ok 2024-03-24T12:16:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:26 smithi193 ceph-mon[33593]: 4.8 deep-scrub starts 2024-03-24T12:16:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:26 smithi193 ceph-mon[33593]: 4.8 deep-scrub ok 2024-03-24T12:16:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:26 smithi193 ceph-mon[33593]: 2.7 scrub starts 2024-03-24T12:16:26.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:26 smithi193 ceph-mon[33593]: 2.7 scrub ok 2024-03-24T12:16:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[42243]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[42243]: 2.4 scrub starts 2024-03-24T12:16:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[42243]: 2.4 scrub ok 2024-03-24T12:16:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[42243]: 4.1f scrub starts 2024-03-24T12:16:27.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[42243]: 4.1f scrub ok 2024-03-24T12:16:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[30223]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[30223]: 2.4 scrub starts 2024-03-24T12:16:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[30223]: 2.4 scrub ok 2024-03-24T12:16:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[30223]: 4.1f scrub starts 2024-03-24T12:16:27.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-mon[30223]: 4.1f scrub ok 2024-03-24T12:16:27.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:27 smithi193 ceph-mon[33593]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:16:27.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:27 smithi193 ceph-mon[33593]: 2.4 scrub starts 2024-03-24T12:16:27.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:27 smithi193 ceph-mon[33593]: 2.4 scrub ok 2024-03-24T12:16:27.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:27 smithi193 ceph-mon[33593]: 4.1f scrub starts 2024-03-24T12:16:27.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:27 smithi193 ceph-mon[33593]: 4.1f scrub ok 2024-03-24T12:16:27.687 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:27 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:16:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:16:28.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:16:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:16:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:28 smithi193 ceph-mon[33593]: 6.0 scrub starts 2024-03-24T12:16:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:28 smithi193 ceph-mon[33593]: 6.0 scrub ok 2024-03-24T12:16:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:28 smithi193 ceph-mon[33593]: 5.14 scrub starts 2024-03-24T12:16:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:28 smithi193 ceph-mon[33593]: 5.14 scrub ok 2024-03-24T12:16:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:28 smithi193 ceph-mon[33593]: 5.f scrub starts 2024-03-24T12:16:28.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:28 smithi193 ceph-mon[33593]: 5.f scrub ok 2024-03-24T12:16:28.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:28 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:28.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[30223]: 6.0 scrub starts 2024-03-24T12:16:28.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[30223]: 6.0 scrub ok 2024-03-24T12:16:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[30223]: 5.14 scrub starts 2024-03-24T12:16:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[30223]: 5.14 scrub ok 2024-03-24T12:16:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[30223]: 5.f scrub starts 2024-03-24T12:16:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[30223]: 5.f scrub ok 2024-03-24T12:16:28.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:28.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[42243]: 6.0 scrub starts 2024-03-24T12:16:28.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[42243]: 6.0 scrub ok 2024-03-24T12:16:28.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[42243]: 5.14 scrub starts 2024-03-24T12:16:28.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[42243]: 5.14 scrub ok 2024-03-24T12:16:28.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[42243]: 5.f scrub starts 2024-03-24T12:16:28.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[42243]: 5.f scrub ok 2024-03-24T12:16:28.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:28 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:29.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:29 smithi064 ceph-mon[30223]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 537 B/s rd, 0 op/s 2024-03-24T12:16:29.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:29 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:29.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:29 smithi064 ceph-mon[42243]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 537 B/s rd, 0 op/s 2024-03-24T12:16:29.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:29 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:29.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:29 smithi193 ceph-mon[33593]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 537 B/s rd, 0 op/s 2024-03-24T12:16:29.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:29 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:30.209 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:30 smithi064 ceph-mon[30223]: 6.6 scrub starts 2024-03-24T12:16:30.209 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:30 smithi064 ceph-mon[30223]: 6.6 scrub ok 2024-03-24T12:16:30.209 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:30 smithi064 ceph-mon[30223]: 5.1a scrub starts 2024-03-24T12:16:30.209 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:30 smithi064 ceph-mon[30223]: 5.1a scrub ok 2024-03-24T12:16:30.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:30 smithi064 ceph-mon[42243]: 6.6 scrub starts 2024-03-24T12:16:30.209 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:30 smithi064 ceph-mon[42243]: 6.6 scrub ok 2024-03-24T12:16:30.210 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:30 smithi064 ceph-mon[42243]: 5.1a scrub starts 2024-03-24T12:16:30.210 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:30 smithi064 ceph-mon[42243]: 5.1a scrub ok 2024-03-24T12:16:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:115964117308 2024-03-24T12:16:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:30 smithi193 ceph-mon[33593]: 6.6 scrub starts 2024-03-24T12:16:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:30 smithi193 ceph-mon[33593]: 6.6 scrub ok 2024-03-24T12:16:30.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:30 smithi193 ceph-mon[33593]: 5.1a scrub starts 2024-03-24T12:16:30.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:30 smithi193 ceph-mon[33593]: 5.1a scrub ok 2024-03-24T12:16:30.391 INFO:teuthology.orchestra.run.smithi064.stdout:55834575174 2024-03-24T12:16:30.949 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117306 got 115964117308 for osd.3 2024-03-24T12:16:30.949 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:31.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[30223]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:31.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4261113941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:16:31.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/353464241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:16:31.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[30223]: 6.1c deep-scrub starts 2024-03-24T12:16:31.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[30223]: 6.1c deep-scrub ok 2024-03-24T12:16:31.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:31.260 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[42243]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:31.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4261113941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:16:31.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/353464241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:16:31.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[42243]: 6.1c deep-scrub starts 2024-03-24T12:16:31.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[42243]: 6.1c deep-scrub ok 2024-03-24T12:16:31.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:31 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:31.572 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575171 got 55834575174 for osd.1 2024-03-24T12:16:31.572 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:31 smithi193 ceph-mon[33593]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:31.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4261113941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:16:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:31 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/353464241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:16:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:31 smithi193 ceph-mon[33593]: 6.1c deep-scrub starts 2024-03-24T12:16:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:31 smithi193 ceph-mon[33593]: 6.1c deep-scrub ok 2024-03-24T12:16:31.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:31 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:32.531 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[30223]: 5.1 scrub starts 2024-03-24T12:16:32.531 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[30223]: 5.1 scrub ok 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[30223]: 2.1d scrub starts 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[30223]: 2.1d scrub ok 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[42243]: 5.1 scrub starts 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[42243]: 5.1 scrub ok 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:16:32.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[42243]: 2.1d scrub starts 2024-03-24T12:16:32.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:32 smithi064 ceph-mon[42243]: 2.1d scrub ok 2024-03-24T12:16:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:32 smithi193 ceph-mon[33593]: 5.1 scrub starts 2024-03-24T12:16:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:32 smithi193 ceph-mon[33593]: 5.1 scrub ok 2024-03-24T12:16:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:32 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:32.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:32 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:32 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:16:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:32 smithi193 ceph-mon[33593]: 2.1d scrub starts 2024-03-24T12:16:32.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:32 smithi193 ceph-mon[33593]: 2.1d scrub ok 2024-03-24T12:16:33.450 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[30223]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:33.450 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[30223]: 6.1b scrub starts 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[30223]: 6.1b scrub ok 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[30223]: 2.f scrub starts 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[30223]: 2.f scrub ok 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[30223]: 5.4 scrub starts 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[30223]: 5.4 scrub ok 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[42243]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[42243]: 6.1b scrub starts 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[42243]: 6.1b scrub ok 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[42243]: 2.f scrub starts 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[42243]: 2.f scrub ok 2024-03-24T12:16:33.451 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[42243]: 5.4 scrub starts 2024-03-24T12:16:33.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:33 smithi064 ceph-mon[42243]: 5.4 scrub ok 2024-03-24T12:16:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:33 smithi193 ceph-mon[33593]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:16:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:33 smithi193 ceph-mon[33593]: 6.1b scrub starts 2024-03-24T12:16:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:33 smithi193 ceph-mon[33593]: 6.1b scrub ok 2024-03-24T12:16:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:33 smithi193 ceph-mon[33593]: 2.f scrub starts 2024-03-24T12:16:33.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:33 smithi193 ceph-mon[33593]: 2.f scrub ok 2024-03-24T12:16:33.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:33 smithi193 ceph-mon[33593]: 5.4 scrub starts 2024-03-24T12:16:33.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:33 smithi193 ceph-mon[33593]: 5.4 scrub ok 2024-03-24T12:16:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:33.870 INFO:teuthology.orchestra.run.smithi064.stdout:261993005351 2024-03-24T12:16:34.090 INFO:teuthology.orchestra.run.smithi064.stdout:34359738700 2024-03-24T12:16:34.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[30223]: 6.18 scrub starts 2024-03-24T12:16:34.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[30223]: 6.18 scrub ok 2024-03-24T12:16:34.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[30223]: 5.13 scrub starts 2024-03-24T12:16:34.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[30223]: 5.13 scrub ok 2024-03-24T12:16:34.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2343828866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:16:34.474 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/3726061965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:16:34.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[42243]: 6.18 scrub starts 2024-03-24T12:16:34.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[42243]: 6.18 scrub ok 2024-03-24T12:16:34.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[42243]: 5.13 scrub starts 2024-03-24T12:16:34.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[42243]: 5.13 scrub ok 2024-03-24T12:16:34.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2343828866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:16:34.474 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:34 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/3726061965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:16:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:34 smithi193 ceph-mon[33593]: 6.18 scrub starts 2024-03-24T12:16:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:34 smithi193 ceph-mon[33593]: 6.18 scrub ok 2024-03-24T12:16:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:34 smithi193 ceph-mon[33593]: 5.13 scrub starts 2024-03-24T12:16:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:34 smithi193 ceph-mon[33593]: 5.13 scrub ok 2024-03-24T12:16:34.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:34 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2343828866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:16:34.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:34 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/3726061965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:16:35.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[30223]: pgmap v1171: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[30223]: 6.b scrub starts 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[30223]: 6.b scrub ok 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[30223]: 1.a scrub starts 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[30223]: 1.a scrub ok 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[42243]: pgmap v1171: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[42243]: 6.b scrub starts 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[42243]: 6.b scrub ok 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[42243]: 1.a scrub starts 2024-03-24T12:16:35.141 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:35 smithi064 ceph-mon[42243]: 1.a scrub ok 2024-03-24T12:16:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:35 smithi193 ceph-mon[33593]: pgmap v1171: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:16:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:35 smithi193 ceph-mon[33593]: 6.b scrub starts 2024-03-24T12:16:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:35 smithi193 ceph-mon[33593]: 6.b scrub ok 2024-03-24T12:16:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:35 smithi193 ceph-mon[33593]: 1.a scrub starts 2024-03-24T12:16:35.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:35 smithi193 ceph-mon[33593]: 1.a scrub ok 2024-03-24T12:16:36.090 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005347 got 261993005351 for osd.7 2024-03-24T12:16:36.090 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:36.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:36 smithi064 ceph-mon[30223]: 5.1d scrub starts 2024-03-24T12:16:36.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:36 smithi064 ceph-mon[30223]: 5.1d scrub ok 2024-03-24T12:16:36.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:36 smithi064 ceph-mon[42243]: 5.1d scrub starts 2024-03-24T12:16:36.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:36 smithi064 ceph-mon[42243]: 5.1d scrub ok 2024-03-24T12:16:36.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:36 smithi193 ceph-mon[33593]: 5.1d scrub starts 2024-03-24T12:16:36.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:36 smithi193 ceph-mon[33593]: 5.1d scrub ok 2024-03-24T12:16:37.006 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738696 got 34359738700 for osd.0 2024-03-24T12:16:37.007 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:37.239 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:37.239 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: 2.1f scrub starts 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: 2.1f scrub ok 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: 2.6 scrub starts 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: 2.6 scrub ok 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: 4.1a scrub starts 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: 4.1a scrub ok 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: 5.6 scrub starts 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[30223]: 5.6 scrub ok 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: 2.1f scrub starts 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: 2.1f scrub ok 2024-03-24T12:16:37.240 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: 2.6 scrub starts 2024-03-24T12:16:37.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: 2.6 scrub ok 2024-03-24T12:16:37.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: 4.1a scrub starts 2024-03-24T12:16:37.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: 4.1a scrub ok 2024-03-24T12:16:37.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: 5.6 scrub starts 2024-03-24T12:16:37.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-mon[42243]: 5.6 scrub ok 2024-03-24T12:16:37.435 INFO:teuthology.orchestra.run.smithi064.stdout:1516123455586 2024-03-24T12:16:37.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:37 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:16:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:16:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: 2.1f scrub starts 2024-03-24T12:16:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: 2.1f scrub ok 2024-03-24T12:16:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: 2.6 scrub starts 2024-03-24T12:16:37.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: 2.6 scrub ok 2024-03-24T12:16:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: 4.1a scrub starts 2024-03-24T12:16:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: 4.1a scrub ok 2024-03-24T12:16:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: 5.6 scrub starts 2024-03-24T12:16:37.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:37 smithi193 ceph-mon[33593]: 5.6 scrub ok 2024-03-24T12:16:37.779 INFO:teuthology.orchestra.run.smithi064.stdout:219043332399 2024-03-24T12:16:38.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/665185853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:16:38.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:38 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4271522024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:16:38.460 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/665185853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:16:38.460 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:38 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4271522024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:16:38.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:16:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:16:38.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/665185853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:16:38.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:38 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4271522024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:16:38.784 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332395 got 219043332399 for osd.6 2024-03-24T12:16:38.784 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:39.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:39 smithi064 ceph-mon[30223]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:39.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:39 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:39.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:39 smithi064 ceph-mon[30223]: 5.2 deep-scrub starts 2024-03-24T12:16:39.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:39 smithi064 ceph-mon[30223]: 5.2 deep-scrub ok 2024-03-24T12:16:39.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:39 smithi064 ceph-mon[42243]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:39.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:39 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:39.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:39 smithi064 ceph-mon[42243]: 5.2 deep-scrub starts 2024-03-24T12:16:39.465 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:39 smithi064 ceph-mon[42243]: 5.2 deep-scrub ok 2024-03-24T12:16:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:39 smithi193 ceph-mon[33593]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:39 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:39.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:39 smithi193 ceph-mon[33593]: 5.2 deep-scrub starts 2024-03-24T12:16:39.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:39 smithi193 ceph-mon[33593]: 5.2 deep-scrub ok 2024-03-24T12:16:39.684 INFO:teuthology.orchestra.run.smithi064.stdout:141733921081 2024-03-24T12:16:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[30223]: 4.3 scrub starts 2024-03-24T12:16:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[30223]: 4.3 scrub ok 2024-03-24T12:16:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[30223]: 5.12 scrub starts 2024-03-24T12:16:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[30223]: 5.12 scrub ok 2024-03-24T12:16:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/2737358887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:16:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[30223]: 2.1 scrub starts 2024-03-24T12:16:40.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[30223]: 2.1 scrub ok 2024-03-24T12:16:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[42243]: 4.3 scrub starts 2024-03-24T12:16:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[42243]: 4.3 scrub ok 2024-03-24T12:16:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[42243]: 5.12 scrub starts 2024-03-24T12:16:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[42243]: 5.12 scrub ok 2024-03-24T12:16:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/2737358887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:16:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[42243]: 2.1 scrub starts 2024-03-24T12:16:40.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:40 smithi064 ceph-mon[42243]: 2.1 scrub ok 2024-03-24T12:16:40.366 INFO:teuthology.orchestra.run.smithi064.stdout:77309411652 2024-03-24T12:16:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:40 smithi193 ceph-mon[33593]: 4.3 scrub starts 2024-03-24T12:16:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:40 smithi193 ceph-mon[33593]: 4.3 scrub ok 2024-03-24T12:16:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:40 smithi193 ceph-mon[33593]: 5.12 scrub starts 2024-03-24T12:16:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:40 smithi193 ceph-mon[33593]: 5.12 scrub ok 2024-03-24T12:16:40.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:40 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/2737358887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:16:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:40 smithi193 ceph-mon[33593]: 2.1 scrub starts 2024-03-24T12:16:40.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:40 smithi193 ceph-mon[33593]: 2.1 scrub ok 2024-03-24T12:16:40.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1516123455583 got 1516123455586 for osd.5 2024-03-24T12:16:40.773 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:41.255 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733921076 got 141733921081 for osd.4 2024-03-24T12:16:41.255 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:41 smithi064 ceph-mon[30223]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:41 smithi064 ceph-mon[30223]: 5.5 scrub starts 2024-03-24T12:16:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:41 smithi064 ceph-mon[30223]: 5.5 scrub ok 2024-03-24T12:16:41.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:41 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/1459845725' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:16:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:41 smithi064 ceph-mon[42243]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:41 smithi064 ceph-mon[42243]: 5.5 scrub starts 2024-03-24T12:16:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:41 smithi064 ceph-mon[42243]: 5.5 scrub ok 2024-03-24T12:16:41.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:41 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/1459845725' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:16:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:41 smithi193 ceph-mon[33593]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:41 smithi193 ceph-mon[33593]: 5.5 scrub starts 2024-03-24T12:16:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:41 smithi193 ceph-mon[33593]: 5.5 scrub ok 2024-03-24T12:16:41.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:41 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/1459845725' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:16:41.952 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411648 got 77309411652 for osd.2 2024-03-24T12:16:41.953 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:41.953 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-24T12:16:41.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:16:42.205 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:42.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:42 smithi064 ceph-mon[30223]: 2.0 scrub starts 2024-03-24T12:16:42.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:42 smithi064 ceph-mon[30223]: 2.0 scrub ok 2024-03-24T12:16:42.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:42 smithi064 ceph-mon[30223]: 5.0 scrub starts 2024-03-24T12:16:42.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:42 smithi064 ceph-mon[30223]: 5.0 scrub ok 2024-03-24T12:16:42.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:42 smithi064 ceph-mon[42243]: 2.0 scrub starts 2024-03-24T12:16:42.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:42 smithi064 ceph-mon[42243]: 2.0 scrub ok 2024-03-24T12:16:42.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:42 smithi064 ceph-mon[42243]: 5.0 scrub starts 2024-03-24T12:16:42.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:42 smithi064 ceph-mon[42243]: 5.0 scrub ok 2024-03-24T12:16:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:42 smithi193 ceph-mon[33593]: 2.0 scrub starts 2024-03-24T12:16:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:42 smithi193 ceph-mon[33593]: 2.0 scrub ok 2024-03-24T12:16:42.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:42 smithi193 ceph-mon[33593]: 5.0 scrub starts 2024-03-24T12:16:42.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:42 smithi193 ceph-mon[33593]: 5.0 scrub ok 2024-03-24T12:16:43.495 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:43 smithi064 ceph-mon[30223]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:43.495 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:43 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:43.495 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:43 smithi064 ceph-mon[42243]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:43.495 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:43 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:43.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:43 smithi193 ceph-mon[33593]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:43.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:43 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:16:44.263 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:16:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[30223]: 6.9 scrub starts 2024-03-24T12:16:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[30223]: 6.9 scrub ok 2024-03-24T12:16:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[30223]: 6.f scrub starts 2024-03-24T12:16:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[30223]: 2.5 scrub starts 2024-03-24T12:16:44.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[30223]: 2.5 scrub ok 2024-03-24T12:16:44.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[42243]: 6.9 scrub starts 2024-03-24T12:16:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[42243]: 6.9 scrub ok 2024-03-24T12:16:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[42243]: 6.f scrub starts 2024-03-24T12:16:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[42243]: 2.5 scrub starts 2024-03-24T12:16:44.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:44 smithi064 ceph-mon[42243]: 2.5 scrub ok 2024-03-24T12:16:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:44 smithi193 ceph-mon[33593]: 6.9 scrub starts 2024-03-24T12:16:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:44 smithi193 ceph-mon[33593]: 6.9 scrub ok 2024-03-24T12:16:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:44 smithi193 ceph-mon[33593]: 6.f scrub starts 2024-03-24T12:16:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:44 smithi193 ceph-mon[33593]: 2.5 scrub starts 2024-03-24T12:16:44.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:44 smithi193 ceph-mon[33593]: 2.5 scrub ok 2024-03-24T12:16:45.111 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1176,"stamp":"2024-03-24T12:16:43.642542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2202,"num_read_kb":1954,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":333387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1584,"ondisk_log_size":1584,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6521244,"kb_used_data":9308,"kb_used_omap":18,"kb_used_meta":6511853,"kb_avail":743439972,"statfs":{"total":767960285184,"available":761282531328,"internally_reserved":0,"allocated":9531392,"data_stored":5735322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":6668137691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002902"},"pg_stats":[{"pgid":"5.9","version":"176'9","reported_seq":1366,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487511+0000","last_change":"2024-03-24T12:16:13.484196+0000","last_active":"2024-03-24T12:16:35.487511+0000","last_peered":"2024-03-24T12:16:35.487511+0000","last_clean":"2024-03-24T12:16:35.487511+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:16:35.487511+0000","last_undegraded":"2024-03-24T12:16:35.487511+0000","last_fullsized":"2024-03-24T12:16:35.487511+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:16:13.484090+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:16:13.484090+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:42.715707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"540'46","reported_seq":992,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:25.276372+0000","last_change":"2024-03-24T12:16:25.276426+0000","last_active":"2024-03-24T12:16:25.276372+0000","last_peered":"2024-03-24T12:16:25.276372+0000","last_clean":"2024-03-24T12:16:25.276372+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:16:25.276372+0000","last_undegraded":"2024-03-24T12:16:25.276372+0000","last_fullsized":"2024-03-24T12:16:25.276372+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"540'46","last_scrub_stamp":"2024-03-24T12:16:25.276296+0000","last_deep_scrub":"540'46","last_deep_scrub_stamp":"2024-03-24T12:16:25.276296+0000","last_clean_scrub_stamp":"2024-03-24T12:16:25.276296+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:33.817464+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067547+0000","last_change":"2024-03-24T12:15:18.254993+0000","last_active":"2024-03-24T12:16:22.067547+0000","last_peered":"2024-03-24T12:16:22.067547+0000","last_clean":"2024-03-24T12:16:22.067547+0000","last_became_active":"2024-03-24T12:15:15.281302+0000","last_became_peered":"2024-03-24T12:15:15.281302+0000","last_unstale":"2024-03-24T12:16:22.067547+0000","last_undegraded":"2024-03-24T12:16:22.067547+0000","last_fullsized":"2024-03-24T12:16:22.067547+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:18.254906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:15:18.254906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:43.101507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":951,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206478+0000","last_change":"2024-03-24T12:15:40.268528+0000","last_active":"2024-03-24T12:16:22.206478+0000","last_peered":"2024-03-24T12:16:22.206478+0000","last_clean":"2024-03-24T12:16:22.206478+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:16:22.206478+0000","last_undegraded":"2024-03-24T12:16:22.206478+0000","last_fullsized":"2024-03-24T12:16:22.206478+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:40.268416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:40.268416+0000","last_clean_scrub_stamp":"2024-03-24T12:15:40.268416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:07.146762+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"450'21","reported_seq":1075,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066792+0000","last_change":"2024-03-24T12:15:47.544260+0000","last_active":"2024-03-24T12:16:22.066792+0000","last_peered":"2024-03-24T12:16:22.066792+0000","last_clean":"2024-03-24T12:16:22.066792+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:16:22.066792+0000","last_undegraded":"2024-03-24T12:16:22.066792+0000","last_fullsized":"2024-03-24T12:16:22.066792+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:15:47.544155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:47.544155+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:58.728413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194918+0000","last_change":"2024-03-24T12:15:42.519236+0000","last_active":"2024-03-24T12:16:22.194918+0000","last_peered":"2024-03-24T12:16:22.194918+0000","last_clean":"2024-03-24T12:16:22.194918+0000","last_became_active":"2024-03-24T12:15:05.225551+0000","last_became_peered":"2024-03-24T12:15:05.225551+0000","last_unstale":"2024-03-24T12:16:22.194918+0000","last_undegraded":"2024-03-24T12:16:22.194918+0000","last_fullsized":"2024-03-24T12:16:22.194918+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:42.519153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:15:42.519153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:54.613976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":525,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.068126+0000","last_change":"2024-03-24T12:16:20.605299+0000","last_active":"2024-03-24T12:16:22.068126+0000","last_peered":"2024-03-24T12:16:22.068126+0000","last_clean":"2024-03-24T12:16:22.068126+0000","last_became_active":"2024-03-24T12:15:19.067107+0000","last_became_peered":"2024-03-24T12:15:19.067107+0000","last_unstale":"2024-03-24T12:16:22.068126+0000","last_undegraded":"2024-03-24T12:16:22.068126+0000","last_fullsized":"2024-03-24T12:16:22.068126+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:20.605210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:16:20.605210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:41.077971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1038,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067519+0000","last_change":"2024-03-24T12:15:29.678702+0000","last_active":"2024-03-24T12:16:22.067519+0000","last_peered":"2024-03-24T12:16:22.067519+0000","last_clean":"2024-03-24T12:16:22.067519+0000","last_became_active":"2024-03-24T12:15:15.274417+0000","last_became_peered":"2024-03-24T12:15:15.274417+0000","last_unstale":"2024-03-24T12:16:22.067519+0000","last_undegraded":"2024-03-24T12:16:22.067519+0000","last_fullsized":"2024-03-24T12:16:22.067519+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:29.678596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:29.678596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:33.519684+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"450'60","reported_seq":1010,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192772+0000","last_change":"2024-03-24T12:15:43.068619+0000","last_active":"2024-03-24T12:16:22.192772+0000","last_peered":"2024-03-24T12:16:22.192772+0000","last_clean":"2024-03-24T12:16:22.192772+0000","last_became_active":"2024-03-24T12:15:43.068230+0000","last_became_peered":"2024-03-24T12:15:43.068230+0000","last_unstale":"2024-03-24T12:16:22.192772+0000","last_undegraded":"2024-03-24T12:16:22.192772+0000","last_fullsized":"2024-03-24T12:16:22.192772+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'60","last_scrub_stamp":"2024-03-24T12:15:30.497376+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:15:30.497376+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:44.841312+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.068230+0000","last_change":"2024-03-24T12:15:49.558254+0000","last_active":"2024-03-24T12:16:22.068230+0000","last_peered":"2024-03-24T12:16:22.068230+0000","last_clean":"2024-03-24T12:16:22.068230+0000","last_became_active":"2024-03-24T12:15:15.277545+0000","last_became_peered":"2024-03-24T12:15:15.277545+0000","last_unstale":"2024-03-24T12:16:22.068230+0000","last_undegraded":"2024-03-24T12:16:22.068230+0000","last_fullsized":"2024-03-24T12:16:22.068230+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:49.558150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:15:49.558150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:04.882336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066248+0000","last_change":"2024-03-24T12:15:41.431643+0000","last_active":"2024-03-24T12:16:22.066248+0000","last_peered":"2024-03-24T12:16:22.066248+0000","last_clean":"2024-03-24T12:16:22.066248+0000","last_became_active":"2024-03-24T12:15:15.273789+0000","last_became_peered":"2024-03-24T12:15:15.273789+0000","last_unstale":"2024-03-24T12:16:22.066248+0000","last_undegraded":"2024-03-24T12:16:22.066248+0000","last_fullsized":"2024-03-24T12:16:22.066248+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:41.431562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:41.431562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:53.760797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1063,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.357534+0000","last_change":"2024-03-24T12:15:44.491981+0000","last_active":"2024-03-24T12:16:22.357534+0000","last_peered":"2024-03-24T12:16:22.357534+0000","last_clean":"2024-03-24T12:16:22.357534+0000","last_became_active":"2024-03-24T12:15:15.276348+0000","last_became_peered":"2024-03-24T12:15:15.276348+0000","last_unstale":"2024-03-24T12:16:22.357534+0000","last_undegraded":"2024-03-24T12:16:22.357534+0000","last_fullsized":"2024-03-24T12:16:22.357534+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:44.491883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:15:44.491883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:10.857536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"450'53","reported_seq":1157,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355610+0000","last_change":"2024-03-24T12:15:25.155210+0000","last_active":"2024-03-24T12:16:22.355610+0000","last_peered":"2024-03-24T12:16:22.355610+0000","last_clean":"2024-03-24T12:16:22.355610+0000","last_became_active":"2024-03-24T12:15:15.282554+0000","last_became_peered":"2024-03-24T12:15:15.282554+0000","last_unstale":"2024-03-24T12:16:22.355610+0000","last_undegraded":"2024-03-24T12:16:22.355610+0000","last_fullsized":"2024-03-24T12:16:22.355610+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'53","last_scrub_stamp":"2024-03-24T12:15:25.155107+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:15:25.155107+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:37.613206+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":735,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066345+0000","last_change":"2024-03-24T12:15:26.271993+0000","last_active":"2024-03-24T12:16:22.066345+0000","last_peered":"2024-03-24T12:16:22.066345+0000","last_clean":"2024-03-24T12:16:22.066345+0000","last_became_active":"2024-03-24T12:15:19.067448+0000","last_became_peered":"2024-03-24T12:15:19.067448+0000","last_unstale":"2024-03-24T12:16:22.066345+0000","last_undegraded":"2024-03-24T12:16:22.066345+0000","last_fullsized":"2024-03-24T12:16:22.066345+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:15:26.271874+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:26.271874+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:55.075355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":790,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194529+0000","last_change":"2024-03-24T12:16:18.178693+0000","last_active":"2024-03-24T12:16:22.194529+0000","last_peered":"2024-03-24T12:16:22.194529+0000","last_clean":"2024-03-24T12:16:22.194529+0000","last_became_active":"2024-03-24T12:15:11.249346+0000","last_became_peered":"2024-03-24T12:15:11.249346+0000","last_unstale":"2024-03-24T12:16:22.194529+0000","last_undegraded":"2024-03-24T12:16:22.194529+0000","last_fullsized":"2024-03-24T12:16:22.194529+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:18.178617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:09.171134+0000","last_clean_scrub_stamp":"2024-03-24T12:16:18.178617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:20.935982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067297+0000","last_change":"2024-03-24T12:15:24.449466+0000","last_active":"2024-03-24T12:16:22.067297+0000","last_peered":"2024-03-24T12:16:22.067297+0000","last_clean":"2024-03-24T12:16:22.067297+0000","last_became_active":"2024-03-24T12:15:15.280821+0000","last_became_peered":"2024-03-24T12:15:15.280821+0000","last_unstale":"2024-03-24T12:16:22.067297+0000","last_undegraded":"2024-03-24T12:16:22.067297+0000","last_fullsized":"2024-03-24T12:16:22.067297+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:24.449337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:24.449337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:37.493390+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":982,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206114+0000","last_change":"2024-03-24T12:15:57.165240+0000","last_active":"2024-03-24T12:16:22.206114+0000","last_peered":"2024-03-24T12:16:22.206114+0000","last_clean":"2024-03-24T12:16:22.206114+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:16:22.206114+0000","last_undegraded":"2024-03-24T12:16:22.206114+0000","last_fullsized":"2024-03-24T12:16:22.206114+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:57.165119+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:57.165119+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:04.254851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"450'35","reported_seq":1081,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355243+0000","last_change":"2024-03-24T12:15:19.211493+0000","last_active":"2024-03-24T12:16:22.355243+0000","last_peered":"2024-03-24T12:16:22.355243+0000","last_clean":"2024-03-24T12:16:22.355243+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:16:22.355243+0000","last_undegraded":"2024-03-24T12:16:22.355243+0000","last_fullsized":"2024-03-24T12:16:22.355243+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:15:19.211337+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:15:19.211337+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:44.078086+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":800,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194372+0000","last_change":"2024-03-24T12:16:17.223324+0000","last_active":"2024-03-24T12:16:22.194372+0000","last_peered":"2024-03-24T12:16:22.194372+0000","last_clean":"2024-03-24T12:16:22.194372+0000","last_became_active":"2024-03-24T12:15:11.657317+0000","last_became_peered":"2024-03-24T12:15:11.657317+0000","last_unstale":"2024-03-24T12:16:22.194372+0000","last_undegraded":"2024-03-24T12:16:22.194372+0000","last_fullsized":"2024-03-24T12:16:22.194372+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:17.223243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:16:17.223243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:45.816048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":1374,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487316+0000","last_change":"2024-03-24T12:15:37.436529+0000","last_active":"2024-03-24T12:16:35.487316+0000","last_peered":"2024-03-24T12:16:35.487316+0000","last_clean":"2024-03-24T12:16:35.487316+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:16:35.487316+0000","last_undegraded":"2024-03-24T12:16:35.487316+0000","last_fullsized":"2024-03-24T12:16:35.487316+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:15:37.436402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:37.436402+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:46.798240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":1073,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355779+0000","last_change":"2024-03-24T12:15:40.311031+0000","last_active":"2024-03-24T12:16:22.355779+0000","last_peered":"2024-03-24T12:16:22.355779+0000","last_clean":"2024-03-24T12:16:22.355779+0000","last_became_active":"2024-03-24T12:15:15.281782+0000","last_became_peered":"2024-03-24T12:15:15.281782+0000","last_unstale":"2024-03-24T12:16:22.355779+0000","last_undegraded":"2024-03-24T12:16:22.355779+0000","last_fullsized":"2024-03-24T12:16:22.355779+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:15:40.310926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:40.310926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:53.249412+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":1387,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.357443+0000","last_change":"2024-03-24T12:15:49.490214+0000","last_active":"2024-03-24T12:16:22.357443+0000","last_peered":"2024-03-24T12:16:22.357443+0000","last_clean":"2024-03-24T12:16:22.357443+0000","last_became_active":"2024-03-24T12:15:18.064798+0000","last_became_peered":"2024-03-24T12:15:18.064798+0000","last_unstale":"2024-03-24T12:16:22.357443+0000","last_undegraded":"2024-03-24T12:16:22.357443+0000","last_fullsized":"2024-03-24T12:16:22.357443+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:49.490065+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:15:49.490065+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.616761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"450'46","reported_seq":1133,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.195001+0000","last_change":"2024-03-24T12:15:50.321951+0000","last_active":"2024-03-24T12:16:22.195001+0000","last_peered":"2024-03-24T12:16:22.195001+0000","last_clean":"2024-03-24T12:16:22.195001+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:16:22.195001+0000","last_undegraded":"2024-03-24T12:16:22.195001+0000","last_fullsized":"2024-03-24T12:16:22.195001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'46","last_scrub_stamp":"2024-03-24T12:15:50.321819+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:15:50.321819+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:17.489542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1078,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194436+0000","last_change":"2024-03-24T12:15:57.255000+0000","last_active":"2024-03-24T12:16:22.194436+0000","last_peered":"2024-03-24T12:16:22.194436+0000","last_clean":"2024-03-24T12:16:22.194436+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:16:22.194436+0000","last_undegraded":"2024-03-24T12:16:22.194436+0000","last_fullsized":"2024-03-24T12:16:22.194436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:57.254904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:15:57.254904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:04.398411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":1343,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487510+0000","last_change":"2024-03-24T12:15:35.280727+0000","last_active":"2024-03-24T12:16:35.487510+0000","last_peered":"2024-03-24T12:16:35.487510+0000","last_clean":"2024-03-24T12:16:35.487510+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:16:35.487510+0000","last_undegraded":"2024-03-24T12:16:35.487510+0000","last_fullsized":"2024-03-24T12:16:35.487510+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:15:35.280649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:35.280649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:43.523617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":971,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206987+0000","last_change":"2024-03-24T12:16:00.342458+0000","last_active":"2024-03-24T12:16:22.206987+0000","last_peered":"2024-03-24T12:16:22.206987+0000","last_clean":"2024-03-24T12:16:22.206987+0000","last_became_active":"2024-03-24T12:15:05.212711+0000","last_became_peered":"2024-03-24T12:15:05.212711+0000","last_unstale":"2024-03-24T12:16:22.206987+0000","last_undegraded":"2024-03-24T12:16:22.206987+0000","last_fullsized":"2024-03-24T12:16:22.206987+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:00.342327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:16:00.342327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:27.837069+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":990,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:34.349194+0000","last_change":"2024-03-24T12:16:34.349218+0000","last_active":"2024-03-24T12:16:34.349194+0000","last_peered":"2024-03-24T12:16:34.349194+0000","last_clean":"2024-03-24T12:16:34.349194+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:16:34.349194+0000","last_undegraded":"2024-03-24T12:16:34.349194+0000","last_fullsized":"2024-03-24T12:16:34.349194+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:16:34.349137+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:16:34.349137+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:52.051167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"450'32","reported_seq":1115,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355533+0000","last_change":"2024-03-24T12:16:06.232953+0000","last_active":"2024-03-24T12:16:22.355533+0000","last_peered":"2024-03-24T12:16:22.355533+0000","last_clean":"2024-03-24T12:16:22.355533+0000","last_became_active":"2024-03-24T12:15:05.234577+0000","last_became_peered":"2024-03-24T12:15:05.234577+0000","last_unstale":"2024-03-24T12:16:22.355533+0000","last_undegraded":"2024-03-24T12:16:22.355533+0000","last_fullsized":"2024-03-24T12:16:22.355533+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'32","last_scrub_stamp":"2024-03-24T12:16:06.232847+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:16:06.232847+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:26.842474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1083,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.663285+0000","last_change":"2024-03-24T12:16:22.663311+0000","last_active":"2024-03-24T12:16:22.663285+0000","last_peered":"2024-03-24T12:16:22.663285+0000","last_clean":"2024-03-24T12:16:22.663285+0000","last_became_active":"2024-03-24T12:15:05.214715+0000","last_became_peered":"2024-03-24T12:15:05.214715+0000","last_unstale":"2024-03-24T12:16:22.663285+0000","last_undegraded":"2024-03-24T12:16:22.663285+0000","last_fullsized":"2024-03-24T12:16:22.663285+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:22.663221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:08.571841+0000","last_clean_scrub_stamp":"2024-03-24T12:16:22.663221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:36.562715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":955,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:27.232721+0000","last_change":"2024-03-24T12:16:27.232768+0000","last_active":"2024-03-24T12:16:27.232721+0000","last_peered":"2024-03-24T12:16:27.232721+0000","last_clean":"2024-03-24T12:16:27.232721+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:16:27.232721+0000","last_undegraded":"2024-03-24T12:16:27.232721+0000","last_fullsized":"2024-03-24T12:16:27.232721+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:27.232629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:27.232629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:49.490654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1036,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065365+0000","last_change":"2024-03-24T12:16:06.221399+0000","last_active":"2024-03-24T12:16:22.065365+0000","last_peered":"2024-03-24T12:16:22.065365+0000","last_clean":"2024-03-24T12:16:22.065365+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:16:22.065365+0000","last_undegraded":"2024-03-24T12:16:22.065365+0000","last_fullsized":"2024-03-24T12:16:22.065365+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:06.221287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:06.221287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:07.930681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"507'10","reported_seq":1356,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:41.128312+0000","last_change":"2024-03-24T12:16:41.128362+0000","last_active":"2024-03-24T12:16:41.128312+0000","last_peered":"2024-03-24T12:16:41.128312+0000","last_clean":"2024-03-24T12:16:41.128312+0000","last_became_active":"2024-03-24T12:15:05.230777+0000","last_became_peered":"2024-03-24T12:15:05.230777+0000","last_unstale":"2024-03-24T12:16:41.128312+0000","last_undegraded":"2024-03-24T12:16:41.128312+0000","last_fullsized":"2024-03-24T12:16:41.128312+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'10","last_scrub_stamp":"2024-03-24T12:16:41.128222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:41.128222+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:54.821274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"450'49","reported_seq":1111,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355052+0000","last_change":"2024-03-24T12:15:48.238042+0000","last_active":"2024-03-24T12:16:22.355052+0000","last_peered":"2024-03-24T12:16:22.355052+0000","last_clean":"2024-03-24T12:16:22.355052+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:16:22.355052+0000","last_undegraded":"2024-03-24T12:16:22.355052+0000","last_fullsized":"2024-03-24T12:16:22.355052+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'49","last_scrub_stamp":"2024-03-24T12:15:48.237913+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:15:48.237913+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:48.663781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":942,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:25.405138+0000","last_change":"2024-03-24T12:16:25.405161+0000","last_active":"2024-03-24T12:16:25.405138+0000","last_peered":"2024-03-24T12:16:25.405138+0000","last_clean":"2024-03-24T12:16:25.405138+0000","last_became_active":"2024-03-24T12:15:43.069574+0000","last_became_peered":"2024-03-24T12:15:43.069574+0000","last_unstale":"2024-03-24T12:16:25.405138+0000","last_undegraded":"2024-03-24T12:16:25.405138+0000","last_fullsized":"2024-03-24T12:16:25.405138+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:25.405071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:16:25.405071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:48.134013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1083,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358259+0000","last_change":"2024-03-24T12:15:35.688152+0000","last_active":"2024-03-24T12:16:22.358259+0000","last_peered":"2024-03-24T12:16:22.358259+0000","last_clean":"2024-03-24T12:16:22.358259+0000","last_became_active":"2024-03-24T12:15:15.275798+0000","last_became_peered":"2024-03-24T12:15:15.275798+0000","last_unstale":"2024-03-24T12:16:22.358259+0000","last_undegraded":"2024-03-24T12:16:22.358259+0000","last_fullsized":"2024-03-24T12:16:22.358259+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:35.688049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:35.688049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:53.039052+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":1117,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065650+0000","last_change":"2024-03-24T12:15:29.122877+0000","last_active":"2024-03-24T12:16:22.065650+0000","last_peered":"2024-03-24T12:16:22.065650+0000","last_clean":"2024-03-24T12:16:22.065650+0000","last_became_active":"2024-03-24T12:10:47.971906+0000","last_became_peered":"2024-03-24T12:10:47.971906+0000","last_unstale":"2024-03-24T12:16:22.065650+0000","last_undegraded":"2024-03-24T12:16:22.065650+0000","last_fullsized":"2024-03-24T12:16:22.065650+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:29.122771+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:29.122771+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.979863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:31.484385+0000","last_change":"2024-03-24T12:16:31.484424+0000","last_active":"2024-03-24T12:16:31.484385+0000","last_peered":"2024-03-24T12:16:31.484385+0000","last_clean":"2024-03-24T12:16:31.484385+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:16:31.484385+0000","last_undegraded":"2024-03-24T12:16:31.484385+0000","last_fullsized":"2024-03-24T12:16:31.484385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:31.484315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:03.195101+0000","last_clean_scrub_stamp":"2024-03-24T12:16:31.484315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:40.086141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"450'50","reported_seq":1113,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067655+0000","last_change":"2024-03-24T12:16:04.225054+0000","last_active":"2024-03-24T12:16:22.067655+0000","last_peered":"2024-03-24T12:16:22.067655+0000","last_clean":"2024-03-24T12:16:22.067655+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:16:22.067655+0000","last_undegraded":"2024-03-24T12:16:22.067655+0000","last_fullsized":"2024-03-24T12:16:22.067655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'50","last_scrub_stamp":"2024-03-24T12:16:04.224949+0000","last_deep_scrub":"241'38","last_deep_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_clean_scrub_stamp":"2024-03-24T12:16:04.224949+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:15.077250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":198,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:36.271134+0000","last_change":"2024-03-24T12:16:36.271165+0000","last_active":"2024-03-24T12:16:36.271134+0000","last_peered":"2024-03-24T12:16:36.271134+0000","last_clean":"2024-03-24T12:16:36.271134+0000","last_became_active":"2024-03-24T12:15:11.248843+0000","last_became_peered":"2024-03-24T12:15:11.248843+0000","last_unstale":"2024-03-24T12:16:36.271134+0000","last_undegraded":"2024-03-24T12:16:36.271134+0000","last_fullsized":"2024-03-24T12:16:36.271134+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:16:36.271056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:36.271056+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:42.478507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1052,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355489+0000","last_change":"2024-03-24T12:15:51.288985+0000","last_active":"2024-03-24T12:16:22.355489+0000","last_peered":"2024-03-24T12:16:22.355489+0000","last_clean":"2024-03-24T12:16:22.355489+0000","last_became_active":"2024-03-24T12:15:15.281413+0000","last_became_peered":"2024-03-24T12:15:15.281413+0000","last_unstale":"2024-03-24T12:16:22.355489+0000","last_undegraded":"2024-03-24T12:16:22.355489+0000","last_fullsized":"2024-03-24T12:16:22.355489+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:51.288883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:51.288883+0000","last_clean_scrub_stamp":"2024-03-24T12:15:51.288883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:51.967766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":1137,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.354766+0000","last_change":"2024-03-24T12:15:58.251741+0000","last_active":"2024-03-24T12:16:22.354766+0000","last_peered":"2024-03-24T12:16:22.354766+0000","last_clean":"2024-03-24T12:16:22.354766+0000","last_became_active":"2024-03-24T12:15:18.067386+0000","last_became_peered":"2024-03-24T12:15:18.067386+0000","last_unstale":"2024-03-24T12:16:22.354766+0000","last_undegraded":"2024-03-24T12:16:22.354766+0000","last_fullsized":"2024-03-24T12:16:22.354766+0000","mapping_epoch":513,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:58.251637+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:58.251637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:26.833069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":936,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:38.454357+0000","last_change":"2024-03-24T12:16:38.454399+0000","last_active":"2024-03-24T12:16:38.454357+0000","last_peered":"2024-03-24T12:16:38.454357+0000","last_clean":"2024-03-24T12:16:38.454357+0000","last_became_active":"2024-03-24T12:15:43.072285+0000","last_became_peered":"2024-03-24T12:15:43.072285+0000","last_unstale":"2024-03-24T12:16:38.454357+0000","last_undegraded":"2024-03-24T12:16:38.454357+0000","last_fullsized":"2024-03-24T12:16:38.454357+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:38.454250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:16:38.454250+0000","last_clean_scrub_stamp":"2024-03-24T12:16:38.454250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:54.348665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"450'47","reported_seq":1117,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:38.746071+0000","last_change":"2024-03-24T12:16:38.746150+0000","last_active":"2024-03-24T12:16:38.746071+0000","last_peered":"2024-03-24T12:16:38.746071+0000","last_clean":"2024-03-24T12:16:38.746071+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:16:38.746071+0000","last_undegraded":"2024-03-24T12:16:38.746071+0000","last_fullsized":"2024-03-24T12:16:38.746071+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'47","last_scrub_stamp":"2024-03-24T12:16:38.745964+0000","last_deep_scrub":"369'37","last_deep_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_clean_scrub_stamp":"2024-03-24T12:16:38.745964+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:44.424883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1071,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358631+0000","last_change":"2024-03-24T12:15:15.679971+0000","last_active":"2024-03-24T12:16:22.358631+0000","last_peered":"2024-03-24T12:16:22.358631+0000","last_clean":"2024-03-24T12:16:22.358631+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:16:22.358631+0000","last_undegraded":"2024-03-24T12:16:22.358631+0000","last_fullsized":"2024-03-24T12:16:22.358631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:15.679884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:15.679884+0000","last_clean_scrub_stamp":"2024-03-24T12:15:15.679884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:38.992156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":468,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194281+0000","last_change":"2024-03-24T12:16:08.305203+0000","last_active":"2024-03-24T12:16:22.194281+0000","last_peered":"2024-03-24T12:16:22.194281+0000","last_clean":"2024-03-24T12:16:22.194281+0000","last_became_active":"2024-03-24T12:15:15.274745+0000","last_became_peered":"2024-03-24T12:15:15.274745+0000","last_unstale":"2024-03-24T12:16:22.194281+0000","last_undegraded":"2024-03-24T12:16:22.194281+0000","last_fullsized":"2024-03-24T12:16:22.194281+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:08.305080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:20.921985+0000","last_clean_scrub_stamp":"2024-03-24T12:16:08.305080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:26.515483+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":1018,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.193100+0000","last_change":"2024-03-24T12:15:43.557623+0000","last_active":"2024-03-24T12:16:22.193100+0000","last_peered":"2024-03-24T12:16:22.193100+0000","last_clean":"2024-03-24T12:16:22.193100+0000","last_became_active":"2024-03-24T12:15:43.067828+0000","last_became_peered":"2024-03-24T12:15:43.067828+0000","last_unstale":"2024-03-24T12:16:22.193100+0000","last_undegraded":"2024-03-24T12:16:22.193100+0000","last_fullsized":"2024-03-24T12:16:22.193100+0000","mapping_epoch":523,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:43.557516+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_clean_scrub_stamp":"2024-03-24T12:15:43.557516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:05.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":933,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.207271+0000","last_change":"2024-03-24T12:16:10.307245+0000","last_active":"2024-03-24T12:16:22.207271+0000","last_peered":"2024-03-24T12:16:22.207271+0000","last_clean":"2024-03-24T12:16:22.207271+0000","last_became_active":"2024-03-24T12:15:05.214114+0000","last_became_peered":"2024-03-24T12:15:05.214114+0000","last_unstale":"2024-03-24T12:16:22.207271+0000","last_undegraded":"2024-03-24T12:16:22.207271+0000","last_fullsized":"2024-03-24T12:16:22.207271+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:10.307140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:21.577764+0000","last_clean_scrub_stamp":"2024-03-24T12:16:10.307140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:34.738268+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"450'42","reported_seq":1142,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067183+0000","last_change":"2024-03-24T12:15:53.551637+0000","last_active":"2024-03-24T12:16:22.067183+0000","last_peered":"2024-03-24T12:16:22.067183+0000","last_clean":"2024-03-24T12:16:22.067183+0000","last_became_active":"2024-03-24T12:15:15.275691+0000","last_became_peered":"2024-03-24T12:15:15.275691+0000","last_unstale":"2024-03-24T12:16:22.067183+0000","last_undegraded":"2024-03-24T12:16:22.067183+0000","last_fullsized":"2024-03-24T12:16:22.067183+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'42","last_scrub_stamp":"2024-03-24T12:15:53.551535+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:15:53.551535+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:11.978629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":882,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065733+0000","last_change":"2024-03-24T12:15:27.560825+0000","last_active":"2024-03-24T12:16:22.065733+0000","last_peered":"2024-03-24T12:16:22.065733+0000","last_clean":"2024-03-24T12:16:22.065733+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:16:22.065733+0000","last_undegraded":"2024-03-24T12:16:22.065733+0000","last_fullsized":"2024-03-24T12:16:22.065733+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:27.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:27.560693+0000","last_clean_scrub_stamp":"2024-03-24T12:15:27.560693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:32.370136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1064,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:23.720152+0000","last_change":"2024-03-24T12:16:23.720190+0000","last_active":"2024-03-24T12:16:23.720152+0000","last_peered":"2024-03-24T12:16:23.720152+0000","last_clean":"2024-03-24T12:16:23.720152+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:16:23.720152+0000","last_undegraded":"2024-03-24T12:16:23.720152+0000","last_fullsized":"2024-03-24T12:16:23.720152+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:23.720063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:16:23.720063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:52.305568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194707+0000","last_change":"2024-03-24T12:15:31.554219+0000","last_active":"2024-03-24T12:16:22.194707+0000","last_peered":"2024-03-24T12:16:22.194707+0000","last_clean":"2024-03-24T12:16:22.194707+0000","last_became_active":"2024-03-24T12:15:05.223578+0000","last_became_peered":"2024-03-24T12:15:05.223578+0000","last_unstale":"2024-03-24T12:16:22.194707+0000","last_undegraded":"2024-03-24T12:16:22.194707+0000","last_fullsized":"2024-03-24T12:16:22.194707+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:31.554139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_clean_scrub_stamp":"2024-03-24T12:15:31.554139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.736890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":381,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194549+0000","last_change":"2024-03-24T12:15:11.420213+0000","last_active":"2024-03-24T12:16:22.194549+0000","last_peered":"2024-03-24T12:16:22.194549+0000","last_clean":"2024-03-24T12:16:22.194549+0000","last_became_active":"2024-03-24T12:15:11.248603+0000","last_became_peered":"2024-03-24T12:15:11.248603+0000","last_unstale":"2024-03-24T12:16:22.194549+0000","last_undegraded":"2024-03-24T12:16:22.194549+0000","last_fullsized":"2024-03-24T12:16:22.194549+0000","mapping_epoch":508,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:11.420131+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:11.420131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:35.607384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1064,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:32.761069+0000","last_change":"2024-03-24T12:16:32.761090+0000","last_active":"2024-03-24T12:16:32.761069+0000","last_peered":"2024-03-24T12:16:32.761069+0000","last_clean":"2024-03-24T12:16:32.761069+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:16:32.761069+0000","last_undegraded":"2024-03-24T12:16:32.761069+0000","last_fullsized":"2024-03-24T12:16:32.761069+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:32.761008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:16:32.761008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:47.256331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"525'42","reported_seq":1271,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487538+0000","last_change":"2024-03-24T12:16:04.507179+0000","last_active":"2024-03-24T12:16:35.487538+0000","last_peered":"2024-03-24T12:16:35.487538+0000","last_clean":"2024-03-24T12:16:35.487538+0000","last_became_active":"2024-03-24T12:15:43.072013+0000","last_became_peered":"2024-03-24T12:15:43.072013+0000","last_unstale":"2024-03-24T12:16:35.487538+0000","last_undegraded":"2024-03-24T12:16:35.487538+0000","last_fullsized":"2024-03-24T12:16:35.487538+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'42","last_scrub_stamp":"2024-03-24T12:16:04.507046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:04.507046+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:14.416757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":1377,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.357652+0000","last_change":"2024-03-24T12:15:28.679148+0000","last_active":"2024-03-24T12:16:22.357652+0000","last_peered":"2024-03-24T12:16:22.357652+0000","last_clean":"2024-03-24T12:16:22.357652+0000","last_became_active":"2024-03-24T12:10:33.461798+0000","last_became_peered":"2024-03-24T12:10:33.461798+0000","last_unstale":"2024-03-24T12:16:22.357652+0000","last_undegraded":"2024-03-24T12:16:22.357652+0000","last_fullsized":"2024-03-24T12:16:22.357652+0000","mapping_epoch":64,"log_start":"83'130","ondisk_log_start":"83'130","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:28.679021+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:10:25.984109+0000","last_clean_scrub_stamp":"2024-03-24T12:15:28.679021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:51.449406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":961,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:24.221701+0000","last_change":"2024-03-24T12:16:24.221726+0000","last_active":"2024-03-24T12:16:24.221701+0000","last_peered":"2024-03-24T12:16:24.221701+0000","last_clean":"2024-03-24T12:16:24.221701+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:16:24.221701+0000","last_undegraded":"2024-03-24T12:16:24.221701+0000","last_fullsized":"2024-03-24T12:16:24.221701+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:24.221629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:24.221629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:31.331207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":1050,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065367+0000","last_change":"2024-03-24T12:16:01.281552+0000","last_active":"2024-03-24T12:16:22.065367+0000","last_peered":"2024-03-24T12:16:22.065367+0000","last_clean":"2024-03-24T12:16:22.065367+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:16:22.065367+0000","last_undegraded":"2024-03-24T12:16:22.065367+0000","last_fullsized":"2024-03-24T12:16:22.065367+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:16:01.281451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:01.281451+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:04.311553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":916,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.207225+0000","last_change":"2024-03-24T12:16:14.296935+0000","last_active":"2024-03-24T12:16:22.207225+0000","last_peered":"2024-03-24T12:16:22.207225+0000","last_clean":"2024-03-24T12:16:22.207225+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:16:22.207225+0000","last_undegraded":"2024-03-24T12:16:22.207225+0000","last_fullsized":"2024-03-24T12:16:22.207225+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:14.296822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:16:14.296822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:21.266968+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:36.777707+0000","last_change":"2024-03-24T12:16:36.777734+0000","last_active":"2024-03-24T12:16:36.777707+0000","last_peered":"2024-03-24T12:16:36.777707+0000","last_clean":"2024-03-24T12:16:36.777707+0000","last_became_active":"2024-03-24T12:15:15.277454+0000","last_became_peered":"2024-03-24T12:15:15.277454+0000","last_unstale":"2024-03-24T12:16:36.777707+0000","last_undegraded":"2024-03-24T12:16:36.777707+0000","last_fullsized":"2024-03-24T12:16:36.777707+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:36.777636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:16:36.777636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:40.070534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":102,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194896+0000","last_change":"2024-03-24T12:15:48.366270+0000","last_active":"2024-03-24T12:16:22.194896+0000","last_peered":"2024-03-24T12:16:22.194896+0000","last_clean":"2024-03-24T12:16:22.194896+0000","last_became_active":"2024-03-24T12:15:05.216691+0000","last_became_peered":"2024-03-24T12:15:05.216691+0000","last_unstale":"2024-03-24T12:16:22.194896+0000","last_undegraded":"2024-03-24T12:16:22.194896+0000","last_fullsized":"2024-03-24T12:16:22.194896+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'26","last_scrub_stamp":"2024-03-24T12:15:48.366102+0000","last_deep_scrub":"526'26","last_deep_scrub_stamp":"2024-03-24T12:15:48.366102+0000","last_clean_scrub_stamp":"2024-03-24T12:15:48.366102+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.766909+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:39.717481+0000","last_change":"2024-03-24T12:16:39.717513+0000","last_active":"2024-03-24T12:16:39.717481+0000","last_peered":"2024-03-24T12:16:39.717481+0000","last_clean":"2024-03-24T12:16:39.717481+0000","last_became_active":"2024-03-24T12:15:15.280046+0000","last_became_peered":"2024-03-24T12:15:15.280046+0000","last_unstale":"2024-03-24T12:16:39.717481+0000","last_undegraded":"2024-03-24T12:16:39.717481+0000","last_fullsized":"2024-03-24T12:16:39.717481+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:39.717394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:39.717394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:55.369319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"462'6","reported_seq":1237,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:41.412226+0000","last_change":"2024-03-24T12:15:23.605179+0000","last_active":"2024-03-24T12:16:41.412226+0000","last_peered":"2024-03-24T12:16:41.412226+0000","last_clean":"2024-03-24T12:16:41.412226+0000","last_became_active":"2024-03-24T12:15:15.274515+0000","last_became_peered":"2024-03-24T12:15:15.274515+0000","last_unstale":"2024-03-24T12:16:41.412226+0000","last_undegraded":"2024-03-24T12:16:41.412226+0000","last_fullsized":"2024-03-24T12:16:41.412226+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'6","last_scrub_stamp":"2024-03-24T12:15:23.605093+0000","last_deep_scrub":"462'6","last_deep_scrub_stamp":"2024-03-24T12:13:14.961156+0000","last_clean_scrub_stamp":"2024-03-24T12:15:23.605093+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":640,"num_read_kb":635,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1051,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358626+0000","last_change":"2024-03-24T12:15:33.694555+0000","last_active":"2024-03-24T12:16:22.358626+0000","last_peered":"2024-03-24T12:16:22.358626+0000","last_clean":"2024-03-24T12:16:22.358626+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:16:22.358626+0000","last_undegraded":"2024-03-24T12:16:22.358626+0000","last_fullsized":"2024-03-24T12:16:22.358626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:33.694432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:33.694432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:42.255085+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":1376,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358083+0000","last_change":"2024-03-24T12:15:26.615976+0000","last_active":"2024-03-24T12:16:22.358083+0000","last_peered":"2024-03-24T12:16:22.358083+0000","last_clean":"2024-03-24T12:16:22.358083+0000","last_became_active":"2024-03-24T12:10:23.398177+0000","last_became_peered":"2024-03-24T12:10:23.398177+0000","last_unstale":"2024-03-24T12:16:22.358083+0000","last_undegraded":"2024-03-24T12:16:22.358083+0000","last_fullsized":"2024-03-24T12:16:22.358083+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:26.615877+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:26.615877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:46.685876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1042,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066068+0000","last_change":"2024-03-24T12:15:11.676736+0000","last_active":"2024-03-24T12:16:22.066068+0000","last_peered":"2024-03-24T12:16:22.066068+0000","last_clean":"2024-03-24T12:16:22.066068+0000","last_became_active":"2024-03-24T12:15:05.219911+0000","last_became_peered":"2024-03-24T12:15:05.219911+0000","last_unstale":"2024-03-24T12:16:22.066068+0000","last_undegraded":"2024-03-24T12:16:22.066068+0000","last_fullsized":"2024-03-24T12:16:22.066068+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:11.676661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:15:11.676661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:31.543716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"511'60","reported_seq":1198,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487273+0000","last_change":"2024-03-24T12:15:59.248196+0000","last_active":"2024-03-24T12:16:35.487273+0000","last_peered":"2024-03-24T12:16:35.487273+0000","last_clean":"2024-03-24T12:16:35.487273+0000","last_became_active":"2024-03-24T12:15:15.276052+0000","last_became_peered":"2024-03-24T12:15:15.276052+0000","last_unstale":"2024-03-24T12:16:35.487273+0000","last_undegraded":"2024-03-24T12:16:35.487273+0000","last_fullsized":"2024-03-24T12:16:35.487273+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'60","last_scrub_stamp":"2024-03-24T12:15:59.248094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:59.248094+0000","objects_scrubbed":10,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:09.068128+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":925,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206392+0000","last_change":"2024-03-24T12:16:06.241708+0000","last_active":"2024-03-24T12:16:22.206392+0000","last_peered":"2024-03-24T12:16:22.206392+0000","last_clean":"2024-03-24T12:16:22.206392+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:16:22.206392+0000","last_undegraded":"2024-03-24T12:16:22.206392+0000","last_fullsized":"2024-03-24T12:16:22.206392+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:06.241596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:16:06.241596+0000","last_clean_scrub_stamp":"2024-03-24T12:16:06.241596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:34.123576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:29.089517+0000","last_change":"2024-03-24T12:16:29.089547+0000","last_active":"2024-03-24T12:16:29.089517+0000","last_peered":"2024-03-24T12:16:29.089517+0000","last_clean":"2024-03-24T12:16:29.089517+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:16:29.089517+0000","last_undegraded":"2024-03-24T12:16:29.089517+0000","last_fullsized":"2024-03-24T12:16:29.089517+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:29.089429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:29.089429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:43.515255+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":1001,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206381+0000","last_change":"2024-03-24T12:15:24.086717+0000","last_active":"2024-03-24T12:16:22.206381+0000","last_peered":"2024-03-24T12:16:22.206381+0000","last_clean":"2024-03-24T12:16:22.206381+0000","last_became_active":"2024-03-24T12:10:26.959434+0000","last_became_peered":"2024-03-24T12:10:26.959434+0000","last_unstale":"2024-03-24T12:16:22.206381+0000","last_undegraded":"2024-03-24T12:16:22.206381+0000","last_fullsized":"2024-03-24T12:16:22.206381+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:24.086638+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:24.086638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:45.070870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":1350,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:30.486523+0000","last_change":"2024-03-24T12:15:59.500305+0000","last_active":"2024-03-24T12:16:30.486523+0000","last_peered":"2024-03-24T12:16:30.486523+0000","last_clean":"2024-03-24T12:16:30.486523+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:16:30.486523+0000","last_undegraded":"2024-03-24T12:16:30.486523+0000","last_fullsized":"2024-03-24T12:16:30.486523+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:15:59.500213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:59.500213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:08.741574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"450'35","reported_seq":97,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194570+0000","last_change":"2024-03-24T12:16:00.237534+0000","last_active":"2024-03-24T12:16:22.194570+0000","last_peered":"2024-03-24T12:16:22.194570+0000","last_clean":"2024-03-24T12:16:22.194570+0000","last_became_active":"2024-03-24T12:15:05.225343+0000","last_became_peered":"2024-03-24T12:15:05.225343+0000","last_unstale":"2024-03-24T12:16:22.194570+0000","last_undegraded":"2024-03-24T12:16:22.194570+0000","last_fullsized":"2024-03-24T12:16:22.194570+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:16:00.237427+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:16:00.237427+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:10.406479+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":807,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:26.240498+0000","last_change":"2024-03-24T12:16:26.240523+0000","last_active":"2024-03-24T12:16:26.240498+0000","last_peered":"2024-03-24T12:16:26.240498+0000","last_clean":"2024-03-24T12:16:26.240498+0000","last_became_active":"2024-03-24T12:15:11.657425+0000","last_became_peered":"2024-03-24T12:15:11.657425+0000","last_unstale":"2024-03-24T12:16:26.240498+0000","last_undegraded":"2024-03-24T12:16:26.240498+0000","last_fullsized":"2024-03-24T12:16:26.240498+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:26.240434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:17.492961+0000","last_clean_scrub_stamp":"2024-03-24T12:16:26.240434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:32.095116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1048,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:26.746576+0000","last_change":"2024-03-24T12:16:26.746576+0000","last_active":"2024-03-24T12:16:26.746576+0000","last_peered":"2024-03-24T12:16:26.746576+0000","last_clean":"2024-03-24T12:16:26.746576+0000","last_became_active":"2024-03-24T12:15:15.273632+0000","last_became_peered":"2024-03-24T12:15:15.273632+0000","last_unstale":"2024-03-24T12:16:26.746576+0000","last_undegraded":"2024-03-24T12:16:26.746576+0000","last_fullsized":"2024-03-24T12:16:26.746576+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:26.746477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:16:26.746477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:30.059907+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":1010,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192960+0000","last_change":"2024-03-24T12:15:50.523000+0000","last_active":"2024-03-24T12:16:22.192960+0000","last_peered":"2024-03-24T12:16:22.192960+0000","last_clean":"2024-03-24T12:16:22.192960+0000","last_became_active":"2024-03-24T12:15:43.069034+0000","last_became_peered":"2024-03-24T12:15:43.069034+0000","last_unstale":"2024-03-24T12:16:22.192960+0000","last_undegraded":"2024-03-24T12:16:22.192960+0000","last_fullsized":"2024-03-24T12:16:22.192960+0000","mapping_epoch":523,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:50.522836+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:15:50.522836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:50.657476+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1070,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066078+0000","last_change":"2024-03-24T12:15:36.400232+0000","last_active":"2024-03-24T12:16:22.066078+0000","last_peered":"2024-03-24T12:16:22.066078+0000","last_clean":"2024-03-24T12:16:22.066078+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:16:22.066078+0000","last_undegraded":"2024-03-24T12:16:22.066078+0000","last_fullsized":"2024-03-24T12:16:22.066078+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:36.400126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:36.400126+0000","last_clean_scrub_stamp":"2024-03-24T12:15:36.400126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:43.605921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"450'24","reported_seq":1135,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355054+0000","last_change":"2024-03-24T12:15:27.201919+0000","last_active":"2024-03-24T12:16:22.355054+0000","last_peered":"2024-03-24T12:16:22.355054+0000","last_clean":"2024-03-24T12:16:22.355054+0000","last_became_active":"2024-03-24T12:15:05.233554+0000","last_became_peered":"2024-03-24T12:15:05.233554+0000","last_unstale":"2024-03-24T12:16:22.355054+0000","last_undegraded":"2024-03-24T12:16:22.355054+0000","last_fullsized":"2024-03-24T12:16:22.355054+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'24","last_scrub_stamp":"2024-03-24T12:15:27.201820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:27.201820+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.494834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":1055,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:32.507867+0000","last_change":"2024-03-24T12:16:32.507919+0000","last_active":"2024-03-24T12:16:32.507867+0000","last_peered":"2024-03-24T12:16:32.507867+0000","last_clean":"2024-03-24T12:16:32.507867+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:16:32.507867+0000","last_undegraded":"2024-03-24T12:16:32.507867+0000","last_fullsized":"2024-03-24T12:16:32.507867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:16:32.507776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:32.507776+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:41.535470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1039,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:34.301514+0000","last_change":"2024-03-24T12:16:34.301539+0000","last_active":"2024-03-24T12:16:34.301514+0000","last_peered":"2024-03-24T12:16:34.301514+0000","last_clean":"2024-03-24T12:16:34.301514+0000","last_became_active":"2024-03-24T12:15:05.230832+0000","last_became_peered":"2024-03-24T12:15:05.230832+0000","last_unstale":"2024-03-24T12:16:34.301514+0000","last_undegraded":"2024-03-24T12:16:34.301514+0000","last_fullsized":"2024-03-24T12:16:34.301514+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:34.301430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:16:34.301430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:58.361767+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"450'25","reported_seq":1087,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067328+0000","last_change":"2024-03-24T12:16:12.615425+0000","last_active":"2024-03-24T12:16:22.067328+0000","last_peered":"2024-03-24T12:16:22.067328+0000","last_clean":"2024-03-24T12:16:22.067328+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:16:22.067328+0000","last_undegraded":"2024-03-24T12:16:22.067328+0000","last_fullsized":"2024-03-24T12:16:22.067328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'25","last_scrub_stamp":"2024-03-24T12:16:12.615322+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:16:12.615322+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:40.586281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067763+0000","last_change":"2024-03-24T12:16:05.526639+0000","last_active":"2024-03-24T12:16:22.067763+0000","last_peered":"2024-03-24T12:16:22.067763+0000","last_clean":"2024-03-24T12:16:22.067763+0000","last_became_active":"2024-03-24T12:15:15.279928+0000","last_became_peered":"2024-03-24T12:15:15.279928+0000","last_unstale":"2024-03-24T12:16:22.067763+0000","last_undegraded":"2024-03-24T12:16:22.067763+0000","last_fullsized":"2024-03-24T12:16:22.067763+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:05.526557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:05.526557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:28.814607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1035,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066085+0000","last_change":"2024-03-24T12:16:16.127155+0000","last_active":"2024-03-24T12:16:22.066085+0000","last_peered":"2024-03-24T12:16:22.066085+0000","last_clean":"2024-03-24T12:16:22.066085+0000","last_became_active":"2024-03-24T12:15:05.223618+0000","last_became_peered":"2024-03-24T12:15:05.223618+0000","last_unstale":"2024-03-24T12:16:22.066085+0000","last_undegraded":"2024-03-24T12:16:22.066085+0000","last_fullsized":"2024-03-24T12:16:22.066085+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:16.127051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:16.127051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:41.798869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.068344+0000","last_change":"2024-03-24T12:15:20.252884+0000","last_active":"2024-03-24T12:16:22.068344+0000","last_peered":"2024-03-24T12:16:22.068344+0000","last_clean":"2024-03-24T12:16:22.068344+0000","last_became_active":"2024-03-24T12:15:15.281396+0000","last_became_peered":"2024-03-24T12:15:15.281396+0000","last_unstale":"2024-03-24T12:16:22.068344+0000","last_undegraded":"2024-03-24T12:16:22.068344+0000","last_fullsized":"2024-03-24T12:16:22.068344+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:20.252770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:20.252770+0000","last_clean_scrub_stamp":"2024-03-24T12:15:20.252770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:38.675260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"450'18","reported_seq":1062,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065967+0000","last_change":"2024-03-24T12:16:13.166830+0000","last_active":"2024-03-24T12:16:22.065967+0000","last_peered":"2024-03-24T12:16:22.065967+0000","last_clean":"2024-03-24T12:16:22.065967+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:16:22.065967+0000","last_undegraded":"2024-03-24T12:16:22.065967+0000","last_fullsized":"2024-03-24T12:16:22.065967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'18","last_scrub_stamp":"2024-03-24T12:16:13.166726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:13.166726+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:21.260703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":942,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192434+0000","last_change":"2024-03-24T12:16:19.333970+0000","last_active":"2024-03-24T12:16:22.192434+0000","last_peered":"2024-03-24T12:16:22.192434+0000","last_clean":"2024-03-24T12:16:22.192434+0000","last_became_active":"2024-03-24T12:15:43.070487+0000","last_became_peered":"2024-03-24T12:15:43.070487+0000","last_unstale":"2024-03-24T12:16:22.192434+0000","last_undegraded":"2024-03-24T12:16:22.192434+0000","last_fullsized":"2024-03-24T12:16:22.192434+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:19.333868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:16:19.333868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:29.144746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":885,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206987+0000","last_change":"2024-03-24T12:15:38.226629+0000","last_active":"2024-03-24T12:16:22.206987+0000","last_peered":"2024-03-24T12:16:22.206987+0000","last_clean":"2024-03-24T12:16:22.206987+0000","last_became_active":"2024-03-24T12:15:05.214305+0000","last_became_peered":"2024-03-24T12:15:05.214305+0000","last_unstale":"2024-03-24T12:16:22.206987+0000","last_undegraded":"2024-03-24T12:16:22.206987+0000","last_fullsized":"2024-03-24T12:16:22.206987+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:38.226550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:38.226550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:01.256309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1053,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358212+0000","last_change":"2024-03-24T12:16:11.482087+0000","last_active":"2024-03-24T12:16:22.358212+0000","last_peered":"2024-03-24T12:16:22.358212+0000","last_clean":"2024-03-24T12:16:22.358212+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:16:22.358212+0000","last_undegraded":"2024-03-24T12:16:22.358212+0000","last_fullsized":"2024-03-24T12:16:22.358212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:11.481976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:16:11.481976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:14.539883+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"450'35","reported_seq":121,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206591+0000","last_change":"2024-03-24T12:16:02.246915+0000","last_active":"2024-03-24T12:16:22.206591+0000","last_peered":"2024-03-24T12:16:22.206591+0000","last_clean":"2024-03-24T12:16:22.206591+0000","last_became_active":"2024-03-24T12:15:05.214237+0000","last_became_peered":"2024-03-24T12:15:05.214237+0000","last_unstale":"2024-03-24T12:16:22.206591+0000","last_undegraded":"2024-03-24T12:16:22.206591+0000","last_fullsized":"2024-03-24T12:16:22.206591+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:16:02.246770+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:16:02.246770+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:25.535207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":924,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206611+0000","last_change":"2024-03-24T12:15:44.175851+0000","last_active":"2024-03-24T12:16:22.206611+0000","last_peered":"2024-03-24T12:16:22.206611+0000","last_clean":"2024-03-24T12:16:22.206611+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:16:22.206611+0000","last_undegraded":"2024-03-24T12:16:22.206611+0000","last_fullsized":"2024-03-24T12:16:22.206611+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:44.175742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:14:26.371072+0000","last_clean_scrub_stamp":"2024-03-24T12:15:44.175742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:10.556371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":1233,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:30.486679+0000","last_change":"2024-03-24T12:15:50.148137+0000","last_active":"2024-03-24T12:16:30.486679+0000","last_peered":"2024-03-24T12:16:30.486679+0000","last_clean":"2024-03-24T12:16:30.486679+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:16:30.486679+0000","last_undegraded":"2024-03-24T12:16:30.486679+0000","last_fullsized":"2024-03-24T12:16:30.486679+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:15:50.147987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:50.147987+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:11.374410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1033,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067129+0000","last_change":"2024-03-24T12:15:58.601408+0000","last_active":"2024-03-24T12:16:22.067129+0000","last_peered":"2024-03-24T12:16:22.067129+0000","last_clean":"2024-03-24T12:16:22.067129+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:16:22.067129+0000","last_undegraded":"2024-03-24T12:16:22.067129+0000","last_fullsized":"2024-03-24T12:16:22.067129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:58.601298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:58.601298+0000","last_clean_scrub_stamp":"2024-03-24T12:15:58.601298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:23.366331+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"450'21","reported_seq":958,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.207124+0000","last_change":"2024-03-24T12:16:09.306345+0000","last_active":"2024-03-24T12:16:22.207124+0000","last_peered":"2024-03-24T12:16:22.207124+0000","last_clean":"2024-03-24T12:16:22.207124+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:16:22.207124+0000","last_undegraded":"2024-03-24T12:16:22.207124+0000","last_fullsized":"2024-03-24T12:16:22.207124+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:16:09.306202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:09.306202+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:31.672976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":919,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066704+0000","last_change":"2024-03-24T12:15:36.530174+0000","last_active":"2024-03-24T12:16:22.066704+0000","last_peered":"2024-03-24T12:16:22.066704+0000","last_clean":"2024-03-24T12:16:22.066704+0000","last_became_active":"2024-03-24T12:10:30.443191+0000","last_became_peered":"2024-03-24T12:10:30.443191+0000","last_unstale":"2024-03-24T12:16:22.066704+0000","last_undegraded":"2024-03-24T12:16:22.066704+0000","last_fullsized":"2024-03-24T12:16:22.066704+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:36.530081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:15:36.530081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:36.546184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"511'10","reported_seq":1353,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487313+0000","last_change":"2024-03-24T12:16:27.079415+0000","last_active":"2024-03-24T12:16:35.487313+0000","last_peered":"2024-03-24T12:16:35.487313+0000","last_clean":"2024-03-24T12:16:35.487313+0000","last_became_active":"2024-03-24T12:15:15.272446+0000","last_became_peered":"2024-03-24T12:15:15.272446+0000","last_unstale":"2024-03-24T12:16:35.487313+0000","last_undegraded":"2024-03-24T12:16:35.487313+0000","last_fullsized":"2024-03-24T12:16:35.487313+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'10","last_scrub_stamp":"2024-03-24T12:16:27.079307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:27.079307+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:42.644218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1032,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355903+0000","last_change":"2024-03-24T12:15:52.524638+0000","last_active":"2024-03-24T12:16:22.355903+0000","last_peered":"2024-03-24T12:16:22.355903+0000","last_clean":"2024-03-24T12:16:22.355903+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:16:22.355903+0000","last_undegraded":"2024-03-24T12:16:22.355903+0000","last_fullsized":"2024-03-24T12:16:22.355903+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:52.524535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:15:52.524535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:07.244672+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"450'21","reported_seq":847,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194351+0000","last_change":"2024-03-24T12:15:37.553854+0000","last_active":"2024-03-24T12:16:22.194351+0000","last_peered":"2024-03-24T12:16:22.194351+0000","last_clean":"2024-03-24T12:16:22.194351+0000","last_became_active":"2024-03-24T12:15:11.247777+0000","last_became_peered":"2024-03-24T12:15:11.247777+0000","last_unstale":"2024-03-24T12:16:22.194351+0000","last_undegraded":"2024-03-24T12:16:22.194351+0000","last_fullsized":"2024-03-24T12:16:22.194351+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:15:37.553753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:37.553753+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:01.610655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":934,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192655+0000","last_change":"2024-03-24T12:15:43.073055+0000","last_active":"2024-03-24T12:16:22.192655+0000","last_peered":"2024-03-24T12:16:22.192655+0000","last_clean":"2024-03-24T12:16:22.192655+0000","last_became_active":"2024-03-24T12:15:43.071443+0000","last_became_peered":"2024-03-24T12:15:43.071443+0000","last_unstale":"2024-03-24T12:16:22.192655+0000","last_undegraded":"2024-03-24T12:16:22.192655+0000","last_fullsized":"2024-03-24T12:16:22.192655+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:30.092224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:15:30.092224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:57.450217+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1037,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:33.095595+0000","last_change":"2024-03-24T12:16:33.095615+0000","last_active":"2024-03-24T12:16:33.095595+0000","last_peered":"2024-03-24T12:16:33.095595+0000","last_clean":"2024-03-24T12:16:33.095595+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:16:33.095595+0000","last_undegraded":"2024-03-24T12:16:33.095595+0000","last_fullsized":"2024-03-24T12:16:33.095595+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:33.095537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:33.095537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:37.325834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1054,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065290+0000","last_change":"2024-03-24T12:15:49.670943+0000","last_active":"2024-03-24T12:16:22.065290+0000","last_peered":"2024-03-24T12:16:22.065290+0000","last_clean":"2024-03-24T12:16:22.065290+0000","last_became_active":"2024-03-24T12:15:05.216451+0000","last_became_peered":"2024-03-24T12:15:05.216451+0000","last_unstale":"2024-03-24T12:16:22.065290+0000","last_undegraded":"2024-03-24T12:16:22.065290+0000","last_fullsized":"2024-03-24T12:16:22.065290+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:49.670785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:49.670785+0000","last_clean_scrub_stamp":"2024-03-24T12:15:49.670785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:15.348655+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"450'27","reported_seq":1110,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355277+0000","last_change":"2024-03-24T12:16:09.312934+0000","last_active":"2024-03-24T12:16:22.355277+0000","last_peered":"2024-03-24T12:16:22.355277+0000","last_clean":"2024-03-24T12:16:22.355277+0000","last_became_active":"2024-03-24T12:15:05.233832+0000","last_became_peered":"2024-03-24T12:15:05.233832+0000","last_unstale":"2024-03-24T12:16:22.355277+0000","last_undegraded":"2024-03-24T12:16:22.355277+0000","last_fullsized":"2024-03-24T12:16:22.355277+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'27","last_scrub_stamp":"2024-03-24T12:16:09.312823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:09.312823+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:36.282369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194842+0000","last_change":"2024-03-24T12:15:33.546429+0000","last_active":"2024-03-24T12:16:22.194842+0000","last_peered":"2024-03-24T12:16:22.194842+0000","last_clean":"2024-03-24T12:16:22.194842+0000","last_became_active":"2024-03-24T12:15:05.216042+0000","last_became_peered":"2024-03-24T12:15:05.216042+0000","last_unstale":"2024-03-24T12:16:22.194842+0000","last_undegraded":"2024-03-24T12:16:22.194842+0000","last_fullsized":"2024-03-24T12:16:22.194842+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:33.546352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:15:33.546352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:44.143860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":693,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:39.297296+0000","last_change":"2024-03-24T12:16:39.297338+0000","last_active":"2024-03-24T12:16:39.297296+0000","last_peered":"2024-03-24T12:16:39.297296+0000","last_clean":"2024-03-24T12:16:39.297296+0000","last_became_active":"2024-03-24T12:15:11.731801+0000","last_became_peered":"2024-03-24T12:15:11.731801+0000","last_unstale":"2024-03-24T12:16:39.297296+0000","last_undegraded":"2024-03-24T12:16:39.297296+0000","last_fullsized":"2024-03-24T12:16:39.297296+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:39.297217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:16:39.297217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:06.748045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065971+0000","last_change":"2024-03-24T12:16:20.158020+0000","last_active":"2024-03-24T12:16:22.065971+0000","last_peered":"2024-03-24T12:16:22.065971+0000","last_clean":"2024-03-24T12:16:22.065971+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:16:22.065971+0000","last_undegraded":"2024-03-24T12:16:22.065971+0000","last_fullsized":"2024-03-24T12:16:22.065971+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:20.157920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:20.157920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:36.888199+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"450'14","reported_seq":1079,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067439+0000","last_change":"2024-03-24T12:15:39.224441+0000","last_active":"2024-03-24T12:16:22.067439+0000","last_peered":"2024-03-24T12:16:22.067439+0000","last_clean":"2024-03-24T12:16:22.067439+0000","last_became_active":"2024-03-24T12:15:05.220086+0000","last_became_peered":"2024-03-24T12:15:05.220086+0000","last_unstale":"2024-03-24T12:16:22.067439+0000","last_undegraded":"2024-03-24T12:16:22.067439+0000","last_fullsized":"2024-03-24T12:16:22.067439+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'14","last_scrub_stamp":"2024-03-24T12:15:39.224267+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:15:39.224267+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:40.001209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":943,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206894+0000","last_change":"2024-03-24T12:15:31.196209+0000","last_active":"2024-03-24T12:16:22.206894+0000","last_peered":"2024-03-24T12:16:22.206894+0000","last_clean":"2024-03-24T12:16:22.206894+0000","last_became_active":"2024-03-24T12:15:15.273721+0000","last_became_peered":"2024-03-24T12:15:15.273721+0000","last_unstale":"2024-03-24T12:16:22.206894+0000","last_undegraded":"2024-03-24T12:16:22.206894+0000","last_fullsized":"2024-03-24T12:16:22.206894+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:31.196132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:15:31.196132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:48.882517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":925,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192441+0000","last_change":"2024-03-24T12:15:43.074062+0000","last_active":"2024-03-24T12:16:22.192441+0000","last_peered":"2024-03-24T12:16:22.192441+0000","last_clean":"2024-03-24T12:16:22.192441+0000","last_became_active":"2024-03-24T12:15:43.073695+0000","last_became_peered":"2024-03-24T12:15:43.073695+0000","last_unstale":"2024-03-24T12:16:22.192441+0000","last_undegraded":"2024-03-24T12:16:22.192441+0000","last_fullsized":"2024-03-24T12:16:22.192441+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:30.173007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:30.173007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:47.767908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1050,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358167+0000","last_change":"2024-03-24T12:16:07.397051+0000","last_active":"2024-03-24T12:16:22.358167+0000","last_peered":"2024-03-24T12:16:22.358167+0000","last_clean":"2024-03-24T12:16:22.358167+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:16:22.358167+0000","last_undegraded":"2024-03-24T12:16:22.358167+0000","last_fullsized":"2024-03-24T12:16:22.358167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:07.396899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:16:07.396899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:29.374194+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"450'27","reported_seq":1070,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066257+0000","last_change":"2024-03-24T12:16:22.041049+0000","last_active":"2024-03-24T12:16:22.066257+0000","last_peered":"2024-03-24T12:16:22.066257+0000","last_clean":"2024-03-24T12:16:22.066257+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:16:22.066257+0000","last_undegraded":"2024-03-24T12:16:22.066257+0000","last_fullsized":"2024-03-24T12:16:22.066257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'27","last_scrub_stamp":"2024-03-24T12:16:22.040954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:22.040954+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:27.335762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":940,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:24.375561+0000","last_change":"2024-03-24T12:16:24.375584+0000","last_active":"2024-03-24T12:16:24.375561+0000","last_peered":"2024-03-24T12:16:24.375561+0000","last_clean":"2024-03-24T12:16:24.375561+0000","last_became_active":"2024-03-24T12:15:43.073733+0000","last_became_peered":"2024-03-24T12:15:43.073733+0000","last_unstale":"2024-03-24T12:16:24.375561+0000","last_undegraded":"2024-03-24T12:16:24.375561+0000","last_fullsized":"2024-03-24T12:16:24.375561+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:24.375491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_clean_scrub_stamp":"2024-03-24T12:16:24.375491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:33.764712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1054,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358373+0000","last_change":"2024-03-24T12:15:32.672271+0000","last_active":"2024-03-24T12:16:22.358373+0000","last_peered":"2024-03-24T12:16:22.358373+0000","last_clean":"2024-03-24T12:16:22.358373+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:16:22.358373+0000","last_undegraded":"2024-03-24T12:16:22.358373+0000","last_fullsized":"2024-03-24T12:16:22.358373+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:32.672168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:15:32.672168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:58.792284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1019,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067277+0000","last_change":"2024-03-24T12:16:12.261196+0000","last_active":"2024-03-24T12:16:22.067277+0000","last_peered":"2024-03-24T12:16:22.067277+0000","last_clean":"2024-03-24T12:16:22.067277+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:16:22.067277+0000","last_undegraded":"2024-03-24T12:16:22.067277+0000","last_fullsized":"2024-03-24T12:16:22.067277+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:12.261087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:12.261087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:37.661608+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"450'35","reported_seq":1093,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067532+0000","last_change":"2024-03-24T12:15:42.472780+0000","last_active":"2024-03-24T12:16:22.067532+0000","last_peered":"2024-03-24T12:16:22.067532+0000","last_clean":"2024-03-24T12:16:22.067532+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:16:22.067532+0000","last_undegraded":"2024-03-24T12:16:22.067532+0000","last_fullsized":"2024-03-24T12:16:22.067532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:15:42.472681+0000","last_deep_scrub":"450'35","last_deep_scrub_stamp":"2024-03-24T12:14:22.685599+0000","last_clean_scrub_stamp":"2024-03-24T12:15:42.472681+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:10.220741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":889,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206191+0000","last_change":"2024-03-24T12:15:46.179963+0000","last_active":"2024-03-24T12:16:22.206191+0000","last_peered":"2024-03-24T12:16:22.206191+0000","last_clean":"2024-03-24T12:16:22.206191+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:16:22.206191+0000","last_undegraded":"2024-03-24T12:16:22.206191+0000","last_fullsized":"2024-03-24T12:16:22.206191+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:46.179863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:46.179863+0000","last_clean_scrub_stamp":"2024-03-24T12:15:46.179863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:56.746059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"525'11","reported_seq":1205,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487520+0000","last_change":"2024-03-24T12:16:22.360528+0000","last_active":"2024-03-24T12:16:35.487520+0000","last_peered":"2024-03-24T12:16:35.487520+0000","last_clean":"2024-03-24T12:16:35.487520+0000","last_became_active":"2024-03-24T12:15:43.070184+0000","last_became_peered":"2024-03-24T12:15:43.070184+0000","last_unstale":"2024-03-24T12:16:35.487520+0000","last_undegraded":"2024-03-24T12:16:35.487520+0000","last_fullsized":"2024-03-24T12:16:35.487520+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'11","last_scrub_stamp":"2024-03-24T12:16:22.360425+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:16:22.360425+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:24.677271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":1068,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:30.703494+0000","last_change":"2024-03-24T12:16:30.703525+0000","last_active":"2024-03-24T12:16:30.703494+0000","last_peered":"2024-03-24T12:16:30.703494+0000","last_clean":"2024-03-24T12:16:30.703494+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:16:30.703494+0000","last_undegraded":"2024-03-24T12:16:30.703494+0000","last_fullsized":"2024-03-24T12:16:30.703494+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:16:30.703415+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:16:30.703415+0000","last_clean_scrub_stamp":"2024-03-24T12:16:30.703415+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:41.855797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"450'27","reported_seq":991,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:26.357997+0000","last_change":"2024-03-24T12:16:26.358035+0000","last_active":"2024-03-24T12:16:26.357997+0000","last_peered":"2024-03-24T12:16:26.357997+0000","last_clean":"2024-03-24T12:16:26.357997+0000","last_became_active":"2024-03-24T12:15:43.074105+0000","last_became_peered":"2024-03-24T12:15:43.074105+0000","last_unstale":"2024-03-24T12:16:26.357997+0000","last_undegraded":"2024-03-24T12:16:26.357997+0000","last_fullsized":"2024-03-24T12:16:26.357997+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'27","last_scrub_stamp":"2024-03-24T12:16:26.357929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:26.357929+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:50.180785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":1032,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066120+0000","last_change":"2024-03-24T12:16:18.048156+0000","last_active":"2024-03-24T12:16:22.066120+0000","last_peered":"2024-03-24T12:16:22.066120+0000","last_clean":"2024-03-24T12:16:22.066120+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:16:22.066120+0000","last_undegraded":"2024-03-24T12:16:22.066120+0000","last_fullsized":"2024-03-24T12:16:22.066120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:16:18.048062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:18.048062+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:42.593354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1063,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067621+0000","last_change":"2024-03-24T12:15:24.522375+0000","last_active":"2024-03-24T12:16:22.067621+0000","last_peered":"2024-03-24T12:16:22.067621+0000","last_clean":"2024-03-24T12:16:22.067621+0000","last_became_active":"2024-03-24T12:15:15.276200+0000","last_became_peered":"2024-03-24T12:15:15.276200+0000","last_unstale":"2024-03-24T12:16:22.067621+0000","last_undegraded":"2024-03-24T12:16:22.067621+0000","last_fullsized":"2024-03-24T12:16:22.067621+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:24.522298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:15:24.522298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.077692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":647,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194809+0000","last_change":"2024-03-24T12:15:52.461926+0000","last_active":"2024-03-24T12:16:22.194809+0000","last_peered":"2024-03-24T12:16:22.194809+0000","last_clean":"2024-03-24T12:16:22.194809+0000","last_became_active":"2024-03-24T12:15:11.732134+0000","last_became_peered":"2024-03-24T12:15:11.732134+0000","last_unstale":"2024-03-24T12:16:22.194809+0000","last_undegraded":"2024-03-24T12:16:22.194809+0000","last_fullsized":"2024-03-24T12:16:22.194809+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:52.461807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:15:52.461807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:19.717802+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"450'46","reported_seq":115,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.068230+0000","last_change":"2024-03-24T12:16:18.531346+0000","last_active":"2024-03-24T12:16:22.068230+0000","last_peered":"2024-03-24T12:16:22.068230+0000","last_clean":"2024-03-24T12:16:22.068230+0000","last_became_active":"2024-03-24T12:15:58.147597+0000","last_became_peered":"2024-03-24T12:15:58.147597+0000","last_unstale":"2024-03-24T12:16:22.068230+0000","last_undegraded":"2024-03-24T12:16:22.068230+0000","last_fullsized":"2024-03-24T12:16:22.068230+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'46","last_scrub_stamp":"2024-03-24T12:16:18.531253+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:16:18.531253+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:31.798219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":933,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192808+0000","last_change":"2024-03-24T12:15:52.467989+0000","last_active":"2024-03-24T12:16:22.192808+0000","last_peered":"2024-03-24T12:16:22.192808+0000","last_clean":"2024-03-24T12:16:22.192808+0000","last_became_active":"2024-03-24T12:15:43.070069+0000","last_became_peered":"2024-03-24T12:15:43.070069+0000","last_unstale":"2024-03-24T12:16:22.192808+0000","last_undegraded":"2024-03-24T12:16:22.192808+0000","last_fullsized":"2024-03-24T12:16:22.192808+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:52.467912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:15:52.467912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:06.639003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":806,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.262983+0000","last_change":"2024-03-24T12:16:35.263003+0000","last_active":"2024-03-24T12:16:35.262983+0000","last_peered":"2024-03-24T12:16:35.262983+0000","last_clean":"2024-03-24T12:16:35.262983+0000","last_became_active":"2024-03-24T12:15:11.249540+0000","last_became_peered":"2024-03-24T12:15:11.249540+0000","last_unstale":"2024-03-24T12:16:35.262983+0000","last_undegraded":"2024-03-24T12:16:35.262983+0000","last_fullsized":"2024-03-24T12:16:35.262983+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:35.262924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:16:35.262924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:01.953332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1037,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355603+0000","last_change":"2024-03-24T12:15:38.352890+0000","last_active":"2024-03-24T12:16:22.355603+0000","last_peered":"2024-03-24T12:16:22.355603+0000","last_clean":"2024-03-24T12:16:22.355603+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:16:22.355603+0000","last_undegraded":"2024-03-24T12:16:22.355603+0000","last_fullsized":"2024-03-24T12:16:22.355603+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:38.352742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:38.352742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:39.207341+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"450'29","reported_seq":1090,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066262+0000","last_change":"2024-03-24T12:15:21.150250+0000","last_active":"2024-03-24T12:16:22.066262+0000","last_peered":"2024-03-24T12:16:22.066262+0000","last_clean":"2024-03-24T12:16:22.066262+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:16:22.066262+0000","last_undegraded":"2024-03-24T12:16:22.066262+0000","last_fullsized":"2024-03-24T12:16:22.066262+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'29","last_scrub_stamp":"2024-03-24T12:15:21.150152+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:15:21.150152+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:40.365739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":1044,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:24.095474+0000","last_change":"2024-03-24T12:16:24.095501+0000","last_active":"2024-03-24T12:16:24.095474+0000","last_peered":"2024-03-24T12:16:24.095474+0000","last_clean":"2024-03-24T12:16:24.095474+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:16:24.095474+0000","last_undegraded":"2024-03-24T12:16:24.095474+0000","last_fullsized":"2024-03-24T12:16:24.095474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:16:24.095405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:24.095405+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:30.400702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1044,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355195+0000","last_change":"2024-03-24T12:15:59.275185+0000","last_active":"2024-03-24T12:16:22.355195+0000","last_peered":"2024-03-24T12:16:22.355195+0000","last_clean":"2024-03-24T12:16:22.355195+0000","last_became_active":"2024-03-24T12:15:15.275842+0000","last_became_peered":"2024-03-24T12:15:15.275842+0000","last_unstale":"2024-03-24T12:16:22.355195+0000","last_undegraded":"2024-03-24T12:16:22.355195+0000","last_fullsized":"2024-03-24T12:16:22.355195+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:59.275042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:59.275042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:24.469337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1032,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065795+0000","last_change":"2024-03-24T12:15:56.422691+0000","last_active":"2024-03-24T12:16:22.065795+0000","last_peered":"2024-03-24T12:16:22.065795+0000","last_clean":"2024-03-24T12:16:22.065795+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:16:22.065795+0000","last_undegraded":"2024-03-24T12:16:22.065795+0000","last_fullsized":"2024-03-24T12:16:22.065795+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:56.422585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:56.422585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:20.505323+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"450'21","reported_seq":1072,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:36.471903+0000","last_change":"2024-03-24T12:16:36.471949+0000","last_active":"2024-03-24T12:16:36.471903+0000","last_peered":"2024-03-24T12:16:36.471903+0000","last_clean":"2024-03-24T12:16:36.471903+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:16:36.471903+0000","last_undegraded":"2024-03-24T12:16:36.471903+0000","last_fullsized":"2024-03-24T12:16:36.471903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:16:36.471828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:36.471828+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:46.178405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1073,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358103+0000","last_change":"2024-03-24T12:15:39.458306+0000","last_active":"2024-03-24T12:16:22.358103+0000","last_peered":"2024-03-24T12:16:22.358103+0000","last_clean":"2024-03-24T12:16:22.358103+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:16:22.358103+0000","last_undegraded":"2024-03-24T12:16:22.358103+0000","last_fullsized":"2024-03-24T12:16:22.358103+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:39.458228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:15:39.458228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:53.581008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":950,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206564+0000","last_change":"2024-03-24T12:15:49.170901+0000","last_active":"2024-03-24T12:16:22.206564+0000","last_peered":"2024-03-24T12:16:22.206564+0000","last_clean":"2024-03-24T12:16:22.206564+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:16:22.206564+0000","last_undegraded":"2024-03-24T12:16:22.206564+0000","last_fullsized":"2024-03-24T12:16:22.206564+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:49.170749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:49.170749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:55.259771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1033,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:32.762832+0000","last_change":"2024-03-24T12:16:32.762832+0000","last_active":"2024-03-24T12:16:32.762832+0000","last_peered":"2024-03-24T12:16:32.762832+0000","last_clean":"2024-03-24T12:16:32.762832+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:16:32.762832+0000","last_undegraded":"2024-03-24T12:16:32.762832+0000","last_fullsized":"2024-03-24T12:16:32.762832+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:32.762735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:16:32.762735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:02.500477+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"450'11","reported_seq":1080,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355251+0000","last_change":"2024-03-24T12:16:10.346536+0000","last_active":"2024-03-24T12:16:22.355251+0000","last_peered":"2024-03-24T12:16:22.355251+0000","last_clean":"2024-03-24T12:16:22.355251+0000","last_became_active":"2024-03-24T12:15:05.234148+0000","last_became_peered":"2024-03-24T12:15:05.234148+0000","last_unstale":"2024-03-24T12:16:22.355251+0000","last_undegraded":"2024-03-24T12:16:22.355251+0000","last_fullsized":"2024-03-24T12:16:22.355251+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'11","last_scrub_stamp":"2024-03-24T12:16:10.346429+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:16:10.346429+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:32.816914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1077,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:31.750817+0000","last_change":"2024-03-24T12:16:31.750856+0000","last_active":"2024-03-24T12:16:31.750817+0000","last_peered":"2024-03-24T12:16:31.750817+0000","last_clean":"2024-03-24T12:16:31.750817+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:16:31.750817+0000","last_undegraded":"2024-03-24T12:16:31.750817+0000","last_fullsized":"2024-03-24T12:16:31.750817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:31.750736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:16:31.750736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:57.025898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1081,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:29.736170+0000","last_change":"2024-03-24T12:16:29.736218+0000","last_active":"2024-03-24T12:16:29.736170+0000","last_peered":"2024-03-24T12:16:29.736170+0000","last_clean":"2024-03-24T12:16:29.736170+0000","last_became_active":"2024-03-24T12:15:05.225818+0000","last_became_peered":"2024-03-24T12:15:05.225818+0000","last_unstale":"2024-03-24T12:16:29.736170+0000","last_undegraded":"2024-03-24T12:16:29.736170+0000","last_fullsized":"2024-03-24T12:16:29.736170+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:29.736056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:02.631340+0000","last_clean_scrub_stamp":"2024-03-24T12:16:29.736056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:37.810931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":968,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206215+0000","last_change":"2024-03-24T12:15:21.206697+0000","last_active":"2024-03-24T12:16:22.206215+0000","last_peered":"2024-03-24T12:16:22.206215+0000","last_clean":"2024-03-24T12:16:22.206215+0000","last_became_active":"2024-03-24T12:15:05.212886+0000","last_became_peered":"2024-03-24T12:15:05.212886+0000","last_unstale":"2024-03-24T12:16:22.206215+0000","last_undegraded":"2024-03-24T12:16:22.206215+0000","last_fullsized":"2024-03-24T12:16:22.206215+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:21.206577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:15:21.206577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:50.971021+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"450'25","reported_seq":1109,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.354838+0000","last_change":"2024-03-24T12:16:16.303225+0000","last_active":"2024-03-24T12:16:22.354838+0000","last_peered":"2024-03-24T12:16:22.354838+0000","last_clean":"2024-03-24T12:16:22.354838+0000","last_became_active":"2024-03-24T12:15:05.232578+0000","last_became_peered":"2024-03-24T12:15:05.232578+0000","last_unstale":"2024-03-24T12:16:22.354838+0000","last_undegraded":"2024-03-24T12:16:22.354838+0000","last_fullsized":"2024-03-24T12:16:22.354838+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'25","last_scrub_stamp":"2024-03-24T12:16:16.303118+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:16:16.303118+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:37.404365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1042,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066682+0000","last_change":"2024-03-24T12:15:50.283561+0000","last_active":"2024-03-24T12:16:22.066682+0000","last_peered":"2024-03-24T12:16:22.066682+0000","last_clean":"2024-03-24T12:16:22.066682+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:16:22.066682+0000","last_undegraded":"2024-03-24T12:16:22.066682+0000","last_fullsized":"2024-03-24T12:16:22.066682+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:50.283462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:50.283462+0000","last_clean_scrub_stamp":"2024-03-24T12:15:50.283462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:11.421671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":457,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194391+0000","last_change":"2024-03-24T12:15:11.732158+0000","last_active":"2024-03-24T12:16:22.194391+0000","last_peered":"2024-03-24T12:16:22.194391+0000","last_clean":"2024-03-24T12:16:22.194391+0000","last_became_active":"2024-03-24T12:15:11.731693+0000","last_became_peered":"2024-03-24T12:15:11.731693+0000","last_unstale":"2024-03-24T12:16:22.194391+0000","last_undegraded":"2024-03-24T12:16:22.194391+0000","last_fullsized":"2024-03-24T12:16:22.194391+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:10.187179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:10.187179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:37.086575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1043,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.354978+0000","last_change":"2024-03-24T12:15:34.308108+0000","last_active":"2024-03-24T12:16:22.354978+0000","last_peered":"2024-03-24T12:16:22.354978+0000","last_clean":"2024-03-24T12:16:22.354978+0000","last_became_active":"2024-03-24T12:15:05.234180+0000","last_became_peered":"2024-03-24T12:15:05.234180+0000","last_unstale":"2024-03-24T12:16:22.354978+0000","last_undegraded":"2024-03-24T12:16:22.354978+0000","last_fullsized":"2024-03-24T12:16:22.354978+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:34.307998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:34.307998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:37.500517+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"450'46","reported_seq":93,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065776+0000","last_change":"2024-03-24T12:15:58.231242+0000","last_active":"2024-03-24T12:16:22.065776+0000","last_peered":"2024-03-24T12:16:22.065776+0000","last_clean":"2024-03-24T12:16:22.065776+0000","last_became_active":"2024-03-24T12:15:15.271403+0000","last_became_peered":"2024-03-24T12:15:15.271403+0000","last_unstale":"2024-03-24T12:16:22.065776+0000","last_undegraded":"2024-03-24T12:16:22.065776+0000","last_fullsized":"2024-03-24T12:16:22.065776+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'46","last_scrub_stamp":"2024-03-24T12:15:58.231106+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:15:58.231106+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:16.795636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1046,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.679522+0000","last_change":"2024-03-24T12:16:35.679543+0000","last_active":"2024-03-24T12:16:35.679522+0000","last_peered":"2024-03-24T12:16:35.679522+0000","last_clean":"2024-03-24T12:16:35.679522+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:16:35.679522+0000","last_undegraded":"2024-03-24T12:16:35.679522+0000","last_fullsized":"2024-03-24T12:16:35.679522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:35.679460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:16:35.679460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:46.601461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":501,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065126+0000","last_change":"2024-03-24T12:15:41.243752+0000","last_active":"2024-03-24T12:16:22.065126+0000","last_peered":"2024-03-24T12:16:22.065126+0000","last_clean":"2024-03-24T12:16:22.065126+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:16:22.065126+0000","last_undegraded":"2024-03-24T12:16:22.065126+0000","last_fullsized":"2024-03-24T12:16:22.065126+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:41.243660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:15:41.243660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:47.786288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1026,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:32.244258+0000","last_change":"2024-03-24T12:16:32.244281+0000","last_active":"2024-03-24T12:16:32.244258+0000","last_peered":"2024-03-24T12:16:32.244258+0000","last_clean":"2024-03-24T12:16:32.244258+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:16:32.244258+0000","last_undegraded":"2024-03-24T12:16:32.244258+0000","last_fullsized":"2024-03-24T12:16:32.244258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:32.244183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:32.244183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:00.084367+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":390,"ondisk_log_size":390,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":640,"num_read_kb":635,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1545,"num_read_kb":1302,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":4044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":1096,"ondisk_log_size":1096,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005354,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880692,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864460,"statfs":{"total":95995035648,"available":95093207040,"internally_reserved":0,"allocated":1290240,"data_stored":823567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":900528586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332401,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862320,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92882832,"statfs":{"total":95995035648,"available":95112019968,"internally_reserved":0,"allocated":958464,"data_stored":495705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":882047430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575177,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960908,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92784244,"statfs":{"total":95995035648,"available":95011065856,"internally_reserved":0,"allocated":987136,"data_stored":495642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738702,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970616,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92774536,"statfs":{"total":95995035648,"available":95001124864,"internally_reserved":0,"allocated":966656,"data_stored":494864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411653,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636276,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93108876,"statfs":{"total":95995035648,"available":95343489024,"internally_reserved":0,"allocated":1224704,"data_stored":758172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117311,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939280,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92805872,"statfs":{"total":95995035648,"available":95033212928,"internally_reserved":0,"allocated":1581056,"data_stored":1086502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":960231879},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921082,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880384,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864768,"statfs":{"total":95995035648,"available":95093522432,"internally_reserved":0,"allocated":974848,"data_stored":495319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455587,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":390768,"kb_used_data":1512,"kb_used_omap":7,"kb_used_meta":389240,"kb_avail":93354384,"statfs":{"total":95995035648,"available":95594889216,"internally_reserved":0,"allocated":1548288,"data_stored":1085551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":398581872},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:16:45.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:16:45.371 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:45.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[30223]: pgmap v1176: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[30223]: 2.13 scrub starts 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[30223]: 2.13 scrub ok 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[30223]: 6.f scrub ok 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[30223]: from='client.36326 v1:172.21.15.64:0/1360960004' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[30223]: 6.5 scrub starts 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[30223]: 6.5 scrub ok 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[42243]: pgmap v1176: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[42243]: 2.13 scrub starts 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[42243]: 2.13 scrub ok 2024-03-24T12:16:45.494 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[42243]: 6.f scrub ok 2024-03-24T12:16:45.495 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[42243]: from='client.36326 v1:172.21.15.64:0/1360960004' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:16:45.495 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[42243]: 6.5 scrub starts 2024-03-24T12:16:45.495 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:45 smithi064 ceph-mon[42243]: 6.5 scrub ok 2024-03-24T12:16:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:45 smithi193 ceph-mon[33593]: pgmap v1176: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:45 smithi193 ceph-mon[33593]: 2.13 scrub starts 2024-03-24T12:16:45.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:45 smithi193 ceph-mon[33593]: 2.13 scrub ok 2024-03-24T12:16:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:45 smithi193 ceph-mon[33593]: 6.f scrub ok 2024-03-24T12:16:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:45 smithi193 ceph-mon[33593]: from='client.36326 v1:172.21.15.64:0/1360960004' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:16:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:45 smithi193 ceph-mon[33593]: 6.5 scrub starts 2024-03-24T12:16:45.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:45 smithi193 ceph-mon[33593]: 6.5 scrub ok 2024-03-24T12:16:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:46 smithi064 ceph-mon[30223]: 4.10 scrub starts 2024-03-24T12:16:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:46 smithi064 ceph-mon[30223]: 4.10 scrub ok 2024-03-24T12:16:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:46 smithi064 ceph-mon[30223]: 1.1 scrub starts 2024-03-24T12:16:46.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:46 smithi064 ceph-mon[30223]: 1.1 scrub ok 2024-03-24T12:16:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:46 smithi064 ceph-mon[42243]: 4.10 scrub starts 2024-03-24T12:16:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:46 smithi064 ceph-mon[42243]: 4.10 scrub ok 2024-03-24T12:16:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:46 smithi064 ceph-mon[42243]: 1.1 scrub starts 2024-03-24T12:16:46.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:46 smithi064 ceph-mon[42243]: 1.1 scrub ok 2024-03-24T12:16:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:46 smithi193 ceph-mon[33593]: 4.10 scrub starts 2024-03-24T12:16:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:46 smithi193 ceph-mon[33593]: 4.10 scrub ok 2024-03-24T12:16:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:46 smithi193 ceph-mon[33593]: 1.1 scrub starts 2024-03-24T12:16:46.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:46 smithi193 ceph-mon[33593]: 1.1 scrub ok 2024-03-24T12:16:47.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[30223]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[30223]: 4.1d scrub starts 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[30223]: 4.1d scrub ok 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[30223]: 6.14 scrub starts 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[30223]: 6.14 scrub ok 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[30223]: 1.2 scrub starts 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[30223]: 1.2 scrub ok 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[42243]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[42243]: 4.1d scrub starts 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[42243]: 4.1d scrub ok 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[42243]: 6.14 scrub starts 2024-03-24T12:16:47.302 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[42243]: 6.14 scrub ok 2024-03-24T12:16:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[42243]: 1.2 scrub starts 2024-03-24T12:16:47.303 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-mon[42243]: 1.2 scrub ok 2024-03-24T12:16:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-24T12:16:47.455 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-24T12:16:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:47 smithi193 ceph-mon[33593]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:47 smithi193 ceph-mon[33593]: 4.1d scrub starts 2024-03-24T12:16:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:47 smithi193 ceph-mon[33593]: 4.1d scrub ok 2024-03-24T12:16:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:47 smithi193 ceph-mon[33593]: 6.14 scrub starts 2024-03-24T12:16:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:47 smithi193 ceph-mon[33593]: 6.14 scrub ok 2024-03-24T12:16:47.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:47 smithi193 ceph-mon[33593]: 1.2 scrub starts 2024-03-24T12:16:47.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:47 smithi193 ceph-mon[33593]: 1.2 scrub ok 2024-03-24T12:16:47.769 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:47 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: ::ffff:172.21.15.193 - - [24/Mar/2024:12:16:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:16:48.225 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1177,"stamp":"2024-03-24T12:16:45.643247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2202,"num_read_kb":1954,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":333387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1584,"ondisk_log_size":1584,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6521244,"kb_used_data":9308,"kb_used_omap":18,"kb_used_meta":6511853,"kb_avail":743439972,"statfs":{"total":767960285184,"available":761282531328,"internally_reserved":0,"allocated":9531392,"data_stored":5735322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19237,"internal_metadata":6668137691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003269"},"pg_stats":[{"pgid":"5.9","version":"176'9","reported_seq":1366,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487511+0000","last_change":"2024-03-24T12:16:13.484196+0000","last_active":"2024-03-24T12:16:35.487511+0000","last_peered":"2024-03-24T12:16:35.487511+0000","last_clean":"2024-03-24T12:16:35.487511+0000","last_became_active":"2024-03-24T11:52:19.386625+0000","last_became_peered":"2024-03-24T11:52:19.386625+0000","last_unstale":"2024-03-24T12:16:35.487511+0000","last_undegraded":"2024-03-24T12:16:35.487511+0000","last_fullsized":"2024-03-24T12:16:35.487511+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-24T12:16:13.484090+0000","last_deep_scrub":"176'9","last_deep_scrub_stamp":"2024-03-24T12:08:46.189592+0000","last_clean_scrub_stamp":"2024-03-24T12:16:13.484090+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:42.715707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"540'46","reported_seq":992,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:25.276372+0000","last_change":"2024-03-24T12:16:25.276426+0000","last_active":"2024-03-24T12:16:25.276372+0000","last_peered":"2024-03-24T12:16:25.276372+0000","last_clean":"2024-03-24T12:16:25.276372+0000","last_became_active":"2024-03-24T12:08:36.152816+0000","last_became_peered":"2024-03-24T12:08:36.152816+0000","last_unstale":"2024-03-24T12:16:25.276372+0000","last_undegraded":"2024-03-24T12:16:25.276372+0000","last_fullsized":"2024-03-24T12:16:25.276372+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"540'46","last_scrub_stamp":"2024-03-24T12:16:25.276296+0000","last_deep_scrub":"540'46","last_deep_scrub_stamp":"2024-03-24T12:16:25.276296+0000","last_clean_scrub_stamp":"2024-03-24T12:16:25.276296+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:33.817464+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067547+0000","last_change":"2024-03-24T12:15:18.254993+0000","last_active":"2024-03-24T12:16:22.067547+0000","last_peered":"2024-03-24T12:16:22.067547+0000","last_clean":"2024-03-24T12:16:22.067547+0000","last_became_active":"2024-03-24T12:15:15.281302+0000","last_became_peered":"2024-03-24T12:15:15.281302+0000","last_unstale":"2024-03-24T12:16:22.067547+0000","last_undegraded":"2024-03-24T12:16:22.067547+0000","last_fullsized":"2024-03-24T12:16:22.067547+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:18.254906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:15:18.254906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:43.101507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":951,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206478+0000","last_change":"2024-03-24T12:15:40.268528+0000","last_active":"2024-03-24T12:16:22.206478+0000","last_peered":"2024-03-24T12:16:22.206478+0000","last_clean":"2024-03-24T12:16:22.206478+0000","last_became_active":"2024-03-24T12:08:36.166674+0000","last_became_peered":"2024-03-24T12:08:36.166674+0000","last_unstale":"2024-03-24T12:16:22.206478+0000","last_undegraded":"2024-03-24T12:16:22.206478+0000","last_fullsized":"2024-03-24T12:16:22.206478+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:40.268416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:40.268416+0000","last_clean_scrub_stamp":"2024-03-24T12:15:40.268416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:07.146762+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"450'21","reported_seq":1075,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066792+0000","last_change":"2024-03-24T12:15:47.544260+0000","last_active":"2024-03-24T12:16:22.066792+0000","last_peered":"2024-03-24T12:16:22.066792+0000","last_clean":"2024-03-24T12:16:22.066792+0000","last_became_active":"2024-03-24T11:52:17.358935+0000","last_became_peered":"2024-03-24T11:52:17.358935+0000","last_unstale":"2024-03-24T12:16:22.066792+0000","last_undegraded":"2024-03-24T12:16:22.066792+0000","last_fullsized":"2024-03-24T12:16:22.066792+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:15:47.544155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:47.544155+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:58.728413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194918+0000","last_change":"2024-03-24T12:15:42.519236+0000","last_active":"2024-03-24T12:16:22.194918+0000","last_peered":"2024-03-24T12:16:22.194918+0000","last_clean":"2024-03-24T12:16:22.194918+0000","last_became_active":"2024-03-24T12:15:05.225551+0000","last_became_peered":"2024-03-24T12:15:05.225551+0000","last_unstale":"2024-03-24T12:16:22.194918+0000","last_undegraded":"2024-03-24T12:16:22.194918+0000","last_fullsized":"2024-03-24T12:16:22.194918+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:42.519153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:28.145774+0000","last_clean_scrub_stamp":"2024-03-24T12:15:42.519153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:54.613976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":525,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.068126+0000","last_change":"2024-03-24T12:16:20.605299+0000","last_active":"2024-03-24T12:16:22.068126+0000","last_peered":"2024-03-24T12:16:22.068126+0000","last_clean":"2024-03-24T12:16:22.068126+0000","last_became_active":"2024-03-24T12:15:19.067107+0000","last_became_peered":"2024-03-24T12:15:19.067107+0000","last_unstale":"2024-03-24T12:16:22.068126+0000","last_undegraded":"2024-03-24T12:16:22.068126+0000","last_fullsized":"2024-03-24T12:16:22.068126+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:20.605210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:49.261483+0000","last_clean_scrub_stamp":"2024-03-24T12:16:20.605210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:41.077971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1046,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:42.797324+0000","last_change":"2024-03-24T12:16:42.797346+0000","last_active":"2024-03-24T12:16:42.797324+0000","last_peered":"2024-03-24T12:16:42.797324+0000","last_clean":"2024-03-24T12:16:42.797324+0000","last_became_active":"2024-03-24T12:15:15.274417+0000","last_became_peered":"2024-03-24T12:15:15.274417+0000","last_unstale":"2024-03-24T12:16:42.797324+0000","last_undegraded":"2024-03-24T12:16:42.797324+0000","last_fullsized":"2024-03-24T12:16:42.797324+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:42.797241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:42.797241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:56.797811+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"450'60","reported_seq":1010,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192772+0000","last_change":"2024-03-24T12:15:43.068619+0000","last_active":"2024-03-24T12:16:22.192772+0000","last_peered":"2024-03-24T12:16:22.192772+0000","last_clean":"2024-03-24T12:16:22.192772+0000","last_became_active":"2024-03-24T12:15:43.068230+0000","last_became_peered":"2024-03-24T12:15:43.068230+0000","last_unstale":"2024-03-24T12:16:22.192772+0000","last_undegraded":"2024-03-24T12:16:22.192772+0000","last_fullsized":"2024-03-24T12:16:22.192772+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'60","last_scrub_stamp":"2024-03-24T12:15:30.497376+0000","last_deep_scrub":"296'44","last_deep_scrub_stamp":"2024-03-24T12:05:14.441098+0000","last_clean_scrub_stamp":"2024-03-24T12:15:30.497376+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:44.841312+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.068230+0000","last_change":"2024-03-24T12:15:49.558254+0000","last_active":"2024-03-24T12:16:22.068230+0000","last_peered":"2024-03-24T12:16:22.068230+0000","last_clean":"2024-03-24T12:16:22.068230+0000","last_became_active":"2024-03-24T12:15:15.277545+0000","last_became_peered":"2024-03-24T12:15:15.277545+0000","last_unstale":"2024-03-24T12:16:22.068230+0000","last_undegraded":"2024-03-24T12:16:22.068230+0000","last_fullsized":"2024-03-24T12:16:22.068230+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:49.558150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:14.639264+0000","last_clean_scrub_stamp":"2024-03-24T12:15:49.558150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:04.882336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066248+0000","last_change":"2024-03-24T12:15:41.431643+0000","last_active":"2024-03-24T12:16:22.066248+0000","last_peered":"2024-03-24T12:16:22.066248+0000","last_clean":"2024-03-24T12:16:22.066248+0000","last_became_active":"2024-03-24T12:15:15.273789+0000","last_became_peered":"2024-03-24T12:15:15.273789+0000","last_unstale":"2024-03-24T12:16:22.066248+0000","last_undegraded":"2024-03-24T12:16:22.066248+0000","last_fullsized":"2024-03-24T12:16:22.066248+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:41.431562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:41.431562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:53.760797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1063,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.357534+0000","last_change":"2024-03-24T12:15:44.491981+0000","last_active":"2024-03-24T12:16:22.357534+0000","last_peered":"2024-03-24T12:16:22.357534+0000","last_clean":"2024-03-24T12:16:22.357534+0000","last_became_active":"2024-03-24T12:15:15.276348+0000","last_became_peered":"2024-03-24T12:15:15.276348+0000","last_unstale":"2024-03-24T12:16:22.357534+0000","last_undegraded":"2024-03-24T12:16:22.357534+0000","last_fullsized":"2024-03-24T12:16:22.357534+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:44.491883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:55:18.299637+0000","last_clean_scrub_stamp":"2024-03-24T12:15:44.491883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:10.857536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"450'53","reported_seq":1157,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355610+0000","last_change":"2024-03-24T12:15:25.155210+0000","last_active":"2024-03-24T12:16:22.355610+0000","last_peered":"2024-03-24T12:16:22.355610+0000","last_clean":"2024-03-24T12:16:22.355610+0000","last_became_active":"2024-03-24T12:15:15.282554+0000","last_became_peered":"2024-03-24T12:15:15.282554+0000","last_unstale":"2024-03-24T12:16:22.355610+0000","last_undegraded":"2024-03-24T12:16:22.355610+0000","last_fullsized":"2024-03-24T12:16:22.355610+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'53","last_scrub_stamp":"2024-03-24T12:15:25.155107+0000","last_deep_scrub":"116'25","last_deep_scrub_stamp":"2024-03-24T12:01:21.467704+0000","last_clean_scrub_stamp":"2024-03-24T12:15:25.155107+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:37.613206+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"262'2","reported_seq":735,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066345+0000","last_change":"2024-03-24T12:15:26.271993+0000","last_active":"2024-03-24T12:16:22.066345+0000","last_peered":"2024-03-24T12:16:22.066345+0000","last_clean":"2024-03-24T12:16:22.066345+0000","last_became_active":"2024-03-24T12:15:19.067448+0000","last_became_peered":"2024-03-24T12:15:19.067448+0000","last_unstale":"2024-03-24T12:16:22.066345+0000","last_undegraded":"2024-03-24T12:16:22.066345+0000","last_fullsized":"2024-03-24T12:16:22.066345+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'2","last_scrub_stamp":"2024-03-24T12:15:26.271874+0000","last_deep_scrub":"262'2","last_deep_scrub_stamp":"2024-03-24T12:03:18.131513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:26.271874+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:55.075355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":790,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194529+0000","last_change":"2024-03-24T12:16:18.178693+0000","last_active":"2024-03-24T12:16:22.194529+0000","last_peered":"2024-03-24T12:16:22.194529+0000","last_clean":"2024-03-24T12:16:22.194529+0000","last_became_active":"2024-03-24T12:15:11.249346+0000","last_became_peered":"2024-03-24T12:15:11.249346+0000","last_unstale":"2024-03-24T12:16:22.194529+0000","last_undegraded":"2024-03-24T12:16:22.194529+0000","last_fullsized":"2024-03-24T12:16:22.194529+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:18.178617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:09.171134+0000","last_clean_scrub_stamp":"2024-03-24T12:16:18.178617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:20.935982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067297+0000","last_change":"2024-03-24T12:15:24.449466+0000","last_active":"2024-03-24T12:16:22.067297+0000","last_peered":"2024-03-24T12:16:22.067297+0000","last_clean":"2024-03-24T12:16:22.067297+0000","last_became_active":"2024-03-24T12:15:15.280821+0000","last_became_peered":"2024-03-24T12:15:15.280821+0000","last_unstale":"2024-03-24T12:16:22.067297+0000","last_undegraded":"2024-03-24T12:16:22.067297+0000","last_fullsized":"2024-03-24T12:16:22.067297+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:24.449337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:24.449337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:37.493390+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'130","reported_seq":982,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206114+0000","last_change":"2024-03-24T12:15:57.165240+0000","last_active":"2024-03-24T12:16:22.206114+0000","last_peered":"2024-03-24T12:16:22.206114+0000","last_clean":"2024-03-24T12:16:22.206114+0000","last_became_active":"2024-03-24T12:08:36.158103+0000","last_became_peered":"2024-03-24T12:08:36.158103+0000","last_unstale":"2024-03-24T12:16:22.206114+0000","last_undegraded":"2024-03-24T12:16:22.206114+0000","last_fullsized":"2024-03-24T12:16:22.206114+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:57.165119+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:57.165119+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:04.254851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"450'35","reported_seq":1081,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355243+0000","last_change":"2024-03-24T12:15:19.211493+0000","last_active":"2024-03-24T12:16:22.355243+0000","last_peered":"2024-03-24T12:16:22.355243+0000","last_clean":"2024-03-24T12:16:22.355243+0000","last_became_active":"2024-03-24T11:52:17.346158+0000","last_became_peered":"2024-03-24T11:52:17.346158+0000","last_unstale":"2024-03-24T12:16:22.355243+0000","last_undegraded":"2024-03-24T12:16:22.355243+0000","last_fullsized":"2024-03-24T12:16:22.355243+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:15:19.211337+0000","last_deep_scrub":"130'15","last_deep_scrub_stamp":"2024-03-24T11:56:28.035859+0000","last_clean_scrub_stamp":"2024-03-24T12:15:19.211337+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:44.078086+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":800,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194372+0000","last_change":"2024-03-24T12:16:17.223324+0000","last_active":"2024-03-24T12:16:22.194372+0000","last_peered":"2024-03-24T12:16:22.194372+0000","last_clean":"2024-03-24T12:16:22.194372+0000","last_became_active":"2024-03-24T12:15:11.657317+0000","last_became_peered":"2024-03-24T12:15:11.657317+0000","last_unstale":"2024-03-24T12:16:22.194372+0000","last_undegraded":"2024-03-24T12:16:22.194372+0000","last_fullsized":"2024-03-24T12:16:22.194372+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:17.223243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:00.667221+0000","last_clean_scrub_stamp":"2024-03-24T12:16:17.223243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:45.816048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":1374,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487316+0000","last_change":"2024-03-24T12:15:37.436529+0000","last_active":"2024-03-24T12:16:35.487316+0000","last_peered":"2024-03-24T12:16:35.487316+0000","last_clean":"2024-03-24T12:16:35.487316+0000","last_became_active":"2024-03-24T12:08:36.160400+0000","last_became_peered":"2024-03-24T12:08:36.160400+0000","last_unstale":"2024-03-24T12:16:35.487316+0000","last_undegraded":"2024-03-24T12:16:35.487316+0000","last_fullsized":"2024-03-24T12:16:35.487316+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:15:37.436402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:37.436402+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:46.798240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"74'1","reported_seq":1073,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355779+0000","last_change":"2024-03-24T12:15:40.311031+0000","last_active":"2024-03-24T12:16:22.355779+0000","last_peered":"2024-03-24T12:16:22.355779+0000","last_clean":"2024-03-24T12:16:22.355779+0000","last_became_active":"2024-03-24T12:15:15.281782+0000","last_became_peered":"2024-03-24T12:15:15.281782+0000","last_unstale":"2024-03-24T12:16:22.355779+0000","last_undegraded":"2024-03-24T12:16:22.355779+0000","last_fullsized":"2024-03-24T12:16:22.355779+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:15:40.310926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:40.310926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:53.249412+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'130","reported_seq":1387,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.357443+0000","last_change":"2024-03-24T12:15:49.490214+0000","last_active":"2024-03-24T12:16:22.357443+0000","last_peered":"2024-03-24T12:16:22.357443+0000","last_clean":"2024-03-24T12:16:22.357443+0000","last_became_active":"2024-03-24T12:15:18.064798+0000","last_became_peered":"2024-03-24T12:15:18.064798+0000","last_unstale":"2024-03-24T12:16:22.357443+0000","last_undegraded":"2024-03-24T12:16:22.357443+0000","last_fullsized":"2024-03-24T12:16:22.357443+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:49.490065+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:16.142552+0000","last_clean_scrub_stamp":"2024-03-24T12:15:49.490065+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.616761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"450'46","reported_seq":1133,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.195001+0000","last_change":"2024-03-24T12:15:50.321951+0000","last_active":"2024-03-24T12:16:22.195001+0000","last_peered":"2024-03-24T12:16:22.195001+0000","last_clean":"2024-03-24T12:16:22.195001+0000","last_became_active":"2024-03-24T11:52:17.358892+0000","last_became_peered":"2024-03-24T11:52:17.358892+0000","last_unstale":"2024-03-24T12:16:22.195001+0000","last_undegraded":"2024-03-24T12:16:22.195001+0000","last_fullsized":"2024-03-24T12:16:22.195001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'46","last_scrub_stamp":"2024-03-24T12:15:50.321819+0000","last_deep_scrub":"130'22","last_deep_scrub_stamp":"2024-03-24T11:56:17.455285+0000","last_clean_scrub_stamp":"2024-03-24T12:15:50.321819+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:17.489542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1078,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194436+0000","last_change":"2024-03-24T12:15:57.255000+0000","last_active":"2024-03-24T12:16:22.194436+0000","last_peered":"2024-03-24T12:16:22.194436+0000","last_clean":"2024-03-24T12:16:22.194436+0000","last_became_active":"2024-03-24T11:52:15.135256+0000","last_became_peered":"2024-03-24T11:52:15.135256+0000","last_unstale":"2024-03-24T12:16:22.194436+0000","last_undegraded":"2024-03-24T12:16:22.194436+0000","last_fullsized":"2024-03-24T12:16:22.194436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:57.254904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:14.593887+0000","last_clean_scrub_stamp":"2024-03-24T12:15:57.254904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:04.398411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'10","reported_seq":1343,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487510+0000","last_change":"2024-03-24T12:15:35.280727+0000","last_active":"2024-03-24T12:16:35.487510+0000","last_peered":"2024-03-24T12:16:35.487510+0000","last_clean":"2024-03-24T12:16:35.487510+0000","last_became_active":"2024-03-24T12:08:36.172816+0000","last_became_peered":"2024-03-24T12:08:36.172816+0000","last_unstale":"2024-03-24T12:16:35.487510+0000","last_undegraded":"2024-03-24T12:16:35.487510+0000","last_fullsized":"2024-03-24T12:16:35.487510+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:15:35.280649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:35.280649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:43.523617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":971,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206987+0000","last_change":"2024-03-24T12:16:00.342458+0000","last_active":"2024-03-24T12:16:22.206987+0000","last_peered":"2024-03-24T12:16:22.206987+0000","last_clean":"2024-03-24T12:16:22.206987+0000","last_became_active":"2024-03-24T12:15:05.212711+0000","last_became_peered":"2024-03-24T12:15:05.212711+0000","last_unstale":"2024-03-24T12:16:22.206987+0000","last_undegraded":"2024-03-24T12:16:22.206987+0000","last_fullsized":"2024-03-24T12:16:22.206987+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:00.342327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:07.943432+0000","last_clean_scrub_stamp":"2024-03-24T12:16:00.342327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:27.837069+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'130","reported_seq":990,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:34.349194+0000","last_change":"2024-03-24T12:16:34.349218+0000","last_active":"2024-03-24T12:16:34.349194+0000","last_peered":"2024-03-24T12:16:34.349194+0000","last_clean":"2024-03-24T12:16:34.349194+0000","last_became_active":"2024-03-24T12:08:36.170488+0000","last_became_peered":"2024-03-24T12:08:36.170488+0000","last_unstale":"2024-03-24T12:16:34.349194+0000","last_undegraded":"2024-03-24T12:16:34.349194+0000","last_fullsized":"2024-03-24T12:16:34.349194+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:16:34.349137+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:16:34.349137+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:52.051167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"450'32","reported_seq":1115,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355533+0000","last_change":"2024-03-24T12:16:06.232953+0000","last_active":"2024-03-24T12:16:22.355533+0000","last_peered":"2024-03-24T12:16:22.355533+0000","last_clean":"2024-03-24T12:16:22.355533+0000","last_became_active":"2024-03-24T12:15:05.234577+0000","last_became_peered":"2024-03-24T12:15:05.234577+0000","last_unstale":"2024-03-24T12:16:22.355533+0000","last_undegraded":"2024-03-24T12:16:22.355533+0000","last_fullsized":"2024-03-24T12:16:22.355533+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'32","last_scrub_stamp":"2024-03-24T12:16:06.232847+0000","last_deep_scrub":"178'16","last_deep_scrub_stamp":"2024-03-24T11:59:18.653047+0000","last_clean_scrub_stamp":"2024-03-24T12:16:06.232847+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:26.842474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1083,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.663285+0000","last_change":"2024-03-24T12:16:22.663311+0000","last_active":"2024-03-24T12:16:22.663285+0000","last_peered":"2024-03-24T12:16:22.663285+0000","last_clean":"2024-03-24T12:16:22.663285+0000","last_became_active":"2024-03-24T12:15:05.214715+0000","last_became_peered":"2024-03-24T12:15:05.214715+0000","last_unstale":"2024-03-24T12:16:22.663285+0000","last_undegraded":"2024-03-24T12:16:22.663285+0000","last_fullsized":"2024-03-24T12:16:22.663285+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:22.663221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:08.571841+0000","last_clean_scrub_stamp":"2024-03-24T12:16:22.663221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:36.562715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":955,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:27.232721+0000","last_change":"2024-03-24T12:16:27.232768+0000","last_active":"2024-03-24T12:16:27.232721+0000","last_peered":"2024-03-24T12:16:27.232721+0000","last_clean":"2024-03-24T12:16:27.232721+0000","last_became_active":"2024-03-24T12:08:36.162999+0000","last_became_peered":"2024-03-24T12:08:36.162999+0000","last_unstale":"2024-03-24T12:16:27.232721+0000","last_undegraded":"2024-03-24T12:16:27.232721+0000","last_fullsized":"2024-03-24T12:16:27.232721+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:27.232629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:27.232629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:49.490654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1036,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065365+0000","last_change":"2024-03-24T12:16:06.221399+0000","last_active":"2024-03-24T12:16:22.065365+0000","last_peered":"2024-03-24T12:16:22.065365+0000","last_clean":"2024-03-24T12:16:22.065365+0000","last_became_active":"2024-03-24T12:08:36.153377+0000","last_became_peered":"2024-03-24T12:08:36.153377+0000","last_unstale":"2024-03-24T12:16:22.065365+0000","last_undegraded":"2024-03-24T12:16:22.065365+0000","last_fullsized":"2024-03-24T12:16:22.065365+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:06.221287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:06.221287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:07.930681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"507'10","reported_seq":1356,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:41.128312+0000","last_change":"2024-03-24T12:16:41.128362+0000","last_active":"2024-03-24T12:16:41.128312+0000","last_peered":"2024-03-24T12:16:41.128312+0000","last_clean":"2024-03-24T12:16:41.128312+0000","last_became_active":"2024-03-24T12:15:05.230777+0000","last_became_peered":"2024-03-24T12:15:05.230777+0000","last_unstale":"2024-03-24T12:16:41.128312+0000","last_undegraded":"2024-03-24T12:16:41.128312+0000","last_fullsized":"2024-03-24T12:16:41.128312+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'10","last_scrub_stamp":"2024-03-24T12:16:41.128222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:41.128222+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:54.821274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"450'49","reported_seq":1111,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355052+0000","last_change":"2024-03-24T12:15:48.238042+0000","last_active":"2024-03-24T12:16:22.355052+0000","last_peered":"2024-03-24T12:16:22.355052+0000","last_clean":"2024-03-24T12:16:22.355052+0000","last_became_active":"2024-03-24T12:08:36.162416+0000","last_became_peered":"2024-03-24T12:08:36.162416+0000","last_unstale":"2024-03-24T12:16:22.355052+0000","last_undegraded":"2024-03-24T12:16:22.355052+0000","last_fullsized":"2024-03-24T12:16:22.355052+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'49","last_scrub_stamp":"2024-03-24T12:15:48.237913+0000","last_deep_scrub":"88'21","last_deep_scrub_stamp":"2024-03-24T11:53:26.665554+0000","last_clean_scrub_stamp":"2024-03-24T12:15:48.237913+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:48.663781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":942,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:25.405138+0000","last_change":"2024-03-24T12:16:25.405161+0000","last_active":"2024-03-24T12:16:25.405138+0000","last_peered":"2024-03-24T12:16:25.405138+0000","last_clean":"2024-03-24T12:16:25.405138+0000","last_became_active":"2024-03-24T12:15:43.069574+0000","last_became_peered":"2024-03-24T12:15:43.069574+0000","last_unstale":"2024-03-24T12:16:25.405138+0000","last_undegraded":"2024-03-24T12:16:25.405138+0000","last_fullsized":"2024-03-24T12:16:25.405138+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:25.405071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:15.292233+0000","last_clean_scrub_stamp":"2024-03-24T12:16:25.405071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:48.134013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1083,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358259+0000","last_change":"2024-03-24T12:15:35.688152+0000","last_active":"2024-03-24T12:16:22.358259+0000","last_peered":"2024-03-24T12:16:22.358259+0000","last_clean":"2024-03-24T12:16:22.358259+0000","last_became_active":"2024-03-24T12:15:15.275798+0000","last_became_peered":"2024-03-24T12:15:15.275798+0000","last_unstale":"2024-03-24T12:16:22.358259+0000","last_undegraded":"2024-03-24T12:16:22.358259+0000","last_fullsized":"2024-03-24T12:16:22.358259+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:35.688049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:35.688049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:53.039052+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'130","reported_seq":1117,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065650+0000","last_change":"2024-03-24T12:15:29.122877+0000","last_active":"2024-03-24T12:16:22.065650+0000","last_peered":"2024-03-24T12:16:22.065650+0000","last_clean":"2024-03-24T12:16:22.065650+0000","last_became_active":"2024-03-24T12:10:47.971906+0000","last_became_peered":"2024-03-24T12:10:47.971906+0000","last_unstale":"2024-03-24T12:16:22.065650+0000","last_undegraded":"2024-03-24T12:16:22.065650+0000","last_fullsized":"2024-03-24T12:16:22.065650+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:29.122771+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:29.122771+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.979863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:31.484385+0000","last_change":"2024-03-24T12:16:31.484424+0000","last_active":"2024-03-24T12:16:31.484385+0000","last_peered":"2024-03-24T12:16:31.484385+0000","last_clean":"2024-03-24T12:16:31.484385+0000","last_became_active":"2024-03-24T11:52:19.389349+0000","last_became_peered":"2024-03-24T11:52:19.389349+0000","last_unstale":"2024-03-24T12:16:31.484385+0000","last_undegraded":"2024-03-24T12:16:31.484385+0000","last_fullsized":"2024-03-24T12:16:31.484385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:31.484315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:03.195101+0000","last_clean_scrub_stamp":"2024-03-24T12:16:31.484315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:40.086141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"450'50","reported_seq":1113,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067655+0000","last_change":"2024-03-24T12:16:04.225054+0000","last_active":"2024-03-24T12:16:22.067655+0000","last_peered":"2024-03-24T12:16:22.067655+0000","last_clean":"2024-03-24T12:16:22.067655+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:16:22.067655+0000","last_undegraded":"2024-03-24T12:16:22.067655+0000","last_fullsized":"2024-03-24T12:16:22.067655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'50","last_scrub_stamp":"2024-03-24T12:16:04.224949+0000","last_deep_scrub":"241'38","last_deep_scrub_stamp":"2024-03-24T12:10:29.733073+0000","last_clean_scrub_stamp":"2024-03-24T12:16:04.224949+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:15.077250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":198,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:36.271134+0000","last_change":"2024-03-24T12:16:36.271165+0000","last_active":"2024-03-24T12:16:36.271134+0000","last_peered":"2024-03-24T12:16:36.271134+0000","last_clean":"2024-03-24T12:16:36.271134+0000","last_became_active":"2024-03-24T12:15:11.248843+0000","last_became_peered":"2024-03-24T12:15:11.248843+0000","last_unstale":"2024-03-24T12:16:36.271134+0000","last_undegraded":"2024-03-24T12:16:36.271134+0000","last_fullsized":"2024-03-24T12:16:36.271134+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:16:36.271056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:36.271056+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:42.478507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1052,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355489+0000","last_change":"2024-03-24T12:15:51.288985+0000","last_active":"2024-03-24T12:16:22.355489+0000","last_peered":"2024-03-24T12:16:22.355489+0000","last_clean":"2024-03-24T12:16:22.355489+0000","last_became_active":"2024-03-24T12:15:15.281413+0000","last_became_peered":"2024-03-24T12:15:15.281413+0000","last_unstale":"2024-03-24T12:16:22.355489+0000","last_undegraded":"2024-03-24T12:16:22.355489+0000","last_fullsized":"2024-03-24T12:16:22.355489+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:51.288883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:51.288883+0000","last_clean_scrub_stamp":"2024-03-24T12:15:51.288883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:51.967766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'130","reported_seq":1137,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.354766+0000","last_change":"2024-03-24T12:15:58.251741+0000","last_active":"2024-03-24T12:16:22.354766+0000","last_peered":"2024-03-24T12:16:22.354766+0000","last_clean":"2024-03-24T12:16:22.354766+0000","last_became_active":"2024-03-24T12:15:18.067386+0000","last_became_peered":"2024-03-24T12:15:18.067386+0000","last_unstale":"2024-03-24T12:16:22.354766+0000","last_undegraded":"2024-03-24T12:16:22.354766+0000","last_fullsized":"2024-03-24T12:16:22.354766+0000","mapping_epoch":513,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:58.251637+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:58.251637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:26.833069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":936,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:38.454357+0000","last_change":"2024-03-24T12:16:38.454399+0000","last_active":"2024-03-24T12:16:38.454357+0000","last_peered":"2024-03-24T12:16:38.454357+0000","last_clean":"2024-03-24T12:16:38.454357+0000","last_became_active":"2024-03-24T12:15:43.072285+0000","last_became_peered":"2024-03-24T12:15:43.072285+0000","last_unstale":"2024-03-24T12:16:38.454357+0000","last_undegraded":"2024-03-24T12:16:38.454357+0000","last_fullsized":"2024-03-24T12:16:38.454357+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:38.454250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:16:38.454250+0000","last_clean_scrub_stamp":"2024-03-24T12:16:38.454250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:54.348665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"450'47","reported_seq":1117,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:38.746071+0000","last_change":"2024-03-24T12:16:38.746150+0000","last_active":"2024-03-24T12:16:38.746071+0000","last_peered":"2024-03-24T12:16:38.746071+0000","last_clean":"2024-03-24T12:16:38.746071+0000","last_became_active":"2024-03-24T12:08:36.155224+0000","last_became_peered":"2024-03-24T12:08:36.155224+0000","last_unstale":"2024-03-24T12:16:38.746071+0000","last_undegraded":"2024-03-24T12:16:38.746071+0000","last_fullsized":"2024-03-24T12:16:38.746071+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'47","last_scrub_stamp":"2024-03-24T12:16:38.745964+0000","last_deep_scrub":"369'37","last_deep_scrub_stamp":"2024-03-24T12:11:37.405400+0000","last_clean_scrub_stamp":"2024-03-24T12:16:38.745964+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:44.424883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1071,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358631+0000","last_change":"2024-03-24T12:15:15.679971+0000","last_active":"2024-03-24T12:16:22.358631+0000","last_peered":"2024-03-24T12:16:22.358631+0000","last_clean":"2024-03-24T12:16:22.358631+0000","last_became_active":"2024-03-24T11:52:15.146096+0000","last_became_peered":"2024-03-24T11:52:15.146096+0000","last_unstale":"2024-03-24T12:16:22.358631+0000","last_undegraded":"2024-03-24T12:16:22.358631+0000","last_fullsized":"2024-03-24T12:16:22.358631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:15.679884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:15.679884+0000","last_clean_scrub_stamp":"2024-03-24T12:15:15.679884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:38.992156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":468,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194281+0000","last_change":"2024-03-24T12:16:08.305203+0000","last_active":"2024-03-24T12:16:22.194281+0000","last_peered":"2024-03-24T12:16:22.194281+0000","last_clean":"2024-03-24T12:16:22.194281+0000","last_became_active":"2024-03-24T12:15:15.274745+0000","last_became_peered":"2024-03-24T12:15:15.274745+0000","last_unstale":"2024-03-24T12:16:22.194281+0000","last_undegraded":"2024-03-24T12:16:22.194281+0000","last_fullsized":"2024-03-24T12:16:22.194281+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:08.305080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:20.921985+0000","last_clean_scrub_stamp":"2024-03-24T12:16:08.305080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:26.515483+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'130","reported_seq":1018,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.193100+0000","last_change":"2024-03-24T12:15:43.557623+0000","last_active":"2024-03-24T12:16:22.193100+0000","last_peered":"2024-03-24T12:16:22.193100+0000","last_clean":"2024-03-24T12:16:22.193100+0000","last_became_active":"2024-03-24T12:15:43.067828+0000","last_became_peered":"2024-03-24T12:15:43.067828+0000","last_unstale":"2024-03-24T12:16:22.193100+0000","last_undegraded":"2024-03-24T12:16:22.193100+0000","last_fullsized":"2024-03-24T12:16:22.193100+0000","mapping_epoch":523,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:43.557516+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:11:45.081517+0000","last_clean_scrub_stamp":"2024-03-24T12:15:43.557516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:05.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":933,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.207271+0000","last_change":"2024-03-24T12:16:10.307245+0000","last_active":"2024-03-24T12:16:22.207271+0000","last_peered":"2024-03-24T12:16:22.207271+0000","last_clean":"2024-03-24T12:16:22.207271+0000","last_became_active":"2024-03-24T12:15:05.214114+0000","last_became_peered":"2024-03-24T12:15:05.214114+0000","last_unstale":"2024-03-24T12:16:22.207271+0000","last_undegraded":"2024-03-24T12:16:22.207271+0000","last_fullsized":"2024-03-24T12:16:22.207271+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:10.307140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:12:21.577764+0000","last_clean_scrub_stamp":"2024-03-24T12:16:10.307140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:34.738268+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"450'42","reported_seq":1142,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067183+0000","last_change":"2024-03-24T12:15:53.551637+0000","last_active":"2024-03-24T12:16:22.067183+0000","last_peered":"2024-03-24T12:16:22.067183+0000","last_clean":"2024-03-24T12:16:22.067183+0000","last_became_active":"2024-03-24T12:15:15.275691+0000","last_became_peered":"2024-03-24T12:15:15.275691+0000","last_unstale":"2024-03-24T12:16:22.067183+0000","last_undegraded":"2024-03-24T12:16:22.067183+0000","last_fullsized":"2024-03-24T12:16:22.067183+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'42","last_scrub_stamp":"2024-03-24T12:15:53.551535+0000","last_deep_scrub":"240'18","last_deep_scrub_stamp":"2024-03-24T12:02:15.738700+0000","last_clean_scrub_stamp":"2024-03-24T12:15:53.551535+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:11.978629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":890,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:40.693581+0000","last_change":"2024-03-24T12:16:40.693601+0000","last_active":"2024-03-24T12:16:40.693581+0000","last_peered":"2024-03-24T12:16:40.693581+0000","last_clean":"2024-03-24T12:16:40.693581+0000","last_became_active":"2024-03-24T11:56:53.906429+0000","last_became_peered":"2024-03-24T11:56:53.906429+0000","last_unstale":"2024-03-24T12:16:40.693581+0000","last_undegraded":"2024-03-24T12:16:40.693581+0000","last_fullsized":"2024-03-24T12:16:40.693581+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:40.693521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:27.560693+0000","last_clean_scrub_stamp":"2024-03-24T12:16:40.693521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:05.373663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1064,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:23.720152+0000","last_change":"2024-03-24T12:16:23.720190+0000","last_active":"2024-03-24T12:16:23.720152+0000","last_peered":"2024-03-24T12:16:23.720152+0000","last_clean":"2024-03-24T12:16:23.720152+0000","last_became_active":"2024-03-24T11:52:15.136614+0000","last_became_peered":"2024-03-24T11:52:15.136614+0000","last_unstale":"2024-03-24T12:16:23.720152+0000","last_undegraded":"2024-03-24T12:16:23.720152+0000","last_fullsized":"2024-03-24T12:16:23.720152+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:23.720063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:27.650987+0000","last_clean_scrub_stamp":"2024-03-24T12:16:23.720063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:52.305568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050646100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194707+0000","last_change":"2024-03-24T12:15:31.554219+0000","last_active":"2024-03-24T12:16:22.194707+0000","last_peered":"2024-03-24T12:16:22.194707+0000","last_clean":"2024-03-24T12:16:22.194707+0000","last_became_active":"2024-03-24T12:15:05.223578+0000","last_became_peered":"2024-03-24T12:15:05.223578+0000","last_unstale":"2024-03-24T12:16:22.194707+0000","last_undegraded":"2024-03-24T12:16:22.194707+0000","last_fullsized":"2024-03-24T12:16:22.194707+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:31.554139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:11:39.903133+0000","last_clean_scrub_stamp":"2024-03-24T12:15:31.554139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.736890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'130","reported_seq":381,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194549+0000","last_change":"2024-03-24T12:15:11.420213+0000","last_active":"2024-03-24T12:16:22.194549+0000","last_peered":"2024-03-24T12:16:22.194549+0000","last_clean":"2024-03-24T12:16:22.194549+0000","last_became_active":"2024-03-24T12:15:11.248603+0000","last_became_peered":"2024-03-24T12:15:11.248603+0000","last_unstale":"2024-03-24T12:16:22.194549+0000","last_undegraded":"2024-03-24T12:16:22.194549+0000","last_fullsized":"2024-03-24T12:16:22.194549+0000","mapping_epoch":508,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:11.420131+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:11.420131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:35.607384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1064,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:32.761069+0000","last_change":"2024-03-24T12:16:32.761090+0000","last_active":"2024-03-24T12:16:32.761069+0000","last_peered":"2024-03-24T12:16:32.761069+0000","last_clean":"2024-03-24T12:16:32.761069+0000","last_became_active":"2024-03-24T12:08:36.162537+0000","last_became_peered":"2024-03-24T12:08:36.162537+0000","last_unstale":"2024-03-24T12:16:32.761069+0000","last_undegraded":"2024-03-24T12:16:32.761069+0000","last_fullsized":"2024-03-24T12:16:32.761069+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:32.761008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:13.979581+0000","last_clean_scrub_stamp":"2024-03-24T12:16:32.761008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:47.256331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"525'42","reported_seq":1271,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487538+0000","last_change":"2024-03-24T12:16:04.507179+0000","last_active":"2024-03-24T12:16:35.487538+0000","last_peered":"2024-03-24T12:16:35.487538+0000","last_clean":"2024-03-24T12:16:35.487538+0000","last_became_active":"2024-03-24T12:15:43.072013+0000","last_became_peered":"2024-03-24T12:15:43.072013+0000","last_unstale":"2024-03-24T12:16:35.487538+0000","last_undegraded":"2024-03-24T12:16:35.487538+0000","last_fullsized":"2024-03-24T12:16:35.487538+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'42","last_scrub_stamp":"2024-03-24T12:16:04.507046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:04.507046+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:14.416757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'130","reported_seq":1377,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.357652+0000","last_change":"2024-03-24T12:15:28.679148+0000","last_active":"2024-03-24T12:16:22.357652+0000","last_peered":"2024-03-24T12:16:22.357652+0000","last_clean":"2024-03-24T12:16:22.357652+0000","last_became_active":"2024-03-24T12:10:33.461798+0000","last_became_peered":"2024-03-24T12:10:33.461798+0000","last_unstale":"2024-03-24T12:16:22.357652+0000","last_undegraded":"2024-03-24T12:16:22.357652+0000","last_fullsized":"2024-03-24T12:16:22.357652+0000","mapping_epoch":64,"log_start":"83'130","ondisk_log_start":"83'130","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:28.679021+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:10:25.984109+0000","last_clean_scrub_stamp":"2024-03-24T12:15:28.679021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:51.449406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":961,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:24.221701+0000","last_change":"2024-03-24T12:16:24.221726+0000","last_active":"2024-03-24T12:16:24.221701+0000","last_peered":"2024-03-24T12:16:24.221701+0000","last_clean":"2024-03-24T12:16:24.221701+0000","last_became_active":"2024-03-24T12:08:36.167963+0000","last_became_peered":"2024-03-24T12:08:36.167963+0000","last_unstale":"2024-03-24T12:16:24.221701+0000","last_undegraded":"2024-03-24T12:16:24.221701+0000","last_fullsized":"2024-03-24T12:16:24.221701+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:24.221629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:24.221629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:31.331207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":1050,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065367+0000","last_change":"2024-03-24T12:16:01.281552+0000","last_active":"2024-03-24T12:16:22.065367+0000","last_peered":"2024-03-24T12:16:22.065367+0000","last_clean":"2024-03-24T12:16:22.065367+0000","last_became_active":"2024-03-24T12:08:36.155393+0000","last_became_peered":"2024-03-24T12:08:36.155393+0000","last_unstale":"2024-03-24T12:16:22.065367+0000","last_undegraded":"2024-03-24T12:16:22.065367+0000","last_fullsized":"2024-03-24T12:16:22.065367+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:16:01.281451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:01.281451+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:04.311553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086732599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":916,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.207225+0000","last_change":"2024-03-24T12:16:14.296935+0000","last_active":"2024-03-24T12:16:22.207225+0000","last_peered":"2024-03-24T12:16:22.207225+0000","last_clean":"2024-03-24T12:16:22.207225+0000","last_became_active":"2024-03-24T12:08:36.161087+0000","last_became_peered":"2024-03-24T12:08:36.161087+0000","last_unstale":"2024-03-24T12:16:22.207225+0000","last_undegraded":"2024-03-24T12:16:22.207225+0000","last_fullsized":"2024-03-24T12:16:22.207225+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:14.296822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:41.822890+0000","last_clean_scrub_stamp":"2024-03-24T12:16:14.296822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:21.266968+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:36.777707+0000","last_change":"2024-03-24T12:16:36.777734+0000","last_active":"2024-03-24T12:16:36.777707+0000","last_peered":"2024-03-24T12:16:36.777707+0000","last_clean":"2024-03-24T12:16:36.777707+0000","last_became_active":"2024-03-24T12:15:15.277454+0000","last_became_peered":"2024-03-24T12:15:15.277454+0000","last_unstale":"2024-03-24T12:16:36.777707+0000","last_undegraded":"2024-03-24T12:16:36.777707+0000","last_fullsized":"2024-03-24T12:16:36.777707+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:36.777636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:19.646040+0000","last_clean_scrub_stamp":"2024-03-24T12:16:36.777636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:40.070534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":102,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194896+0000","last_change":"2024-03-24T12:15:48.366270+0000","last_active":"2024-03-24T12:16:22.194896+0000","last_peered":"2024-03-24T12:16:22.194896+0000","last_clean":"2024-03-24T12:16:22.194896+0000","last_became_active":"2024-03-24T12:15:05.216691+0000","last_became_peered":"2024-03-24T12:15:05.216691+0000","last_unstale":"2024-03-24T12:16:22.194896+0000","last_undegraded":"2024-03-24T12:16:22.194896+0000","last_fullsized":"2024-03-24T12:16:22.194896+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'26","last_scrub_stamp":"2024-03-24T12:15:48.366102+0000","last_deep_scrub":"526'26","last_deep_scrub_stamp":"2024-03-24T12:15:48.366102+0000","last_clean_scrub_stamp":"2024-03-24T12:15:48.366102+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.766909+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:39.717481+0000","last_change":"2024-03-24T12:16:39.717513+0000","last_active":"2024-03-24T12:16:39.717481+0000","last_peered":"2024-03-24T12:16:39.717481+0000","last_clean":"2024-03-24T12:16:39.717481+0000","last_became_active":"2024-03-24T12:15:15.280046+0000","last_became_peered":"2024-03-24T12:15:15.280046+0000","last_unstale":"2024-03-24T12:16:39.717481+0000","last_undegraded":"2024-03-24T12:16:39.717481+0000","last_fullsized":"2024-03-24T12:16:39.717481+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:39.717394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:39.717394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:55.369319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"462'6","reported_seq":1237,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:41.412226+0000","last_change":"2024-03-24T12:15:23.605179+0000","last_active":"2024-03-24T12:16:41.412226+0000","last_peered":"2024-03-24T12:16:41.412226+0000","last_clean":"2024-03-24T12:16:41.412226+0000","last_became_active":"2024-03-24T12:15:15.274515+0000","last_became_peered":"2024-03-24T12:15:15.274515+0000","last_unstale":"2024-03-24T12:16:41.412226+0000","last_undegraded":"2024-03-24T12:16:41.412226+0000","last_fullsized":"2024-03-24T12:16:41.412226+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'6","last_scrub_stamp":"2024-03-24T12:15:23.605093+0000","last_deep_scrub":"462'6","last_deep_scrub_stamp":"2024-03-24T12:13:14.961156+0000","last_clean_scrub_stamp":"2024-03-24T12:15:23.605093+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001104856,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":640,"num_read_kb":635,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1051,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358626+0000","last_change":"2024-03-24T12:15:33.694555+0000","last_active":"2024-03-24T12:16:22.358626+0000","last_peered":"2024-03-24T12:16:22.358626+0000","last_clean":"2024-03-24T12:16:22.358626+0000","last_became_active":"2024-03-24T11:52:21.850912+0000","last_became_peered":"2024-03-24T11:52:21.850912+0000","last_unstale":"2024-03-24T12:16:22.358626+0000","last_undegraded":"2024-03-24T12:16:22.358626+0000","last_fullsized":"2024-03-24T12:16:22.358626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:33.694432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:33.694432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:42.255085+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'130","reported_seq":1376,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358083+0000","last_change":"2024-03-24T12:15:26.615976+0000","last_active":"2024-03-24T12:16:22.358083+0000","last_peered":"2024-03-24T12:16:22.358083+0000","last_clean":"2024-03-24T12:16:22.358083+0000","last_became_active":"2024-03-24T12:10:23.398177+0000","last_became_peered":"2024-03-24T12:10:23.398177+0000","last_unstale":"2024-03-24T12:16:22.358083+0000","last_undegraded":"2024-03-24T12:16:22.358083+0000","last_fullsized":"2024-03-24T12:16:22.358083+0000","mapping_epoch":189,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:26.615877+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:26.615877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:46.685876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1050,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:39.731399+0000","last_change":"2024-03-24T12:16:39.731420+0000","last_active":"2024-03-24T12:16:39.731399+0000","last_peered":"2024-03-24T12:16:39.731399+0000","last_clean":"2024-03-24T12:16:39.731399+0000","last_became_active":"2024-03-24T12:15:05.219911+0000","last_became_peered":"2024-03-24T12:15:05.219911+0000","last_unstale":"2024-03-24T12:16:39.731399+0000","last_undegraded":"2024-03-24T12:16:39.731399+0000","last_fullsized":"2024-03-24T12:16:39.731399+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:39.731342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:58.906274+0000","last_clean_scrub_stamp":"2024-03-24T12:16:39.731342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:09.646451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"511'60","reported_seq":1198,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487273+0000","last_change":"2024-03-24T12:15:59.248196+0000","last_active":"2024-03-24T12:16:35.487273+0000","last_peered":"2024-03-24T12:16:35.487273+0000","last_clean":"2024-03-24T12:16:35.487273+0000","last_became_active":"2024-03-24T12:15:15.276052+0000","last_became_peered":"2024-03-24T12:15:15.276052+0000","last_unstale":"2024-03-24T12:16:35.487273+0000","last_undegraded":"2024-03-24T12:16:35.487273+0000","last_fullsized":"2024-03-24T12:16:35.487273+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'60","last_scrub_stamp":"2024-03-24T12:15:59.248094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:59.248094+0000","objects_scrubbed":10,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:09.068128+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":925,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206392+0000","last_change":"2024-03-24T12:16:06.241708+0000","last_active":"2024-03-24T12:16:22.206392+0000","last_peered":"2024-03-24T12:16:22.206392+0000","last_clean":"2024-03-24T12:16:22.206392+0000","last_became_active":"2024-03-24T12:08:36.167478+0000","last_became_peered":"2024-03-24T12:08:36.167478+0000","last_unstale":"2024-03-24T12:16:22.206392+0000","last_undegraded":"2024-03-24T12:16:22.206392+0000","last_fullsized":"2024-03-24T12:16:22.206392+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:06.241596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:16:06.241596+0000","last_clean_scrub_stamp":"2024-03-24T12:16:06.241596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:34.123576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:29.089517+0000","last_change":"2024-03-24T12:16:29.089547+0000","last_active":"2024-03-24T12:16:29.089517+0000","last_peered":"2024-03-24T12:16:29.089517+0000","last_clean":"2024-03-24T12:16:29.089517+0000","last_became_active":"2024-03-24T11:52:21.386142+0000","last_became_peered":"2024-03-24T11:52:21.386142+0000","last_unstale":"2024-03-24T12:16:29.089517+0000","last_undegraded":"2024-03-24T12:16:29.089517+0000","last_fullsized":"2024-03-24T12:16:29.089517+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:29.089429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:29.089429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:43.515255+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'130","reported_seq":1001,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206381+0000","last_change":"2024-03-24T12:15:24.086717+0000","last_active":"2024-03-24T12:16:22.206381+0000","last_peered":"2024-03-24T12:16:22.206381+0000","last_clean":"2024-03-24T12:16:22.206381+0000","last_became_active":"2024-03-24T12:10:26.959434+0000","last_became_peered":"2024-03-24T12:10:26.959434+0000","last_unstale":"2024-03-24T12:16:22.206381+0000","last_undegraded":"2024-03-24T12:16:22.206381+0000","last_fullsized":"2024-03-24T12:16:22.206381+0000","mapping_epoch":353,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:24.086638+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T11:58:09.128250+0000","last_clean_scrub_stamp":"2024-03-24T12:15:24.086638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:45.070870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"354'10","reported_seq":1351,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487537+0000","last_change":"2024-03-24T12:15:59.500305+0000","last_active":"2024-03-24T12:16:35.487537+0000","last_peered":"2024-03-24T12:16:35.487537+0000","last_clean":"2024-03-24T12:16:35.487537+0000","last_became_active":"2024-03-24T12:08:36.163613+0000","last_became_peered":"2024-03-24T12:08:36.163613+0000","last_unstale":"2024-03-24T12:16:35.487537+0000","last_undegraded":"2024-03-24T12:16:35.487537+0000","last_fullsized":"2024-03-24T12:16:35.487537+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:15:59.500213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:59.500213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:08.741574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"450'35","reported_seq":97,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194570+0000","last_change":"2024-03-24T12:16:00.237534+0000","last_active":"2024-03-24T12:16:22.194570+0000","last_peered":"2024-03-24T12:16:22.194570+0000","last_clean":"2024-03-24T12:16:22.194570+0000","last_became_active":"2024-03-24T12:15:05.225343+0000","last_became_peered":"2024-03-24T12:15:05.225343+0000","last_unstale":"2024-03-24T12:16:22.194570+0000","last_undegraded":"2024-03-24T12:16:22.194570+0000","last_fullsized":"2024-03-24T12:16:22.194570+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:16:00.237427+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T11:57:05.447425+0000","last_clean_scrub_stamp":"2024-03-24T12:16:00.237427+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:10.406479+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":807,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:26.240498+0000","last_change":"2024-03-24T12:16:26.240523+0000","last_active":"2024-03-24T12:16:26.240498+0000","last_peered":"2024-03-24T12:16:26.240498+0000","last_clean":"2024-03-24T12:16:26.240498+0000","last_became_active":"2024-03-24T12:15:11.657425+0000","last_became_peered":"2024-03-24T12:15:11.657425+0000","last_unstale":"2024-03-24T12:16:26.240498+0000","last_undegraded":"2024-03-24T12:16:26.240498+0000","last_fullsized":"2024-03-24T12:16:26.240498+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:26.240434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:17.492961+0000","last_clean_scrub_stamp":"2024-03-24T12:16:26.240434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:32.095116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1048,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:26.746576+0000","last_change":"2024-03-24T12:16:26.746576+0000","last_active":"2024-03-24T12:16:26.746576+0000","last_peered":"2024-03-24T12:16:26.746576+0000","last_clean":"2024-03-24T12:16:26.746576+0000","last_became_active":"2024-03-24T12:15:15.273632+0000","last_became_peered":"2024-03-24T12:15:15.273632+0000","last_unstale":"2024-03-24T12:16:26.746576+0000","last_undegraded":"2024-03-24T12:16:26.746576+0000","last_fullsized":"2024-03-24T12:16:26.746576+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:26.746477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:18.621992+0000","last_clean_scrub_stamp":"2024-03-24T12:16:26.746477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:30.059907+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'130","reported_seq":1010,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192960+0000","last_change":"2024-03-24T12:15:50.523000+0000","last_active":"2024-03-24T12:16:22.192960+0000","last_peered":"2024-03-24T12:16:22.192960+0000","last_clean":"2024-03-24T12:16:22.192960+0000","last_became_active":"2024-03-24T12:15:43.069034+0000","last_became_peered":"2024-03-24T12:15:43.069034+0000","last_unstale":"2024-03-24T12:16:22.192960+0000","last_undegraded":"2024-03-24T12:16:22.192960+0000","last_fullsized":"2024-03-24T12:16:22.192960+0000","mapping_epoch":523,"log_start":"83'130","ondisk_log_start":"83'130","created":184,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'130","last_scrub_stamp":"2024-03-24T12:15:50.522836+0000","last_deep_scrub":"83'130","last_deep_scrub_stamp":"2024-03-24T12:05:23.690315+0000","last_clean_scrub_stamp":"2024-03-24T12:15:50.522836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:50.657476+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1070,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066078+0000","last_change":"2024-03-24T12:15:36.400232+0000","last_active":"2024-03-24T12:16:22.066078+0000","last_peered":"2024-03-24T12:16:22.066078+0000","last_clean":"2024-03-24T12:16:22.066078+0000","last_became_active":"2024-03-24T11:52:19.377911+0000","last_became_peered":"2024-03-24T11:52:19.377911+0000","last_unstale":"2024-03-24T12:16:22.066078+0000","last_undegraded":"2024-03-24T12:16:22.066078+0000","last_fullsized":"2024-03-24T12:16:22.066078+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:36.400126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:36.400126+0000","last_clean_scrub_stamp":"2024-03-24T12:15:36.400126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:43.605921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"450'24","reported_seq":1135,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355054+0000","last_change":"2024-03-24T12:15:27.201919+0000","last_active":"2024-03-24T12:16:22.355054+0000","last_peered":"2024-03-24T12:16:22.355054+0000","last_clean":"2024-03-24T12:16:22.355054+0000","last_became_active":"2024-03-24T12:15:05.233554+0000","last_became_peered":"2024-03-24T12:15:05.233554+0000","last_unstale":"2024-03-24T12:16:22.355054+0000","last_undegraded":"2024-03-24T12:16:22.355054+0000","last_fullsized":"2024-03-24T12:16:22.355054+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'24","last_scrub_stamp":"2024-03-24T12:15:27.201820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:27.201820+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.494834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":1055,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:32.507867+0000","last_change":"2024-03-24T12:16:32.507919+0000","last_active":"2024-03-24T12:16:32.507867+0000","last_peered":"2024-03-24T12:16:32.507867+0000","last_clean":"2024-03-24T12:16:32.507867+0000","last_became_active":"2024-03-24T11:52:15.135809+0000","last_became_peered":"2024-03-24T11:52:15.135809+0000","last_unstale":"2024-03-24T12:16:32.507867+0000","last_undegraded":"2024-03-24T12:16:32.507867+0000","last_fullsized":"2024-03-24T12:16:32.507867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:16:32.507776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:32.507776+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:41.535470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1039,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:34.301514+0000","last_change":"2024-03-24T12:16:34.301539+0000","last_active":"2024-03-24T12:16:34.301514+0000","last_peered":"2024-03-24T12:16:34.301514+0000","last_clean":"2024-03-24T12:16:34.301514+0000","last_became_active":"2024-03-24T12:15:05.230832+0000","last_became_peered":"2024-03-24T12:15:05.230832+0000","last_unstale":"2024-03-24T12:16:34.301514+0000","last_undegraded":"2024-03-24T12:16:34.301514+0000","last_fullsized":"2024-03-24T12:16:34.301514+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:34.301430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:23.435452+0000","last_clean_scrub_stamp":"2024-03-24T12:16:34.301430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:58.361767+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"450'25","reported_seq":1087,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067328+0000","last_change":"2024-03-24T12:16:12.615425+0000","last_active":"2024-03-24T12:16:22.067328+0000","last_peered":"2024-03-24T12:16:22.067328+0000","last_clean":"2024-03-24T12:16:22.067328+0000","last_became_active":"2024-03-24T11:52:17.425344+0000","last_became_peered":"2024-03-24T11:52:17.425344+0000","last_unstale":"2024-03-24T12:16:22.067328+0000","last_undegraded":"2024-03-24T12:16:22.067328+0000","last_fullsized":"2024-03-24T12:16:22.067328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'25","last_scrub_stamp":"2024-03-24T12:16:12.615322+0000","last_deep_scrub":"115'13","last_deep_scrub_stamp":"2024-03-24T11:55:16.093668+0000","last_clean_scrub_stamp":"2024-03-24T12:16:12.615322+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:40.586281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067763+0000","last_change":"2024-03-24T12:16:05.526639+0000","last_active":"2024-03-24T12:16:22.067763+0000","last_peered":"2024-03-24T12:16:22.067763+0000","last_clean":"2024-03-24T12:16:22.067763+0000","last_became_active":"2024-03-24T12:15:15.279928+0000","last_became_peered":"2024-03-24T12:15:15.279928+0000","last_unstale":"2024-03-24T12:16:22.067763+0000","last_undegraded":"2024-03-24T12:16:22.067763+0000","last_fullsized":"2024-03-24T12:16:22.067763+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:05.526557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:05.526557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:28.814607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1035,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066085+0000","last_change":"2024-03-24T12:16:16.127155+0000","last_active":"2024-03-24T12:16:22.066085+0000","last_peered":"2024-03-24T12:16:22.066085+0000","last_clean":"2024-03-24T12:16:22.066085+0000","last_became_active":"2024-03-24T12:15:05.223618+0000","last_became_peered":"2024-03-24T12:15:05.223618+0000","last_unstale":"2024-03-24T12:16:22.066085+0000","last_undegraded":"2024-03-24T12:16:22.066085+0000","last_fullsized":"2024-03-24T12:16:22.066085+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:16.127051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:16.127051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:41.798869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":77,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.068344+0000","last_change":"2024-03-24T12:15:20.252884+0000","last_active":"2024-03-24T12:16:22.068344+0000","last_peered":"2024-03-24T12:16:22.068344+0000","last_clean":"2024-03-24T12:16:22.068344+0000","last_became_active":"2024-03-24T12:15:15.281396+0000","last_became_peered":"2024-03-24T12:15:15.281396+0000","last_unstale":"2024-03-24T12:16:22.068344+0000","last_undegraded":"2024-03-24T12:16:22.068344+0000","last_fullsized":"2024-03-24T12:16:22.068344+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:20.252770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:20.252770+0000","last_clean_scrub_stamp":"2024-03-24T12:15:20.252770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:38.675260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"450'18","reported_seq":1062,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065967+0000","last_change":"2024-03-24T12:16:13.166830+0000","last_active":"2024-03-24T12:16:22.065967+0000","last_peered":"2024-03-24T12:16:22.065967+0000","last_clean":"2024-03-24T12:16:22.065967+0000","last_became_active":"2024-03-24T11:52:17.425311+0000","last_became_peered":"2024-03-24T11:52:17.425311+0000","last_unstale":"2024-03-24T12:16:22.065967+0000","last_undegraded":"2024-03-24T12:16:22.065967+0000","last_fullsized":"2024-03-24T12:16:22.065967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'18","last_scrub_stamp":"2024-03-24T12:16:13.166726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:13.166726+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:21.260703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":942,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192434+0000","last_change":"2024-03-24T12:16:19.333970+0000","last_active":"2024-03-24T12:16:22.192434+0000","last_peered":"2024-03-24T12:16:22.192434+0000","last_clean":"2024-03-24T12:16:22.192434+0000","last_became_active":"2024-03-24T12:15:43.070487+0000","last_became_peered":"2024-03-24T12:15:43.070487+0000","last_unstale":"2024-03-24T12:16:22.192434+0000","last_undegraded":"2024-03-24T12:16:22.192434+0000","last_fullsized":"2024-03-24T12:16:22.192434+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:19.333868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:56:26.760297+0000","last_clean_scrub_stamp":"2024-03-24T12:16:19.333868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:29.144746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":885,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206987+0000","last_change":"2024-03-24T12:15:38.226629+0000","last_active":"2024-03-24T12:16:22.206987+0000","last_peered":"2024-03-24T12:16:22.206987+0000","last_clean":"2024-03-24T12:16:22.206987+0000","last_became_active":"2024-03-24T12:15:05.214305+0000","last_became_peered":"2024-03-24T12:15:05.214305+0000","last_unstale":"2024-03-24T12:16:22.206987+0000","last_undegraded":"2024-03-24T12:16:22.206987+0000","last_fullsized":"2024-03-24T12:16:22.206987+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:38.226550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:38.226550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:01.256309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1053,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358212+0000","last_change":"2024-03-24T12:16:11.482087+0000","last_active":"2024-03-24T12:16:22.358212+0000","last_peered":"2024-03-24T12:16:22.358212+0000","last_clean":"2024-03-24T12:16:22.358212+0000","last_became_active":"2024-03-24T11:52:21.851875+0000","last_became_peered":"2024-03-24T11:52:21.851875+0000","last_unstale":"2024-03-24T12:16:22.358212+0000","last_undegraded":"2024-03-24T12:16:22.358212+0000","last_fullsized":"2024-03-24T12:16:22.358212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:11.481976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:17.165061+0000","last_clean_scrub_stamp":"2024-03-24T12:16:11.481976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:14.539883+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"450'35","reported_seq":121,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206591+0000","last_change":"2024-03-24T12:16:02.246915+0000","last_active":"2024-03-24T12:16:22.206591+0000","last_peered":"2024-03-24T12:16:22.206591+0000","last_clean":"2024-03-24T12:16:22.206591+0000","last_became_active":"2024-03-24T12:15:05.214237+0000","last_became_peered":"2024-03-24T12:15:05.214237+0000","last_unstale":"2024-03-24T12:16:22.206591+0000","last_undegraded":"2024-03-24T12:16:22.206591+0000","last_fullsized":"2024-03-24T12:16:22.206591+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:16:02.246770+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-24T11:53:47.583025+0000","last_clean_scrub_stamp":"2024-03-24T12:16:02.246770+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:25.535207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":924,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206611+0000","last_change":"2024-03-24T12:15:44.175851+0000","last_active":"2024-03-24T12:16:22.206611+0000","last_peered":"2024-03-24T12:16:22.206611+0000","last_clean":"2024-03-24T12:16:22.206611+0000","last_became_active":"2024-03-24T12:08:36.171310+0000","last_became_peered":"2024-03-24T12:08:36.171310+0000","last_unstale":"2024-03-24T12:16:22.206611+0000","last_undegraded":"2024-03-24T12:16:22.206611+0000","last_fullsized":"2024-03-24T12:16:22.206611+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:44.175742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:14:26.371072+0000","last_clean_scrub_stamp":"2024-03-24T12:15:44.175742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:10.556371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"354'10","reported_seq":1234,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487511+0000","last_change":"2024-03-24T12:15:50.148137+0000","last_active":"2024-03-24T12:16:35.487511+0000","last_peered":"2024-03-24T12:16:35.487511+0000","last_clean":"2024-03-24T12:16:35.487511+0000","last_became_active":"2024-03-24T12:08:36.164544+0000","last_became_peered":"2024-03-24T12:08:36.164544+0000","last_unstale":"2024-03-24T12:16:35.487511+0000","last_undegraded":"2024-03-24T12:16:35.487511+0000","last_fullsized":"2024-03-24T12:16:35.487511+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:15:50.147987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:50.147987+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:11.374410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1033,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067129+0000","last_change":"2024-03-24T12:15:58.601408+0000","last_active":"2024-03-24T12:16:22.067129+0000","last_peered":"2024-03-24T12:16:22.067129+0000","last_clean":"2024-03-24T12:16:22.067129+0000","last_became_active":"2024-03-24T11:52:21.382325+0000","last_became_peered":"2024-03-24T11:52:21.382325+0000","last_unstale":"2024-03-24T12:16:22.067129+0000","last_undegraded":"2024-03-24T12:16:22.067129+0000","last_fullsized":"2024-03-24T12:16:22.067129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:58.601298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:58.601298+0000","last_clean_scrub_stamp":"2024-03-24T12:15:58.601298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:23.366331+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"450'21","reported_seq":958,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.207124+0000","last_change":"2024-03-24T12:16:09.306345+0000","last_active":"2024-03-24T12:16:22.207124+0000","last_peered":"2024-03-24T12:16:22.207124+0000","last_clean":"2024-03-24T12:16:22.207124+0000","last_became_active":"2024-03-24T12:08:36.153733+0000","last_became_peered":"2024-03-24T12:08:36.153733+0000","last_unstale":"2024-03-24T12:16:22.207124+0000","last_undegraded":"2024-03-24T12:16:22.207124+0000","last_fullsized":"2024-03-24T12:16:22.207124+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:16:09.306202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:09.306202+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:31.672976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":927,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:43.662057+0000","last_change":"2024-03-24T12:16:43.662079+0000","last_active":"2024-03-24T12:16:43.662057+0000","last_peered":"2024-03-24T12:16:43.662057+0000","last_clean":"2024-03-24T12:16:43.662057+0000","last_became_active":"2024-03-24T12:10:30.443191+0000","last_became_peered":"2024-03-24T12:10:30.443191+0000","last_unstale":"2024-03-24T12:16:43.662057+0000","last_undegraded":"2024-03-24T12:16:43.662057+0000","last_fullsized":"2024-03-24T12:16:43.662057+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:43.661991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:43.661991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:48.329966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"511'10","reported_seq":1353,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487313+0000","last_change":"2024-03-24T12:16:27.079415+0000","last_active":"2024-03-24T12:16:35.487313+0000","last_peered":"2024-03-24T12:16:35.487313+0000","last_clean":"2024-03-24T12:16:35.487313+0000","last_became_active":"2024-03-24T12:15:15.272446+0000","last_became_peered":"2024-03-24T12:15:15.272446+0000","last_unstale":"2024-03-24T12:16:35.487313+0000","last_undegraded":"2024-03-24T12:16:35.487313+0000","last_fullsized":"2024-03-24T12:16:35.487313+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'10","last_scrub_stamp":"2024-03-24T12:16:27.079307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:27.079307+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:42.644218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1032,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355903+0000","last_change":"2024-03-24T12:15:52.524638+0000","last_active":"2024-03-24T12:16:22.355903+0000","last_peered":"2024-03-24T12:16:22.355903+0000","last_clean":"2024-03-24T12:16:22.355903+0000","last_became_active":"2024-03-24T12:08:36.160208+0000","last_became_peered":"2024-03-24T12:08:36.160208+0000","last_unstale":"2024-03-24T12:16:22.355903+0000","last_undegraded":"2024-03-24T12:16:22.355903+0000","last_fullsized":"2024-03-24T12:16:22.355903+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:52.524535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:59:12.776686+0000","last_clean_scrub_stamp":"2024-03-24T12:15:52.524535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:07.244672+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"450'21","reported_seq":847,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194351+0000","last_change":"2024-03-24T12:15:37.553854+0000","last_active":"2024-03-24T12:16:22.194351+0000","last_peered":"2024-03-24T12:16:22.194351+0000","last_clean":"2024-03-24T12:16:22.194351+0000","last_became_active":"2024-03-24T12:15:11.247777+0000","last_became_peered":"2024-03-24T12:15:11.247777+0000","last_unstale":"2024-03-24T12:16:22.194351+0000","last_undegraded":"2024-03-24T12:16:22.194351+0000","last_fullsized":"2024-03-24T12:16:22.194351+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:15:37.553753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:37.553753+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:01.610655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":934,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192655+0000","last_change":"2024-03-24T12:15:43.073055+0000","last_active":"2024-03-24T12:16:22.192655+0000","last_peered":"2024-03-24T12:16:22.192655+0000","last_clean":"2024-03-24T12:16:22.192655+0000","last_became_active":"2024-03-24T12:15:43.071443+0000","last_became_peered":"2024-03-24T12:15:43.071443+0000","last_unstale":"2024-03-24T12:16:22.192655+0000","last_undegraded":"2024-03-24T12:16:22.192655+0000","last_fullsized":"2024-03-24T12:16:22.192655+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:30.092224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:26.299382+0000","last_clean_scrub_stamp":"2024-03-24T12:15:30.092224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:57.450217+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1037,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:33.095595+0000","last_change":"2024-03-24T12:16:33.095615+0000","last_active":"2024-03-24T12:16:33.095595+0000","last_peered":"2024-03-24T12:16:33.095595+0000","last_clean":"2024-03-24T12:16:33.095595+0000","last_became_active":"2024-03-24T12:05:56.886957+0000","last_became_peered":"2024-03-24T12:05:56.886957+0000","last_unstale":"2024-03-24T12:16:33.095595+0000","last_undegraded":"2024-03-24T12:16:33.095595+0000","last_fullsized":"2024-03-24T12:16:33.095595+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:33.095537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:16:33.095537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:37.325834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1054,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065290+0000","last_change":"2024-03-24T12:15:49.670943+0000","last_active":"2024-03-24T12:16:22.065290+0000","last_peered":"2024-03-24T12:16:22.065290+0000","last_clean":"2024-03-24T12:16:22.065290+0000","last_became_active":"2024-03-24T12:15:05.216451+0000","last_became_peered":"2024-03-24T12:15:05.216451+0000","last_unstale":"2024-03-24T12:16:22.065290+0000","last_undegraded":"2024-03-24T12:16:22.065290+0000","last_fullsized":"2024-03-24T12:16:22.065290+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:49.670785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:49.670785+0000","last_clean_scrub_stamp":"2024-03-24T12:15:49.670785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:15.348655+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"450'27","reported_seq":1110,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355277+0000","last_change":"2024-03-24T12:16:09.312934+0000","last_active":"2024-03-24T12:16:22.355277+0000","last_peered":"2024-03-24T12:16:22.355277+0000","last_clean":"2024-03-24T12:16:22.355277+0000","last_became_active":"2024-03-24T12:15:05.233832+0000","last_became_peered":"2024-03-24T12:15:05.233832+0000","last_unstale":"2024-03-24T12:16:22.355277+0000","last_undegraded":"2024-03-24T12:16:22.355277+0000","last_fullsized":"2024-03-24T12:16:22.355277+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'27","last_scrub_stamp":"2024-03-24T12:16:09.312823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:09.312823+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:36.282369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":85,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194842+0000","last_change":"2024-03-24T12:15:33.546429+0000","last_active":"2024-03-24T12:16:22.194842+0000","last_peered":"2024-03-24T12:16:22.194842+0000","last_clean":"2024-03-24T12:16:22.194842+0000","last_became_active":"2024-03-24T12:15:05.216042+0000","last_became_peered":"2024-03-24T12:15:05.216042+0000","last_unstale":"2024-03-24T12:16:22.194842+0000","last_undegraded":"2024-03-24T12:16:22.194842+0000","last_fullsized":"2024-03-24T12:16:22.194842+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:33.546352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:15:33.546352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:44.143860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":693,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:39.297296+0000","last_change":"2024-03-24T12:16:39.297338+0000","last_active":"2024-03-24T12:16:39.297296+0000","last_peered":"2024-03-24T12:16:39.297296+0000","last_clean":"2024-03-24T12:16:39.297296+0000","last_became_active":"2024-03-24T12:15:11.731801+0000","last_became_peered":"2024-03-24T12:15:11.731801+0000","last_unstale":"2024-03-24T12:16:39.297296+0000","last_undegraded":"2024-03-24T12:16:39.297296+0000","last_fullsized":"2024-03-24T12:16:39.297296+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:39.297217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:44.948146+0000","last_clean_scrub_stamp":"2024-03-24T12:16:39.297217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:06.748045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065971+0000","last_change":"2024-03-24T12:16:20.158020+0000","last_active":"2024-03-24T12:16:22.065971+0000","last_peered":"2024-03-24T12:16:22.065971+0000","last_clean":"2024-03-24T12:16:22.065971+0000","last_became_active":"2024-03-24T12:08:36.144875+0000","last_became_peered":"2024-03-24T12:08:36.144875+0000","last_unstale":"2024-03-24T12:16:22.065971+0000","last_undegraded":"2024-03-24T12:16:22.065971+0000","last_fullsized":"2024-03-24T12:16:22.065971+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:20.157920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:20.157920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:36.888199+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"450'14","reported_seq":1079,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067439+0000","last_change":"2024-03-24T12:15:39.224441+0000","last_active":"2024-03-24T12:16:22.067439+0000","last_peered":"2024-03-24T12:16:22.067439+0000","last_clean":"2024-03-24T12:16:22.067439+0000","last_became_active":"2024-03-24T12:15:05.220086+0000","last_became_peered":"2024-03-24T12:15:05.220086+0000","last_unstale":"2024-03-24T12:16:22.067439+0000","last_undegraded":"2024-03-24T12:16:22.067439+0000","last_fullsized":"2024-03-24T12:16:22.067439+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'14","last_scrub_stamp":"2024-03-24T12:15:39.224267+0000","last_deep_scrub":"180'6","last_deep_scrub_stamp":"2024-03-24T11:59:25.156553+0000","last_clean_scrub_stamp":"2024-03-24T12:15:39.224267+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:40.001209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":943,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206894+0000","last_change":"2024-03-24T12:15:31.196209+0000","last_active":"2024-03-24T12:16:22.206894+0000","last_peered":"2024-03-24T12:16:22.206894+0000","last_clean":"2024-03-24T12:16:22.206894+0000","last_became_active":"2024-03-24T12:15:15.273721+0000","last_became_peered":"2024-03-24T12:15:15.273721+0000","last_unstale":"2024-03-24T12:16:22.206894+0000","last_undegraded":"2024-03-24T12:16:22.206894+0000","last_fullsized":"2024-03-24T12:16:22.206894+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:31.196132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:15:31.196132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:48.882517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":925,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192441+0000","last_change":"2024-03-24T12:15:43.074062+0000","last_active":"2024-03-24T12:16:22.192441+0000","last_peered":"2024-03-24T12:16:22.192441+0000","last_clean":"2024-03-24T12:16:22.192441+0000","last_became_active":"2024-03-24T12:15:43.073695+0000","last_became_peered":"2024-03-24T12:15:43.073695+0000","last_unstale":"2024-03-24T12:16:22.192441+0000","last_undegraded":"2024-03-24T12:16:22.192441+0000","last_fullsized":"2024-03-24T12:16:22.192441+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:30.173007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:30.173007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:47.767908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1050,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358167+0000","last_change":"2024-03-24T12:16:07.397051+0000","last_active":"2024-03-24T12:16:22.358167+0000","last_peered":"2024-03-24T12:16:22.358167+0000","last_clean":"2024-03-24T12:16:22.358167+0000","last_became_active":"2024-03-24T11:52:21.389935+0000","last_became_peered":"2024-03-24T11:52:21.389935+0000","last_unstale":"2024-03-24T12:16:22.358167+0000","last_undegraded":"2024-03-24T12:16:22.358167+0000","last_fullsized":"2024-03-24T12:16:22.358167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:07.396899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:01.133663+0000","last_clean_scrub_stamp":"2024-03-24T12:16:07.396899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:29.374194+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"450'27","reported_seq":1070,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066257+0000","last_change":"2024-03-24T12:16:22.041049+0000","last_active":"2024-03-24T12:16:22.066257+0000","last_peered":"2024-03-24T12:16:22.066257+0000","last_clean":"2024-03-24T12:16:22.066257+0000","last_became_active":"2024-03-24T11:52:17.423510+0000","last_became_peered":"2024-03-24T11:52:17.423510+0000","last_unstale":"2024-03-24T12:16:22.066257+0000","last_undegraded":"2024-03-24T12:16:22.066257+0000","last_fullsized":"2024-03-24T12:16:22.066257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'27","last_scrub_stamp":"2024-03-24T12:16:22.040954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:22.040954+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:27.335762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":940,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:24.375561+0000","last_change":"2024-03-24T12:16:24.375584+0000","last_active":"2024-03-24T12:16:24.375561+0000","last_peered":"2024-03-24T12:16:24.375561+0000","last_clean":"2024-03-24T12:16:24.375561+0000","last_became_active":"2024-03-24T12:15:43.073733+0000","last_became_peered":"2024-03-24T12:15:43.073733+0000","last_unstale":"2024-03-24T12:16:24.375561+0000","last_undegraded":"2024-03-24T12:16:24.375561+0000","last_fullsized":"2024-03-24T12:16:24.375561+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:24.375491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:09:38.737417+0000","last_clean_scrub_stamp":"2024-03-24T12:16:24.375491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:33.764712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1054,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358373+0000","last_change":"2024-03-24T12:15:32.672271+0000","last_active":"2024-03-24T12:16:22.358373+0000","last_peered":"2024-03-24T12:16:22.358373+0000","last_clean":"2024-03-24T12:16:22.358373+0000","last_became_active":"2024-03-24T11:52:19.384899+0000","last_became_peered":"2024-03-24T11:52:19.384899+0000","last_unstale":"2024-03-24T12:16:22.358373+0000","last_undegraded":"2024-03-24T12:16:22.358373+0000","last_fullsized":"2024-03-24T12:16:22.358373+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:32.672168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:00.121445+0000","last_clean_scrub_stamp":"2024-03-24T12:15:32.672168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:58.792284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1019,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067277+0000","last_change":"2024-03-24T12:16:12.261196+0000","last_active":"2024-03-24T12:16:22.067277+0000","last_peered":"2024-03-24T12:16:22.067277+0000","last_clean":"2024-03-24T12:16:22.067277+0000","last_became_active":"2024-03-24T11:52:21.853802+0000","last_became_peered":"2024-03-24T11:52:21.853802+0000","last_unstale":"2024-03-24T12:16:22.067277+0000","last_undegraded":"2024-03-24T12:16:22.067277+0000","last_fullsized":"2024-03-24T12:16:22.067277+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:12.261087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:12.261087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:37.661608+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"450'35","reported_seq":1093,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067532+0000","last_change":"2024-03-24T12:15:42.472780+0000","last_active":"2024-03-24T12:16:22.067532+0000","last_peered":"2024-03-24T12:16:22.067532+0000","last_clean":"2024-03-24T12:16:22.067532+0000","last_became_active":"2024-03-24T11:52:17.425813+0000","last_became_peered":"2024-03-24T11:52:17.425813+0000","last_unstale":"2024-03-24T12:16:22.067532+0000","last_undegraded":"2024-03-24T12:16:22.067532+0000","last_fullsized":"2024-03-24T12:16:22.067532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'35","last_scrub_stamp":"2024-03-24T12:15:42.472681+0000","last_deep_scrub":"450'35","last_deep_scrub_stamp":"2024-03-24T12:14:22.685599+0000","last_clean_scrub_stamp":"2024-03-24T12:15:42.472681+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:10.220741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":889,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206191+0000","last_change":"2024-03-24T12:15:46.179963+0000","last_active":"2024-03-24T12:16:22.206191+0000","last_peered":"2024-03-24T12:16:22.206191+0000","last_clean":"2024-03-24T12:16:22.206191+0000","last_became_active":"2024-03-24T12:08:36.159196+0000","last_became_peered":"2024-03-24T12:08:36.159196+0000","last_unstale":"2024-03-24T12:16:22.206191+0000","last_undegraded":"2024-03-24T12:16:22.206191+0000","last_fullsized":"2024-03-24T12:16:22.206191+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:46.179863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:46.179863+0000","last_clean_scrub_stamp":"2024-03-24T12:15:46.179863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:56.746059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"525'11","reported_seq":1205,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.487520+0000","last_change":"2024-03-24T12:16:22.360528+0000","last_active":"2024-03-24T12:16:35.487520+0000","last_peered":"2024-03-24T12:16:35.487520+0000","last_clean":"2024-03-24T12:16:35.487520+0000","last_became_active":"2024-03-24T12:15:43.070184+0000","last_became_peered":"2024-03-24T12:15:43.070184+0000","last_unstale":"2024-03-24T12:16:35.487520+0000","last_undegraded":"2024-03-24T12:16:35.487520+0000","last_fullsized":"2024-03-24T12:16:35.487520+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'11","last_scrub_stamp":"2024-03-24T12:16:22.360425+0000","last_deep_scrub":"274'9","last_deep_scrub_stamp":"2024-03-24T12:03:56.395650+0000","last_clean_scrub_stamp":"2024-03-24T12:16:22.360425+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:24.677271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"194'2","reported_seq":1068,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:30.703494+0000","last_change":"2024-03-24T12:16:30.703525+0000","last_active":"2024-03-24T12:16:30.703494+0000","last_peered":"2024-03-24T12:16:30.703494+0000","last_clean":"2024-03-24T12:16:30.703494+0000","last_became_active":"2024-03-24T12:08:36.161142+0000","last_became_peered":"2024-03-24T12:08:36.161142+0000","last_unstale":"2024-03-24T12:16:30.703494+0000","last_undegraded":"2024-03-24T12:16:30.703494+0000","last_fullsized":"2024-03-24T12:16:30.703494+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-03-24T12:16:30.703415+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-03-24T12:16:30.703415+0000","last_clean_scrub_stamp":"2024-03-24T12:16:30.703415+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:41.855797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"450'27","reported_seq":991,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:26.357997+0000","last_change":"2024-03-24T12:16:26.358035+0000","last_active":"2024-03-24T12:16:26.357997+0000","last_peered":"2024-03-24T12:16:26.357997+0000","last_clean":"2024-03-24T12:16:26.357997+0000","last_became_active":"2024-03-24T12:15:43.074105+0000","last_became_peered":"2024-03-24T12:15:43.074105+0000","last_unstale":"2024-03-24T12:16:26.357997+0000","last_undegraded":"2024-03-24T12:16:26.357997+0000","last_fullsized":"2024-03-24T12:16:26.357997+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'27","last_scrub_stamp":"2024-03-24T12:16:26.357929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:26.357929+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:50.180785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'1","reported_seq":1032,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066120+0000","last_change":"2024-03-24T12:16:18.048156+0000","last_active":"2024-03-24T12:16:22.066120+0000","last_peered":"2024-03-24T12:16:22.066120+0000","last_clean":"2024-03-24T12:16:22.066120+0000","last_became_active":"2024-03-24T11:52:15.135982+0000","last_became_peered":"2024-03-24T11:52:15.135982+0000","last_unstale":"2024-03-24T12:16:22.066120+0000","last_undegraded":"2024-03-24T12:16:22.066120+0000","last_fullsized":"2024-03-24T12:16:22.066120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:16:18.048062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:18.048062+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:42.593354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1063,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.067621+0000","last_change":"2024-03-24T12:15:24.522375+0000","last_active":"2024-03-24T12:16:22.067621+0000","last_peered":"2024-03-24T12:16:22.067621+0000","last_clean":"2024-03-24T12:16:22.067621+0000","last_became_active":"2024-03-24T12:15:15.276200+0000","last_became_peered":"2024-03-24T12:15:15.276200+0000","last_unstale":"2024-03-24T12:16:22.067621+0000","last_undegraded":"2024-03-24T12:16:22.067621+0000","last_fullsized":"2024-03-24T12:16:22.067621+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:24.522298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:58:07.457593+0000","last_clean_scrub_stamp":"2024-03-24T12:15:24.522298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:49.077692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":647,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194809+0000","last_change":"2024-03-24T12:15:52.461926+0000","last_active":"2024-03-24T12:16:22.194809+0000","last_peered":"2024-03-24T12:16:22.194809+0000","last_clean":"2024-03-24T12:16:22.194809+0000","last_became_active":"2024-03-24T12:15:11.732134+0000","last_became_peered":"2024-03-24T12:15:11.732134+0000","last_unstale":"2024-03-24T12:16:22.194809+0000","last_undegraded":"2024-03-24T12:16:22.194809+0000","last_fullsized":"2024-03-24T12:16:22.194809+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:52.461807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:04:14.188600+0000","last_clean_scrub_stamp":"2024-03-24T12:15:52.461807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:19.717802+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"450'46","reported_seq":115,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.068230+0000","last_change":"2024-03-24T12:16:18.531346+0000","last_active":"2024-03-24T12:16:22.068230+0000","last_peered":"2024-03-24T12:16:22.068230+0000","last_clean":"2024-03-24T12:16:22.068230+0000","last_became_active":"2024-03-24T12:15:58.147597+0000","last_became_peered":"2024-03-24T12:15:58.147597+0000","last_unstale":"2024-03-24T12:16:22.068230+0000","last_undegraded":"2024-03-24T12:16:22.068230+0000","last_fullsized":"2024-03-24T12:16:22.068230+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'46","last_scrub_stamp":"2024-03-24T12:16:18.531253+0000","last_deep_scrub":"115'22","last_deep_scrub_stamp":"2024-03-24T11:55:12.618646+0000","last_clean_scrub_stamp":"2024-03-24T12:16:18.531253+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:31.798219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":933,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.192808+0000","last_change":"2024-03-24T12:15:52.467989+0000","last_active":"2024-03-24T12:16:22.192808+0000","last_peered":"2024-03-24T12:16:22.192808+0000","last_clean":"2024-03-24T12:16:22.192808+0000","last_became_active":"2024-03-24T12:15:43.070069+0000","last_became_peered":"2024-03-24T12:15:43.070069+0000","last_unstale":"2024-03-24T12:16:22.192808+0000","last_undegraded":"2024-03-24T12:16:22.192808+0000","last_fullsized":"2024-03-24T12:16:22.192808+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:52.467912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:21.570940+0000","last_clean_scrub_stamp":"2024-03-24T12:15:52.467912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:06.639003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":806,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.262983+0000","last_change":"2024-03-24T12:16:35.263003+0000","last_active":"2024-03-24T12:16:35.262983+0000","last_peered":"2024-03-24T12:16:35.262983+0000","last_clean":"2024-03-24T12:16:35.262983+0000","last_became_active":"2024-03-24T12:15:11.249540+0000","last_became_peered":"2024-03-24T12:15:11.249540+0000","last_unstale":"2024-03-24T12:16:35.262983+0000","last_undegraded":"2024-03-24T12:16:35.262983+0000","last_fullsized":"2024-03-24T12:16:35.262983+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:35.262924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:07:53.354213+0000","last_clean_scrub_stamp":"2024-03-24T12:16:35.262924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:01.953332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1037,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355603+0000","last_change":"2024-03-24T12:15:38.352890+0000","last_active":"2024-03-24T12:16:22.355603+0000","last_peered":"2024-03-24T12:16:22.355603+0000","last_clean":"2024-03-24T12:16:22.355603+0000","last_became_active":"2024-03-24T12:08:36.166629+0000","last_became_peered":"2024-03-24T12:08:36.166629+0000","last_unstale":"2024-03-24T12:16:22.355603+0000","last_undegraded":"2024-03-24T12:16:22.355603+0000","last_fullsized":"2024-03-24T12:16:22.355603+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:38.352742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:02:18.004438+0000","last_clean_scrub_stamp":"2024-03-24T12:15:38.352742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:39.207341+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"450'29","reported_seq":1090,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066262+0000","last_change":"2024-03-24T12:15:21.150250+0000","last_active":"2024-03-24T12:16:22.066262+0000","last_peered":"2024-03-24T12:16:22.066262+0000","last_clean":"2024-03-24T12:16:22.066262+0000","last_became_active":"2024-03-24T11:52:37.791263+0000","last_became_peered":"2024-03-24T11:52:37.791263+0000","last_unstale":"2024-03-24T12:16:22.066262+0000","last_undegraded":"2024-03-24T12:16:22.066262+0000","last_fullsized":"2024-03-24T12:16:22.066262+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'29","last_scrub_stamp":"2024-03-24T12:15:21.150152+0000","last_deep_scrub":"242'23","last_deep_scrub_stamp":"2024-03-24T12:02:26.446597+0000","last_clean_scrub_stamp":"2024-03-24T12:15:21.150152+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:40.365739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"67'1","reported_seq":1044,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:24.095474+0000","last_change":"2024-03-24T12:16:24.095501+0000","last_active":"2024-03-24T12:16:24.095474+0000","last_peered":"2024-03-24T12:16:24.095474+0000","last_clean":"2024-03-24T12:16:24.095474+0000","last_became_active":"2024-03-24T11:52:15.137815+0000","last_became_peered":"2024-03-24T11:52:15.137815+0000","last_unstale":"2024-03-24T12:16:24.095474+0000","last_undegraded":"2024-03-24T12:16:24.095474+0000","last_fullsized":"2024-03-24T12:16:24.095474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:16:24.095405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:14.117087+0000","last_clean_scrub_stamp":"2024-03-24T12:16:24.095405+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:30.400702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1044,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355195+0000","last_change":"2024-03-24T12:15:59.275185+0000","last_active":"2024-03-24T12:16:22.355195+0000","last_peered":"2024-03-24T12:16:22.355195+0000","last_clean":"2024-03-24T12:16:22.355195+0000","last_became_active":"2024-03-24T12:15:15.275842+0000","last_became_peered":"2024-03-24T12:15:15.275842+0000","last_unstale":"2024-03-24T12:16:22.355195+0000","last_undegraded":"2024-03-24T12:16:22.355195+0000","last_fullsized":"2024-03-24T12:16:22.355195+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:59.275042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:59.275042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:24.469337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1032,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065795+0000","last_change":"2024-03-24T12:15:56.422691+0000","last_active":"2024-03-24T12:16:22.065795+0000","last_peered":"2024-03-24T12:16:22.065795+0000","last_clean":"2024-03-24T12:16:22.065795+0000","last_became_active":"2024-03-24T12:08:36.156539+0000","last_became_peered":"2024-03-24T12:08:36.156539+0000","last_unstale":"2024-03-24T12:16:22.065795+0000","last_undegraded":"2024-03-24T12:16:22.065795+0000","last_fullsized":"2024-03-24T12:16:22.065795+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:56.422585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:56.422585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:20.505323+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"450'21","reported_seq":1072,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:36.471903+0000","last_change":"2024-03-24T12:16:36.471949+0000","last_active":"2024-03-24T12:16:36.471903+0000","last_peered":"2024-03-24T12:16:36.471903+0000","last_clean":"2024-03-24T12:16:36.471903+0000","last_became_active":"2024-03-24T11:52:17.341956+0000","last_became_peered":"2024-03-24T11:52:17.341956+0000","last_unstale":"2024-03-24T12:16:36.471903+0000","last_undegraded":"2024-03-24T12:16:36.471903+0000","last_fullsized":"2024-03-24T12:16:36.471903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'21","last_scrub_stamp":"2024-03-24T12:16:36.471828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:16.324438+0000","last_clean_scrub_stamp":"2024-03-24T12:16:36.471828+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:46.178405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1073,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.358103+0000","last_change":"2024-03-24T12:15:39.458306+0000","last_active":"2024-03-24T12:16:22.358103+0000","last_peered":"2024-03-24T12:16:22.358103+0000","last_clean":"2024-03-24T12:16:22.358103+0000","last_became_active":"2024-03-24T12:08:36.163320+0000","last_became_peered":"2024-03-24T12:08:36.163320+0000","last_unstale":"2024-03-24T12:16:22.358103+0000","last_undegraded":"2024-03-24T12:16:22.358103+0000","last_fullsized":"2024-03-24T12:16:22.358103+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:39.458228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:09.177509+0000","last_clean_scrub_stamp":"2024-03-24T12:15:39.458228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:53.581008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":950,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206564+0000","last_change":"2024-03-24T12:15:49.170901+0000","last_active":"2024-03-24T12:16:22.206564+0000","last_peered":"2024-03-24T12:16:22.206564+0000","last_clean":"2024-03-24T12:16:22.206564+0000","last_became_active":"2024-03-24T12:08:36.169981+0000","last_became_peered":"2024-03-24T12:08:36.169981+0000","last_unstale":"2024-03-24T12:16:22.206564+0000","last_undegraded":"2024-03-24T12:16:22.206564+0000","last_fullsized":"2024-03-24T12:16:22.206564+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:49.170749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:49.170749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:55.259771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1033,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:32.762832+0000","last_change":"2024-03-24T12:16:32.762832+0000","last_active":"2024-03-24T12:16:32.762832+0000","last_peered":"2024-03-24T12:16:32.762832+0000","last_clean":"2024-03-24T12:16:32.762832+0000","last_became_active":"2024-03-24T11:52:21.391460+0000","last_became_peered":"2024-03-24T11:52:21.391460+0000","last_unstale":"2024-03-24T12:16:32.762832+0000","last_undegraded":"2024-03-24T12:16:32.762832+0000","last_fullsized":"2024-03-24T12:16:32.762832+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:32.762735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:03:19.827956+0000","last_clean_scrub_stamp":"2024-03-24T12:16:32.762735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:02.500477+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"450'11","reported_seq":1080,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.355251+0000","last_change":"2024-03-24T12:16:10.346536+0000","last_active":"2024-03-24T12:16:22.355251+0000","last_peered":"2024-03-24T12:16:22.355251+0000","last_clean":"2024-03-24T12:16:22.355251+0000","last_became_active":"2024-03-24T12:15:05.234148+0000","last_became_peered":"2024-03-24T12:15:05.234148+0000","last_unstale":"2024-03-24T12:16:22.355251+0000","last_undegraded":"2024-03-24T12:16:22.355251+0000","last_fullsized":"2024-03-24T12:16:22.355251+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'11","last_scrub_stamp":"2024-03-24T12:16:10.346429+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-24T11:56:24.027577+0000","last_clean_scrub_stamp":"2024-03-24T12:16:10.346429+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:32.816914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1077,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:31.750817+0000","last_change":"2024-03-24T12:16:31.750856+0000","last_active":"2024-03-24T12:16:31.750817+0000","last_peered":"2024-03-24T12:16:31.750817+0000","last_clean":"2024-03-24T12:16:31.750817+0000","last_became_active":"2024-03-24T11:52:15.146759+0000","last_became_peered":"2024-03-24T11:52:15.146759+0000","last_unstale":"2024-03-24T12:16:31.750817+0000","last_undegraded":"2024-03-24T12:16:31.750817+0000","last_fullsized":"2024-03-24T12:16:31.750817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:31.750736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:00:03.053839+0000","last_clean_scrub_stamp":"2024-03-24T12:16:31.750736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:57.025898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1081,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:29.736170+0000","last_change":"2024-03-24T12:16:29.736218+0000","last_active":"2024-03-24T12:16:29.736170+0000","last_peered":"2024-03-24T12:16:29.736170+0000","last_clean":"2024-03-24T12:16:29.736170+0000","last_became_active":"2024-03-24T12:15:05.225818+0000","last_became_peered":"2024-03-24T12:15:05.225818+0000","last_unstale":"2024-03-24T12:16:29.736170+0000","last_undegraded":"2024-03-24T12:16:29.736170+0000","last_fullsized":"2024-03-24T12:16:29.736170+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:29.736056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:02.631340+0000","last_clean_scrub_stamp":"2024-03-24T12:16:29.736056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:37.810931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":968,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.206215+0000","last_change":"2024-03-24T12:15:21.206697+0000","last_active":"2024-03-24T12:16:22.206215+0000","last_peered":"2024-03-24T12:16:22.206215+0000","last_clean":"2024-03-24T12:16:22.206215+0000","last_became_active":"2024-03-24T12:15:05.212886+0000","last_became_peered":"2024-03-24T12:15:05.212886+0000","last_unstale":"2024-03-24T12:16:22.206215+0000","last_undegraded":"2024-03-24T12:16:22.206215+0000","last_fullsized":"2024-03-24T12:16:22.206215+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:21.206577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:53:25.779615+0000","last_clean_scrub_stamp":"2024-03-24T12:15:21.206577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:50.971021+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"450'25","reported_seq":1109,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.354838+0000","last_change":"2024-03-24T12:16:16.303225+0000","last_active":"2024-03-24T12:16:22.354838+0000","last_peered":"2024-03-24T12:16:22.354838+0000","last_clean":"2024-03-24T12:16:22.354838+0000","last_became_active":"2024-03-24T12:15:05.232578+0000","last_became_peered":"2024-03-24T12:15:05.232578+0000","last_unstale":"2024-03-24T12:16:22.354838+0000","last_undegraded":"2024-03-24T12:16:22.354838+0000","last_fullsized":"2024-03-24T12:16:22.354838+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'25","last_scrub_stamp":"2024-03-24T12:16:16.303118+0000","last_deep_scrub":"176'13","last_deep_scrub_stamp":"2024-03-24T11:59:14.935318+0000","last_clean_scrub_stamp":"2024-03-24T12:16:16.303118+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:37.404365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1042,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.066682+0000","last_change":"2024-03-24T12:15:50.283561+0000","last_active":"2024-03-24T12:16:22.066682+0000","last_peered":"2024-03-24T12:16:22.066682+0000","last_clean":"2024-03-24T12:16:22.066682+0000","last_became_active":"2024-03-24T12:08:36.153384+0000","last_became_peered":"2024-03-24T12:08:36.153384+0000","last_unstale":"2024-03-24T12:16:22.066682+0000","last_undegraded":"2024-03-24T12:16:22.066682+0000","last_fullsized":"2024-03-24T12:16:22.066682+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:50.283462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:15:50.283462+0000","last_clean_scrub_stamp":"2024-03-24T12:15:50.283462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:11.421671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":457,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.194391+0000","last_change":"2024-03-24T12:15:11.732158+0000","last_active":"2024-03-24T12:16:22.194391+0000","last_peered":"2024-03-24T12:16:22.194391+0000","last_clean":"2024-03-24T12:16:22.194391+0000","last_became_active":"2024-03-24T12:15:11.731693+0000","last_became_peered":"2024-03-24T12:15:11.731693+0000","last_unstale":"2024-03-24T12:16:22.194391+0000","last_undegraded":"2024-03-24T12:16:22.194391+0000","last_fullsized":"2024-03-24T12:16:22.194391+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:10.187179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:18.347513+0000","last_clean_scrub_stamp":"2024-03-24T12:15:10.187179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:37.086575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1043,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.354978+0000","last_change":"2024-03-24T12:15:34.308108+0000","last_active":"2024-03-24T12:16:22.354978+0000","last_peered":"2024-03-24T12:16:22.354978+0000","last_clean":"2024-03-24T12:16:22.354978+0000","last_became_active":"2024-03-24T12:15:05.234180+0000","last_became_peered":"2024-03-24T12:15:05.234180+0000","last_unstale":"2024-03-24T12:16:22.354978+0000","last_undegraded":"2024-03-24T12:16:22.354978+0000","last_fullsized":"2024-03-24T12:16:22.354978+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:34.307998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:15:34.307998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:37.500517+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"450'46","reported_seq":93,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065776+0000","last_change":"2024-03-24T12:15:58.231242+0000","last_active":"2024-03-24T12:16:22.065776+0000","last_peered":"2024-03-24T12:16:22.065776+0000","last_clean":"2024-03-24T12:16:22.065776+0000","last_became_active":"2024-03-24T12:15:15.271403+0000","last_became_peered":"2024-03-24T12:15:15.271403+0000","last_unstale":"2024-03-24T12:16:22.065776+0000","last_undegraded":"2024-03-24T12:16:22.065776+0000","last_fullsized":"2024-03-24T12:16:22.065776+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'46","last_scrub_stamp":"2024-03-24T12:15:58.231106+0000","last_deep_scrub":"117'22","last_deep_scrub_stamp":"2024-03-24T11:55:21.286727+0000","last_clean_scrub_stamp":"2024-03-24T12:15:58.231106+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:16.795636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1046,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:35.679522+0000","last_change":"2024-03-24T12:16:35.679543+0000","last_active":"2024-03-24T12:16:35.679522+0000","last_peered":"2024-03-24T12:16:35.679522+0000","last_clean":"2024-03-24T12:16:35.679522+0000","last_became_active":"2024-03-24T11:52:15.137381+0000","last_became_peered":"2024-03-24T11:52:15.137381+0000","last_unstale":"2024-03-24T12:16:35.679522+0000","last_undegraded":"2024-03-24T12:16:35.679522+0000","last_fullsized":"2024-03-24T12:16:35.679522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:35.679460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:54:13.817437+0000","last_clean_scrub_stamp":"2024-03-24T12:16:35.679460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:46.601461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":501,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:22.065126+0000","last_change":"2024-03-24T12:15:41.243752+0000","last_active":"2024-03-24T12:16:22.065126+0000","last_peered":"2024-03-24T12:16:22.065126+0000","last_clean":"2024-03-24T12:16:22.065126+0000","last_became_active":"2024-03-24T12:05:56.862892+0000","last_became_peered":"2024-03-24T12:05:56.862892+0000","last_unstale":"2024-03-24T12:16:22.065126+0000","last_undegraded":"2024-03-24T12:16:22.065126+0000","last_fullsized":"2024-03-24T12:16:22.065126+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:15:41.243660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:05:12.732801+0000","last_clean_scrub_stamp":"2024-03-24T12:15:41.243660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:16:47.786288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1026,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-24T12:16:32.244258+0000","last_change":"2024-03-24T12:16:32.244281+0000","last_active":"2024-03-24T12:16:32.244258+0000","last_peered":"2024-03-24T12:16:32.244258+0000","last_clean":"2024-03-24T12:16:32.244258+0000","last_became_active":"2024-03-24T11:52:21.854027+0000","last_became_peered":"2024-03-24T11:52:21.854027+0000","last_unstale":"2024-03-24T12:16:32.244258+0000","last_undegraded":"2024-03-24T12:16:32.244258+0000","last_fullsized":"2024-03-24T12:16:32.244258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:16:32.244183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:52:20.361608+0000","last_clean_scrub_stamp":"2024-03-24T12:16:32.244183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:00.084367+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":390,"ondisk_log_size":390,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":640,"num_read_kb":635,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1545,"num_read_kb":1302,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":4044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},"log_size":1096,"ondisk_log_size":1096,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005354,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880692,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864460,"statfs":{"total":95995035648,"available":95093207040,"internally_reserved":0,"allocated":1290240,"data_stored":823567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":900528586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332401,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862320,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92882832,"statfs":{"total":95995035648,"available":95112019968,"internally_reserved":0,"allocated":958464,"data_stored":495705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":882047430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575177,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960908,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92784244,"statfs":{"total":95995035648,"available":95011065856,"internally_reserved":0,"allocated":987136,"data_stored":495642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738703,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970616,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92774536,"statfs":{"total":95995035648,"available":95001124864,"internally_reserved":0,"allocated":966656,"data_stored":494864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411653,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636276,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93108876,"statfs":{"total":95995035648,"available":95343489024,"internally_reserved":0,"allocated":1224704,"data_stored":758172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117311,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939280,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92805872,"statfs":{"total":95995035648,"available":95033212928,"internally_reserved":0,"allocated":1581056,"data_stored":1086502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":960231879},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921082,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880384,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864768,"statfs":{"total":95995035648,"available":95093522432,"internally_reserved":0,"allocated":974848,"data_stored":495319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":353,"seq":1516123455588,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":390768,"kb_used_data":1512,"kb_used_omap":7,"kb_used_meta":389240,"kb_avail":93354384,"statfs":{"total":95995035648,"available":95594889216,"internally_reserved":0,"allocated":1548288,"data_stored":1085551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8080,"internal_metadata":398581872},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:16:48.230 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-24T12:16:48.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-24T12:16:48.479 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:16:48.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:48 smithi064 ceph-mon[42243]: 6.1a scrub starts 2024-03-24T12:16:48.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:48 smithi064 ceph-mon[42243]: 6.1a scrub ok 2024-03-24T12:16:48.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:48 smithi064 ceph-mon[30223]: 6.1a scrub starts 2024-03-24T12:16:48.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:48 smithi064 ceph-mon[30223]: 6.1a scrub ok 2024-03-24T12:16:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:48 smithi193 ceph-mon[33593]: 6.1a scrub starts 2024-03-24T12:16:48.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:48 smithi193 ceph-mon[33593]: 6.1a scrub ok 2024-03-24T12:16:48.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:16:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug there is no tcmu-runner data available 2024-03-24T12:16:49.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: from='client.36332 v1:172.21.15.64:0/1860040460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:16:49.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:49.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:49.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: 1.3 scrub starts 2024-03-24T12:16:49.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: 1.3 scrub ok 2024-03-24T12:16:49.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: 4.a deep-scrub starts 2024-03-24T12:16:49.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: 4.a deep-scrub ok 2024-03-24T12:16:49.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: 4.d scrub starts 2024-03-24T12:16:49.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: 4.d scrub ok 2024-03-24T12:16:49.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: 2.e deep-scrub starts 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[30223]: 2.e deep-scrub ok 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: from='client.36332 v1:172.21.15.64:0/1860040460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: 1.3 scrub starts 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: 1.3 scrub ok 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: 4.a deep-scrub starts 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: 4.a deep-scrub ok 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: 4.d scrub starts 2024-03-24T12:16:49.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: 4.d scrub ok 2024-03-24T12:16:49.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: 2.e deep-scrub starts 2024-03-24T12:16:49.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:49 smithi064 ceph-mon[42243]: 2.e deep-scrub ok 2024-03-24T12:16:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: from='client.36332 v1:172.21.15.64:0/1860040460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:16:49.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: from='client.34439 v1:172.21.15.193:0/689878447' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: 1.3 scrub starts 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: 1.3 scrub ok 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: 4.a deep-scrub starts 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: 4.a deep-scrub ok 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: 4.d scrub starts 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: 4.d scrub ok 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: 2.e deep-scrub starts 2024-03-24T12:16:49.586 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:49 smithi193 ceph-mon[33593]: 2.e deep-scrub ok 2024-03-24T12:16:50.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-mon[30223]: 2.16 deep-scrub starts 2024-03-24T12:16:50.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-mon[30223]: 2.16 deep-scrub ok 2024-03-24T12:16:50.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-mon[30223]: 6.1e scrub starts 2024-03-24T12:16:50.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-mon[30223]: 6.1e scrub ok 2024-03-24T12:16:50.324 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-mon[42243]: 2.16 deep-scrub starts 2024-03-24T12:16:50.324 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-mon[42243]: 2.16 deep-scrub ok 2024-03-24T12:16:50.324 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-mon[42243]: 6.1e scrub starts 2024-03-24T12:16:50.324 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-mon[42243]: 6.1e scrub ok 2024-03-24T12:16:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:50 smithi193 ceph-mon[33593]: 2.16 deep-scrub starts 2024-03-24T12:16:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:50 smithi193 ceph-mon[33593]: 2.16 deep-scrub ok 2024-03-24T12:16:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:50 smithi193 ceph-mon[33593]: 6.1e scrub starts 2024-03-24T12:16:50.585 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:50 smithi193 ceph-mon[33593]: 6.1e scrub ok 2024-03-24T12:16:50.668 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:50] ENGINE Bus STOPPING 2024-03-24T12:16:50.668 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:16:50.668 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:50] ENGINE Bus STOPPED 2024-03-24T12:16:50.668 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:50] ENGINE Bus STARTING 2024-03-24T12:16:50.668 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:50] ENGINE Serving on http://:::9283 2024-03-24T12:16:50.668 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:50 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:50] ENGINE Bus STARTED 2024-03-24T12:16:51.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:51.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: 1.9 scrub starts 2024-03-24T12:16:51.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: 1.9 scrub ok 2024-03-24T12:16:51.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: 5.18 scrub starts 2024-03-24T12:16:51.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: 5.18 scrub ok 2024-03-24T12:16:51.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: 5.11 scrub starts 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: 5.11 scrub ok 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: 5.e scrub starts 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: 5.e scrub ok 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4084962536' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: from='client.? v1:172.21.15.64:0/4084962536' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[30223]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: 1.9 scrub starts 2024-03-24T12:16:51.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: 1.9 scrub ok 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: 5.18 scrub starts 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: 5.18 scrub ok 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: 5.11 scrub starts 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: 5.11 scrub ok 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: 5.e scrub starts 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: 5.e scrub ok 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4084962536' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: from='client.? v1:172.21.15.64:0/4084962536' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:51.272 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-mon[42243]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:16:51.311 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T12:16:51.324 INFO:tasks.cephadm:Teardown begin 2024-03-24T12:16:51.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:16:51.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: 1.9 scrub starts 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: 1.9 scrub ok 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: 5.18 scrub starts 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: 5.18 scrub ok 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: 5.11 scrub starts 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: 5.11 scrub ok 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: 5.e scrub starts 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: 5.e scrub ok 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4084962536' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T12:16:51.424 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: from='client.? v1:172.21.15.64:0/4084962536' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T12:16:51.425 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:51.425 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:51.425 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:51.425 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:51 smithi193 ceph-mon[33593]: from='mgr.34493 v1:172.21.15.64:0/715494133' entity='mgr.y' 2024-03-24T12:16:51.426 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T12:16:51.426 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:16:51.454 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:16:51.477 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T12:16:51.477 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T12:16:51.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.a 2024-03-24T12:16:51.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 systemd[1]: Stopping Ceph mon.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:16:52.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a[30200]: 2024-03-24T12:16:51.818+0000 7f33fece2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:52.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a[30200]: 2024-03-24T12:16:51.818+0000 7f33fece2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T12:16:52.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:51 smithi064 podman[241553]: 2024-03-24 12:16:51.910114141 +0000 UTC m=+0.208849104 container died 14686e0dd6a64be73fc7f01107e418099640a63202422466e0d2b74ea16fc72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T12:16:52.216 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:51] ENGINE Bus STOPPING 2024-03-24T12:16:52.216 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:16:52.216 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:51] ENGINE Bus STOPPED 2024-03-24T12:16:52.216 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:51] ENGINE Bus STARTING 2024-03-24T12:16:52.216 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:51] ENGINE Serving on http://:::9283 2024-03-24T12:16:52.216 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:51 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:51] ENGINE Bus STARTED 2024-03-24T12:16:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:52 smithi064 podman[241553]: 2024-03-24 12:16:52.35788698 +0000 UTC m=+0.656621947 container cleanup 14686e0dd6a64be73fc7f01107e418099640a63202422466e0d2b74ea16fc72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:16:52.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Mar 24 12:16:52 smithi064 bash[241553]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-a 2024-03-24T12:16:52.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T12:16:52.834 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:52.834 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T12:16:52.834 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-24T12:16:52.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.c 2024-03-24T12:16:53.051 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:52 smithi064 systemd[1]: Stopping Ceph mon.c for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:16:53.492 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:53] ENGINE Bus STOPPING 2024-03-24T12:16:53.492 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:16:53.492 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:53] ENGINE Bus STOPPED 2024-03-24T12:16:53.492 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:53] ENGINE Bus STARTING 2024-03-24T12:16:53.493 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:53] ENGINE Serving on http://:::9283 2024-03-24T12:16:53.493 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: [24/Mar/2024:12:16:53] ENGINE Bus STARTED 2024-03-24T12:16:53.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-c[42220]: 2024-03-24T12:16:53.147+0000 7fdaa0fdc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:53.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:53 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-c[42220]: 2024-03-24T12:16:53.147+0000 7fdaa0fdc700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T12:16:53.493 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:53 smithi064 podman[241694]: 2024-03-24 12:16:53.179325134 +0000 UTC m=+0.164596639 container died 1616be2bab01d62ce834526346f1820d4353d8f61be752ff6b38e678b3992e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-c, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:16:53.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:53 smithi064 podman[241694]: 2024-03-24 12:16:53.596287924 +0000 UTC m=+0.581559431 container cleanup 1616be2bab01d62ce834526346f1820d4353d8f61be752ff6b38e678b3992e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-c, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:16:53.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Mar 24 12:16:53 smithi064 bash[241694]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-c 2024-03-24T12:16:53.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T12:16:53.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:53.962 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-24T12:16:53.963 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-24T12:16:53.963 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.b 2024-03-24T12:16:54.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:54 smithi193 systemd[1]: Stopping Ceph mon.b for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:16:54.605 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-b[33556]: 2024-03-24T12:16:54.406+0000 7f9dba72e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:54.605 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-b[33556]: 2024-03-24T12:16:54.406+0000 7f9dba72e700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T12:16:54.605 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:54 smithi193 podman[166318]: 2024-03-24 12:16:54.55015168 +0000 UTC m=+0.281703495 container died b9bde26803e0620d57cac292d3fee13e72c40e6a9cbcb079cedb37fd91aba3f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=squid-db0330b) 2024-03-24T12:16:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:55 smithi193 podman[166318]: 2024-03-24 12:16:55.088996171 +0000 UTC m=+0.820547985 container cleanup b9bde26803e0620d57cac292d3fee13e72c40e6a9cbcb079cedb37fd91aba3f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T12:16:55.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:55 smithi193 bash[166318]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-b 2024-03-24T12:16:55.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T12:16:55.594 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:55 smithi193 podman[166338]: 2024-03-24 12:16:55.340721641 +0000 UTC m=+0.781799697 container remove b9bde26803e0620d57cac292d3fee13e72c40e6a9cbcb079cedb37fd91aba3f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mon-b, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-24T12:16:55.595 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:55 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-24T12:16:55.595 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:55 smithi193 systemd[1]: Stopped Ceph mon.b for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T12:16:55.595 INFO:journalctl@ceph.mon.b.smithi193.stdout:Mar 24 12:16:55 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mon.b.service: Consumed 23.222s CPU time. 2024-03-24T12:16:55.610 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:55.611 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-24T12:16:55.611 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T12:16:55.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.y 2024-03-24T12:16:55.910 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:55 smithi064 systemd[1]: Stopping Ceph mgr.y for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:16:56.247 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T12:16:55.910+0000 7f5625551700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:56.247 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:55 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y[30485]: 2024-03-24T12:16:55.910+0000 7f5625551700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-24T12:16:56.247 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:55 smithi064 podman[241837]: 2024-03-24 12:16:55.942652467 +0000 UTC m=+0.136945187 container died 561c09f8ef41fd5d9946247d3cca2ce2a51bde0757b8ece3744f062ce3b2a597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5) 2024-03-24T12:16:56.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:56 smithi064 podman[241837]: 2024-03-24 12:16:56.434128496 +0000 UTC m=+0.628421211 container cleanup 561c09f8ef41fd5d9946247d3cca2ce2a51bde0757b8ece3744f062ce3b2a597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T12:16:56.551 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Mar 24 12:16:56 smithi064 bash[241837]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-y 2024-03-24T12:16:56.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T12:16:56.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:56.830 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T12:16:56.830 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T12:16:56.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.x 2024-03-24T12:16:57.168 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 12:16:56 smithi193 systemd[1]: Stopping Ceph mgr.x for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:16:57.509 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 12:16:57 smithi193 podman[166459]: 2024-03-24 12:16:57.169389279 +0000 UTC m=+0.152824209 container died 2e068a588fac7a63c588868dbe3bdf974a8f98e24f194cc5ed690fae4cb95f9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T12:16:57.835 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 12:16:57 smithi193 podman[166459]: 2024-03-24 12:16:57.602326415 +0000 UTC m=+0.585761319 container cleanup 2e068a588fac7a63c588868dbe3bdf974a8f98e24f194cc5ed690fae4cb95f9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T12:16:57.835 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Mar 24 12:16:57 smithi193 bash[166459]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-mgr-x 2024-03-24T12:16:58.070 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T12:16:58.107 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:58.107 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T12:16:58.107 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T12:16:58.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.0 2024-03-24T12:16:58.423 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:16:58 smithi064 systemd[1]: Stopping Ceph osd.0 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:16:58.801 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:16:58 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0[46656]: 2024-03-24T12:16:58.422+0000 7ff3cb995700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:58.801 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:16:58 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0[46656]: 2024-03-24T12:16:58.422+0000 7ff3cb995700 -1 osd.0 540 *** Got signal Terminated *** 2024-03-24T12:16:58.802 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:16:58 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0[46656]: 2024-03-24T12:16:58.422+0000 7ff3cb995700 -1 osd.0 540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:04.289 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:04 smithi064 podman[241967]: 2024-03-24 12:17:04.012909947 +0000 UTC m=+5.715291492 container died 302606f01d1bc502181cc689f9f03cc9904a7b4feb8f4253edae6e3dad52ed36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:04.551 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:04 smithi064 podman[241967]: 2024-03-24 12:17:04.463661176 +0000 UTC m=+6.166042722 container cleanup 302606f01d1bc502181cc689f9f03cc9904a7b4feb8f4253edae6e3dad52ed36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-24T12:17:04.551 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:04 smithi064 bash[241967]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0 2024-03-24T12:17:05.051 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:04 smithi064 podman[241987]: 2024-03-24 12:17:04.790006693 +0000 UTC m=+0.775620102 container remove 302606f01d1bc502181cc689f9f03cc9904a7b4feb8f4253edae6e3dad52ed36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:05.492 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:05 smithi064 podman[242084]: 2024-03-24 12:17:05.042477069 +0000 UTC m=+0.019047672 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:05.492 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:05 smithi064 podman[242084]: 2024-03-24 12:17:05.166177104 +0000 UTC m=+0.142747742 container create 6386fa46d41825d3fa5947621cb401a1f28b0f44f0156474ea3df95c1488e2c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:17:05.492 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:05 smithi064 podman[242084]: 2024-03-24 12:17:05.376636682 +0000 UTC m=+0.353207281 container init 6386fa46d41825d3fa5947621cb401a1f28b0f44f0156474ea3df95c1488e2c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T12:17:05.492 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:05 smithi064 podman[242084]: 2024-03-24 12:17:05.381876188 +0000 UTC m=+0.358446778 container start 6386fa46d41825d3fa5947621cb401a1f28b0f44f0156474ea3df95c1488e2c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T12:17:05.492 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:05 smithi064 podman[242084]: 2024-03-24 12:17:05.493118747 +0000 UTC m=+0.469689341 container attach 6386fa46d41825d3fa5947621cb401a1f28b0f44f0156474ea3df95c1488e2c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T12:17:06.301 INFO:journalctl@ceph.osd.0.smithi064.stdout:Mar 24 12:17:05 smithi064 podman[242084]: 2024-03-24 12:17:05.982095479 +0000 UTC m=+0.958666060 container died 6386fa46d41825d3fa5947621cb401a1f28b0f44f0156474ea3df95c1488e2c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T12:17:06.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T12:17:06.865 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:06.866 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T12:17:06.866 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T12:17:06.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.1 2024-03-24T12:17:07.202 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:06 smithi064 systemd[1]: Stopping Ceph osd.1 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:17:07.551 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1[52921]: 2024-03-24T12:17:07.202+0000 7f3955929700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:07.551 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1[52921]: 2024-03-24T12:17:07.202+0000 7f3955929700 -1 osd.1 540 *** Got signal Terminated *** 2024-03-24T12:17:07.551 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:07 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1[52921]: 2024-03-24T12:17:07.202+0000 7f3955929700 -1 osd.1 540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:13.116 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:12 smithi064 podman[242218]: 2024-03-24 12:17:12.833006314 +0000 UTC m=+5.743044665 container died 19ab11ccc1dc3160fae9b35c621b0d1faa2a45469c88a4aeec75540d6c9223be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:13.551 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:13 smithi064 podman[242218]: 2024-03-24 12:17:13.308098178 +0000 UTC m=+6.218136536 container cleanup 19ab11ccc1dc3160fae9b35c621b0d1faa2a45469c88a4aeec75540d6c9223be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5) 2024-03-24T12:17:13.551 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:13 smithi064 bash[242218]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1 2024-03-24T12:17:13.930 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:13 smithi064 podman[242241]: 2024-03-24 12:17:13.596712738 +0000 UTC m=+0.763273877 container remove 19ab11ccc1dc3160fae9b35c621b0d1faa2a45469c88a4aeec75540d6c9223be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:17:14.195 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:13 smithi064 podman[242336]: 2024-03-24 12:17:13.836031097 +0000 UTC m=+0.019099798 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:14.195 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:13 smithi064 podman[242336]: 2024-03-24 12:17:13.943887146 +0000 UTC m=+0.126955844 container create e4734bb2cdd2b1eba210eae880fe711177964f99e1e644e9ccee7ebdb384c72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T12:17:14.495 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:14 smithi064 podman[242336]: 2024-03-24 12:17:14.196348383 +0000 UTC m=+0.379417093 container init e4734bb2cdd2b1eba210eae880fe711177964f99e1e644e9ccee7ebdb384c72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T12:17:14.496 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:14 smithi064 podman[242336]: 2024-03-24 12:17:14.201725091 +0000 UTC m=+0.384793790 container start e4734bb2cdd2b1eba210eae880fe711177964f99e1e644e9ccee7ebdb384c72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:14.496 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:14 smithi064 podman[242336]: 2024-03-24 12:17:14.287753391 +0000 UTC m=+0.470822104 container attach e4734bb2cdd2b1eba210eae880fe711177964f99e1e644e9ccee7ebdb384c72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T12:17:14.801 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 24 12:17:14 smithi064 podman[242336]: 2024-03-24 12:17:14.497143829 +0000 UTC m=+0.680212538 container died e4734bb2cdd2b1eba210eae880fe711177964f99e1e644e9ccee7ebdb384c72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T12:17:15.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T12:17:15.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:15.268 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T12:17:15.268 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T12:17:15.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.2 2024-03-24T12:17:15.551 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:15 smithi064 systemd[1]: Stopping Ceph osd.2 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:17:16.051 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:15 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2[58928]: 2024-03-24T12:17:15.621+0000 7fdf2be9f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:16.052 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:15 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2[58928]: 2024-03-24T12:17:15.621+0000 7fdf2be9f700 -1 osd.2 540 *** Got signal Terminated *** 2024-03-24T12:17:16.052 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:15 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2[58928]: 2024-03-24T12:17:15.621+0000 7fdf2be9f700 -1 osd.2 540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:21.352 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:21 smithi064 podman[242465]: 2024-03-24 12:17:21.068805555 +0000 UTC m=+5.568551497 container died 186dab23db4df6075116015f86261cf9f87db56d0c6a206d76d0db3a014332a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T12:17:21.801 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:21 smithi064 podman[242465]: 2024-03-24 12:17:21.518926461 +0000 UTC m=+6.018672410 container cleanup 186dab23db4df6075116015f86261cf9f87db56d0c6a206d76d0db3a014332a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T12:17:21.801 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:21 smithi064 bash[242465]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2 2024-03-24T12:17:22.190 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:21 smithi064 podman[242490]: 2024-03-24 12:17:21.837137143 +0000 UTC m=+0.771629183 container remove 186dab23db4df6075116015f86261cf9f87db56d0c6a206d76d0db3a014332a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T12:17:22.444 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:22 smithi064 podman[242586]: 2024-03-24 12:17:22.094065163 +0000 UTC m=+0.017463537 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:22.444 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:22 smithi064 podman[242586]: 2024-03-24 12:17:22.213001934 +0000 UTC m=+0.136400286 container create 4345e1474043d14fad94152cabc3b51cc6aeab7e7e47c5e4d5b81ab0b9798d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-db0330b) 2024-03-24T12:17:22.444 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:22 smithi064 podman[242586]: 2024-03-24 12:17:22.440180479 +0000 UTC m=+0.363578844 container init 4345e1474043d14fad94152cabc3b51cc6aeab7e7e47c5e4d5b81ab0b9798d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-24T12:17:22.444 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:22 smithi064 podman[242586]: 2024-03-24 12:17:22.445248364 +0000 UTC m=+0.368646716 container start 4345e1474043d14fad94152cabc3b51cc6aeab7e7e47c5e4d5b81ab0b9798d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-24T12:17:22.740 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:22 smithi064 podman[242586]: 2024-03-24 12:17:22.548404131 +0000 UTC m=+0.471802491 container attach 4345e1474043d14fad94152cabc3b51cc6aeab7e7e47c5e4d5b81ab0b9798d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T12:17:23.051 INFO:journalctl@ceph.osd.2.smithi064.stdout:Mar 24 12:17:22 smithi064 podman[242586]: 2024-03-24 12:17:22.742267099 +0000 UTC m=+0.665665455 container died 4345e1474043d14fad94152cabc3b51cc6aeab7e7e47c5e4d5b81ab0b9798d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T12:17:23.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T12:17:23.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:23.586 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T12:17:23.587 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T12:17:23.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.3 2024-03-24T12:17:23.801 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:23 smithi064 systemd[1]: Stopping Ceph osd.3 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:17:24.301 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:23 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3[65507]: 2024-03-24T12:17:23.907+0000 7f7763eb8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:24.301 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:23 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3[65507]: 2024-03-24T12:17:23.907+0000 7f7763eb8700 -1 osd.3 540 *** Got signal Terminated *** 2024-03-24T12:17:24.302 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:23 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3[65507]: 2024-03-24T12:17:23.907+0000 7f7763eb8700 -1 osd.3 540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:25.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:25 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:25.564+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:26.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:26 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:26.575+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:27.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:27.555+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:27.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:27 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:27.575+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:28.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:28.555+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:28.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:28.564+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:28.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:28 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:28.433+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:29.763 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:29 smithi064 podman[242715]: 2024-03-24 12:17:29.455893746 +0000 UTC m=+5.680691414 container died 59e29c9466ce954f4d97121002e5861c2d61b673349eab9f9baa6f1d0e441e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T12:17:29.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:29 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:29.531+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:29.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:29 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:29.472+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:29.836 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:29 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:29.549+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:30.051 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:29 smithi064 podman[242715]: 2024-03-24 12:17:29.929677815 +0000 UTC m=+6.154475525 container cleanup 59e29c9466ce954f4d97121002e5861c2d61b673349eab9f9baa6f1d0e441e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:30.051 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:29 smithi064 bash[242715]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3 2024-03-24T12:17:30.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:29 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:29.948+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:30.627 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:30 smithi064 podman[242739]: 2024-03-24 12:17:30.323199809 +0000 UTC m=+0.871135435 container remove 59e29c9466ce954f4d97121002e5861c2d61b673349eab9f9baa6f1d0e441e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T12:17:30.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:30 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:30.472+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:30.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:30 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:30.562+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:30.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:30 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:30.566+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:30.917 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:30 smithi064 podman[242834]: 2024-03-24 12:17:30.532073147 +0000 UTC m=+0.018010596 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:30.917 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:30 smithi064 podman[242834]: 2024-03-24 12:17:30.632151579 +0000 UTC m=+0.118089016 container create 161aee09f2b5142d00a7a81211ae72168d0bb95237332f2a0cc58d945beb7163 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T12:17:30.917 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:30 smithi064 podman[242834]: 2024-03-24 12:17:30.834624774 +0000 UTC m=+0.320562219 container init 161aee09f2b5142d00a7a81211ae72168d0bb95237332f2a0cc58d945beb7163 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T12:17:30.917 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:30 smithi064 podman[242834]: 2024-03-24 12:17:30.840239687 +0000 UTC m=+0.326177123 container start 161aee09f2b5142d00a7a81211ae72168d0bb95237332f2a0cc58d945beb7163 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:30.917 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:30 smithi064 podman[242834]: 2024-03-24 12:17:30.917697639 +0000 UTC m=+0.403635075 container attach 161aee09f2b5142d00a7a81211ae72168d0bb95237332f2a0cc58d945beb7163 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T12:17:31.301 INFO:journalctl@ceph.osd.3.smithi064.stdout:Mar 24 12:17:31 smithi064 podman[242834]: 2024-03-24 12:17:31.133429633 +0000 UTC m=+0.619367067 container died 161aee09f2b5142d00a7a81211ae72168d0bb95237332f2a0cc58d945beb7163 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T12:17:31.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:30 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:30.985+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:31.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T12:17:31.800 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:31.800 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T12:17:31.800 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T12:17:31.800 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.4 2024-03-24T12:17:31.832 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:31 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:31.543+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:31.832 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:31 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:31.595+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:31.832 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:31 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:31.500+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:32.085 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:31 smithi193 systemd[1]: Stopping Ceph osd.4 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:17:32.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:31 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:31.997+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:32.523 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:32 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:32.157+0000 7fc418bec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:32.524 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:32 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:32.157+0000 7fc418bec700 -1 osd.4 540 *** Got signal Terminated *** 2024-03-24T12:17:32.524 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:32 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:32.157+0000 7fc418bec700 -1 osd.4 540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:32.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:32 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:32.525+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:32.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:32 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:32.614+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:32.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:32 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:32.524+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:33.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:33 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:33.004+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:33.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:33 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:33.581+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:33.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:33 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:33.476+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:33.836 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:33 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:33.479+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:33.836 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:33 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:33.479+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.530954+0000 front 2024-03-24T12:17:08.530915+0000 (oldest deadline 2024-03-24T12:17:33.230950+0000) 2024-03-24T12:17:34.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:34.018+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:34.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:34.630+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:34.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:34.441+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:34.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:34.441+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.530954+0000 front 2024-03-24T12:17:08.530915+0000 (oldest deadline 2024-03-24T12:17:33.230950+0000) 2024-03-24T12:17:34.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:34.488+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:35.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:34 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:34.976+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:35.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:35 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:35.585+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:35.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:35 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:35.452+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:35.836 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:35 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:35.465+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:35.836 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:35 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:35.465+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.530954+0000 front 2024-03-24T12:17:08.530915+0000 (oldest deadline 2024-03-24T12:17:33.230950+0000) 2024-03-24T12:17:36.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:36.005+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:36.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:36.494+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.829672+0000 front 2024-03-24T12:17:02.829795+0000 (oldest deadline 2024-03-24T12:17:24.529620+0000) 2024-03-24T12:17:36.835 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4[38997]: 2024-03-24T12:17:36.494+0000 7fc4111d8700 -1 osd.4 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.530954+0000 front 2024-03-24T12:17:08.530915+0000 (oldest deadline 2024-03-24T12:17:33.230950+0000) 2024-03-24T12:17:36.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:36.402+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:36.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:36.570+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:36.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:36 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:36.570+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:37.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:37.045+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:37.647 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:37.371+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:37.647 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:37.583+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:37.647 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:37 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:37.583+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:37.922 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:37 smithi193 podman[166604]: 2024-03-24 12:17:37.648353265 +0000 UTC m=+5.640183784 container died 7380a6dafb12d2d836be33623899438935e29f61245b2faa1ee393790d1d24d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:38.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:38.064+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:38.335 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:38 smithi193 podman[166604]: 2024-03-24 12:17:38.121024618 +0000 UTC m=+6.112855138 container cleanup 7380a6dafb12d2d836be33623899438935e29f61245b2faa1ee393790d1d24d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-24T12:17:38.335 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:38 smithi193 bash[166604]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4 2024-03-24T12:17:38.714 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:38 smithi193 podman[166627]: 2024-03-24 12:17:38.381213478 +0000 UTC m=+0.730204233 container remove 7380a6dafb12d2d836be33623899438935e29f61245b2faa1ee393790d1d24d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T12:17:38.714 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:38.590+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:38.714 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:38.590+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:38.714 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:38.350+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:38.715 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:38 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:38.350+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:39.033 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:38 smithi193 podman[166718]: 2024-03-24 12:17:38.71540356 +0000 UTC m=+0.104859190 container create 7efa22b5f9e44e8ae621703e5a1d6fb44cff325f41f0b4a972eb55cda3a83319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-db0330b) 2024-03-24T12:17:39.033 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:38 smithi193 podman[166718]: 2024-03-24 12:17:38.629517325 +0000 UTC m=+0.018972909 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:39.033 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:38 smithi193 podman[166718]: 2024-03-24 12:17:38.926134983 +0000 UTC m=+0.315590567 container init 7efa22b5f9e44e8ae621703e5a1d6fb44cff325f41f0b4a972eb55cda3a83319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T12:17:39.033 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:38 smithi193 podman[166718]: 2024-03-24 12:17:38.931859587 +0000 UTC m=+0.321315174 container start 7efa22b5f9e44e8ae621703e5a1d6fb44cff325f41f0b4a972eb55cda3a83319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:17:39.033 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:39 smithi193 podman[166718]: 2024-03-24 12:17:39.034395365 +0000 UTC m=+0.423851012 container attach 7efa22b5f9e44e8ae621703e5a1d6fb44cff325f41f0b4a972eb55cda3a83319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T12:17:39.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:39.034+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:39.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:39.034+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:39.336 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:17:39.088Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-03-24T12:17:39.336 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:17:39.088Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-03-24T12:17:39.336 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:17:39.091Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-03-24T12:17:39.336 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:17:39.092Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-03-24T12:17:39.627 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:39.369+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:39.627 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:39.369+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:39.991 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 24 12:17:39 smithi193 podman[166718]: 2024-03-24 12:17:39.794847577 +0000 UTC m=+1.184303173 container died 7efa22b5f9e44e8ae621703e5a1d6fb44cff325f41f0b4a972eb55cda3a83319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:39.991 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:39.628+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:39.991 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:39.628+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:40.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:39.992+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:40.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:39 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:39.992+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:40.521 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T12:17:40.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:40.603 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T12:17:40.603 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T12:17:40.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.5 2024-03-24T12:17:40.666 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:40.401+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:40.666 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:40.401+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:40.968 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:40.676+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:40.968 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:40.676+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:40.968 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:40 smithi193 systemd[1]: Stopping Ceph osd.5 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:17:40.968 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:40.969+0000 7f2d01d02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:40.968 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:40.969+0000 7f2d01d02700 -1 osd.5 540 *** Got signal Terminated *** 2024-03-24T12:17:40.969 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:40 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:40.969+0000 7f2d01d02700 -1 osd.5 540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:41.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:41.030+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:41.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:41.030+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:41.696 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:41.423+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:41.696 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:41.423+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:41.995 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:41.698+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:41.995 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:41.698+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:42.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:41.996+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:42.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:41.996+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:42.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:41 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:41.996+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:42.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:42.470+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:42.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:42.470+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:42.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:42.704+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:42.835 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:42.704+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:43.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:42.953+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:43.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:42.953+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:43.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:42 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:42.953+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:43.750 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:43.432+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:43.750 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:43.432+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:44.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:43.750+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:44.086 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:43.750+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:44.086 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:43.750+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:17.041946+0000 front 2024-03-24T12:17:17.041845+0000 (oldest deadline 2024-03-24T12:17:42.941702+0000) 2024-03-24T12:17:44.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:43.906+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:44.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:43.906+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:44.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:43 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:43.906+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:44.747 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:44.428+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:44.747 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:44.428+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:44.747 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:44.428+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:45.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:44.747+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:45.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:44.747+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:45.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:44.747+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:17.041946+0000 front 2024-03-24T12:17:17.041845+0000 (oldest deadline 2024-03-24T12:17:42.941702+0000) 2024-03-24T12:17:45.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:44.947+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:45.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:44.947+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:45.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:44 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:44.947+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:45.755 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:45.436+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:45.756 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:45.436+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:45.756 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:45.436+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:46.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:45.755+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.439663+0000 front 2024-03-24T12:17:02.439753+0000 (oldest deadline 2024-03-24T12:17:27.139628+0000) 2024-03-24T12:17:46.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:45.755+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:10.640668+0000 front 2024-03-24T12:17:10.640768+0000 (oldest deadline 2024-03-24T12:17:35.940747+0000) 2024-03-24T12:17:46.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5[135259]: 2024-03-24T12:17:45.755+0000 7f2cfe2f6700 -1 osd.5 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:17.041946+0000 front 2024-03-24T12:17:17.041845+0000 (oldest deadline 2024-03-24T12:17:42.941702+0000) 2024-03-24T12:17:46.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:45.905+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:46.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:45.905+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:46.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:45 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:45.905+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:46.826 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:46 smithi193 podman[166849]: 2024-03-24 12:17:46.535141867 +0000 UTC m=+5.705112213 container died f0dfecc84f55207dcc31a0efda53523fa109a691d44617bedd4025ea0457f7ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T12:17:46.827 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:46.468+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:46.827 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:46.468+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:46.827 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:46.468+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:47.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:46 smithi193 podman[166849]: 2024-03-24 12:17:46.999845626 +0000 UTC m=+6.169816020 container cleanup f0dfecc84f55207dcc31a0efda53523fa109a691d44617bedd4025ea0457f7ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T12:17:47.085 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:47 smithi193 bash[166849]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5 2024-03-24T12:17:47.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:46.875+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:47.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:46.875+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:47.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:46 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:46.875+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:47.692 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:47 smithi193 podman[166871]: 2024-03-24 12:17:47.368252693 +0000 UTC m=+0.827993715 container remove f0dfecc84f55207dcc31a0efda53523fa109a691d44617bedd4025ea0457f7ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:47.692 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:47.502+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:47.692 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:47.502+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:47.693 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:47.502+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:47.954 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:47 smithi193 podman[166964]: 2024-03-24 12:17:47.596824812 +0000 UTC m=+0.018538006 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:47.954 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:47 smithi193 podman[166964]: 2024-03-24 12:17:47.69404793 +0000 UTC m=+0.115761123 container create 06fddd791d9013d80e35ab61b72df744cd962ea42eef086a95a76220c83df11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b) 2024-03-24T12:17:47.954 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:47.869+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:47.954 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:47.869+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:47.954 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:47 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:47.869+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:48.249 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:47 smithi193 podman[166964]: 2024-03-24 12:17:47.95553773 +0000 UTC m=+0.377250916 container init 06fddd791d9013d80e35ab61b72df744cd962ea42eef086a95a76220c83df11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:48.249 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:47 smithi193 podman[166964]: 2024-03-24 12:17:47.961206959 +0000 UTC m=+0.382920154 container start 06fddd791d9013d80e35ab61b72df744cd962ea42eef086a95a76220c83df11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T12:17:48.249 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:48 smithi193 podman[166964]: 2024-03-24 12:17:48.046936802 +0000 UTC m=+0.468650000 container attach 06fddd791d9013d80e35ab61b72df744cd962ea42eef086a95a76220c83df11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T12:17:48.544 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:48 smithi193 podman[166964]: 2024-03-24 12:17:48.250737214 +0000 UTC m=+0.672450407 container died 06fddd791d9013d80e35ab61b72df744cd962ea42eef086a95a76220c83df11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T12:17:48.794 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:48 smithi193 podman[167010]: 2024-03-24 12:17:48.775174505 +0000 UTC m=+0.519542767 container remove 06fddd791d9013d80e35ab61b72df744cd962ea42eef086a95a76220c83df11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T12:17:48.794 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 12:17:48 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-24T12:17:48.794 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:48.544+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:48.794 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:48.544+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:48.794 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:48.544+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:48.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T12:17:48.878 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:48.878 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T12:17:48.879 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T12:17:48.879 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.6 2024-03-24T12:17:49.085 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:48 smithi193 systemd[1]: Stopping Ceph osd.6 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:17:49.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:48.824+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:49.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:48.824+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:49.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:48 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:48.824+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:49.572 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:49.204+0000 7f9f98ae3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:49.572 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:49.204+0000 7f9f98ae3700 -1 osd.6 540 *** Got signal Terminated *** 2024-03-24T12:17:49.572 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:49.204+0000 7f9f98ae3700 -1 osd.6 540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:49.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:49.572+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:49.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:49.572+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:49.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:49.572+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:49.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:49.794+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:49.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:49.794+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:49.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:49 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:49.794+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:50.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:50.558+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:50.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:50.558+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:50.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:50.558+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:50.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:50.805+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:50.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:50.805+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:50.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:50 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:50.805+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:51.774 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:51.509+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:51.774 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:51.509+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:51.774 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:51.509+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:52.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:51.774+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:52.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:51.774+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:52.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:51 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:51.774+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:52.771 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:52.505+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:52.771 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:52.505+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:52.771 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:52.505+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:52.771 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:52.505+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:25.800765+0000 front 2024-03-24T12:17:25.800821+0000 (oldest deadline 2024-03-24T12:17:51.700755+0000) 2024-03-24T12:17:53.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:52.771+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:53.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:52.771+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:53.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:52 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:52.771+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:53.793 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:53.474+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:17:02.298121+0000 front 2024-03-24T12:17:02.298192+0000 (oldest deadline 2024-03-24T12:17:28.198172+0000) 2024-03-24T12:17:53.793 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:53.474+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:08.198621+0000 front 2024-03-24T12:17:08.198650+0000 (oldest deadline 2024-03-24T12:17:38.199203+0000) 2024-03-24T12:17:53.793 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:53.474+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.199767+0000 front 2024-03-24T12:17:18.199867+0000 (oldest deadline 2024-03-24T12:17:44.099719+0000) 2024-03-24T12:17:53.793 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6[51539]: 2024-03-24T12:17:53.474+0000 7f9f910cf700 -1 osd.6 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:25.800765+0000 front 2024-03-24T12:17:25.800821+0000 (oldest deadline 2024-03-24T12:17:51.700755+0000) 2024-03-24T12:17:54.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:53.793+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:54.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:53.793+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:54.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:53.793+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:54.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:53 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:53.793+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:17:55.085 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:54 smithi193 podman[167095]: 2024-03-24 12:17:54.811496889 +0000 UTC m=+5.704899603 container died 3d003493051dcbd2b22bfd713903f79018f4984d85b704465e7692451bbf3f9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:55.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:54.759+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:55.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:54.759+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:55.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:54.759+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:55.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:54 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:54.759+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:17:55.585 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:55 smithi193 podman[167095]: 2024-03-24 12:17:55.303015875 +0000 UTC m=+6.196418599 container cleanup 3d003493051dcbd2b22bfd713903f79018f4984d85b704465e7692451bbf3f9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T12:17:55.585 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:55 smithi193 bash[167095]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6 2024-03-24T12:17:55.962 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:55 smithi193 podman[167117]: 2024-03-24 12:17:55.604403963 +0000 UTC m=+0.768496182 container remove 3d003493051dcbd2b22bfd713903f79018f4984d85b704465e7692451bbf3f9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T12:17:55.962 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:55 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:55.712+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:55.963 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:55 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:55.712+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:55.963 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:55 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:55.712+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:55.963 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:55 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:55.712+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:17:56.213 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:55 smithi193 podman[167212]: 2024-03-24 12:17:55.866403134 +0000 UTC m=+0.018584864 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:56.213 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:55 smithi193 podman[167212]: 2024-03-24 12:17:55.984747445 +0000 UTC m=+0.136929198 container create 6771dfb0d85f15308c66a6d087da5583292292b3bb06368aee916ab72a007078 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:56.213 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:56 smithi193 podman[167212]: 2024-03-24 12:17:56.208852835 +0000 UTC m=+0.361034547 container init 6771dfb0d85f15308c66a6d087da5583292292b3bb06368aee916ab72a007078 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T12:17:56.213 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:56 smithi193 podman[167212]: 2024-03-24 12:17:56.214575846 +0000 UTC m=+0.366757558 container start 6771dfb0d85f15308c66a6d087da5583292292b3bb06368aee916ab72a007078 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-24T12:17:56.495 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:56 smithi193 podman[167212]: 2024-03-24 12:17:56.282657867 +0000 UTC m=+0.434839592 container attach 6771dfb0d85f15308c66a6d087da5583292292b3bb06368aee916ab72a007078 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True) 2024-03-24T12:17:56.792 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 24 12:17:56 smithi193 podman[167212]: 2024-03-24 12:17:56.497701617 +0000 UTC m=+0.649883329 container died 6771dfb0d85f15308c66a6d087da5583292292b3bb06368aee916ab72a007078 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T12:17:56.793 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:56.684+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:56.793 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:56.684+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:56.793 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:56.684+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:56.793 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:56 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:56.684+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:17:57.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T12:17:57.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:57.113 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T12:17:57.113 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T12:17:57.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.7 2024-03-24T12:17:57.334 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:57 smithi193 systemd[1]: Stopping Ceph osd.7 for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:17:57.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:57.466+0000 7fda872be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:57.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:57.466+0000 7fda872be700 -1 osd.7 540 *** Got signal Terminated *** 2024-03-24T12:17:57.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:57.466+0000 7fda872be700 -1 osd.7 540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:58.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:57.652+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:58.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:57.652+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:58.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:57.652+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:58.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:57 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:57.652+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:17:59.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:58 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:58.632+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:17:59.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:58 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:58.632+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:17:59.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:58 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:58.632+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:17:59.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:58 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:58.632+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:18:00.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:59 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:59.622+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:18:00.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:59 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:59.622+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:18:00.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:59 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:59.622+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:18:00.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:17:59 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:17:59.622+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:18:01.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:00 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:18:00.611+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:18:01.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:00 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:18:00.611+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:18:01.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:00 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:18:00.611+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:18:01.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:00 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:18:00.611+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:18:02.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:18:01.596+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6803 osd.0 since back 2024-03-24T12:16:58.770643+0000 front 2024-03-24T12:16:58.770580+0000 (oldest deadline 2024-03-24T12:17:29.371018+0000) 2024-03-24T12:18:02.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:18:01.596+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6807 osd.1 since back 2024-03-24T12:17:09.371490+0000 front 2024-03-24T12:17:09.371598+0000 (oldest deadline 2024-03-24T12:17:38.171903+0000) 2024-03-24T12:18:02.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:18:01.596+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6811 osd.2 since back 2024-03-24T12:17:18.172335+0000 front 2024-03-24T12:17:18.172402+0000 (oldest deadline 2024-03-24T12:17:41.072265+0000) 2024-03-24T12:18:02.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:01 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7[57960]: 2024-03-24T12:18:01.596+0000 7fda838b2700 -1 osd.7 540 heartbeat_check: no reply from 172.21.15.64:6815 osd.3 since back 2024-03-24T12:17:27.974198+0000 front 2024-03-24T12:17:27.974166+0000 (oldest deadline 2024-03-24T12:17:53.274127+0000) 2024-03-24T12:18:03.239 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:02 smithi193 podman[167344]: 2024-03-24 12:18:02.966415443 +0000 UTC m=+5.626566618 container died 65ae4fe944f30252f69be9626782ba0ace3325d9127777515c5b513ab64cb917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T12:18:03.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:03 smithi193 podman[167344]: 2024-03-24 12:18:03.41400767 +0000 UTC m=+6.074158888 container cleanup 65ae4fe944f30252f69be9626782ba0ace3325d9127777515c5b513ab64cb917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T12:18:03.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:03 smithi193 bash[167344]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7 2024-03-24T12:18:04.031 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:03 smithi193 podman[167380]: 2024-03-24 12:18:03.698629841 +0000 UTC m=+0.728823016 container remove 65ae4fe944f30252f69be9626782ba0ace3325d9127777515c5b513ab64cb917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T12:18:04.325 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:04 smithi193 podman[167476]: 2024-03-24 12:18:04.0328198 +0000 UTC m=+0.113263208 container create db3f7206ae8b450b6b618dc60ed7a4b27391c9b4ea14b593bc5eafa712dfd71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T12:18:04.325 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:04 smithi193 podman[167476]: 2024-03-24 12:18:03.937531705 +0000 UTC m=+0.017975074 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:18:04.325 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:04 smithi193 podman[167476]: 2024-03-24 12:18:04.235114708 +0000 UTC m=+0.315558080 container init db3f7206ae8b450b6b618dc60ed7a4b27391c9b4ea14b593bc5eafa712dfd71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:18:04.325 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:04 smithi193 podman[167476]: 2024-03-24 12:18:04.241272097 +0000 UTC m=+0.321715459 container start db3f7206ae8b450b6b618dc60ed7a4b27391c9b4ea14b593bc5eafa712dfd71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T12:18:04.326 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:04 smithi193 podman[167476]: 2024-03-24 12:18:04.326916999 +0000 UTC m=+0.407360373 container attach db3f7206ae8b450b6b618dc60ed7a4b27391c9b4ea14b593bc5eafa712dfd71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T12:18:04.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 24 12:18:04 smithi193 podman[167476]: 2024-03-24 12:18:04.540846275 +0000 UTC m=+0.621289625 container died db3f7206ae8b450b6b618dc60ed7a4b27391c9b4ea14b593bc5eafa712dfd71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:18:05.125 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T12:18:05.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:18:05.167 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T12:18:05.167 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-24T12:18:05.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-24T12:18:05.489 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 12:18:05 smithi064 systemd[1]: Stopping Ceph rgw.foo.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:18:05.801 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 12:18:05 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a[70597]: 2024-03-24T12:18:05.489+0000 7f6016843700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:18:05.801 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 12:18:05 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a[70597]: 2024-03-24T12:18:05.489+0000 7f60250b3ac0 -1 shutting down 2024-03-24T12:18:15.800 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 12:18:15 smithi064 bash[242965]: time="2024-03-24T12:18:15Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-24T12:18:15.800 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 12:18:15 smithi064 podman[242965]: 2024-03-24 12:18:15.512449966 +0000 UTC m=+10.148442814 container died a18f861b4e00de151e1f3e658101d5414e4ede25b6f542b1403dbf18e0ca195b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:18:16.051 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 12:18:15 smithi064 podman[242965]: 2024-03-24 12:18:15.976836234 +0000 UTC m=+10.612829081 container cleanup a18f861b4e00de151e1f3e658101d5414e4ede25b6f542b1403dbf18e0ca195b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:18:16.051 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Mar 24 12:18:15 smithi064 bash[242965]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-24T12:18:16.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-24T12:18:16.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:18:16.413 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-24T12:18:16.413 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-24T12:18:16.414 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-06f23456-e9d4-11ee-95cd-87774f69a715@prometheus.a 2024-03-24T12:18:16.749 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 systemd[1]: Stopping Ceph prometheus.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.750Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.751Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.751Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.751Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.751Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.751Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.751Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.751Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T12:18:17.040 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.762Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T12:18:17.041 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.762Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T12:18:17.041 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.762Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T12:18:17.041 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a[70387]: ts=2024-03-24T12:18:16.762Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T12:18:17.041 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:16 smithi193 podman[167605]: 2024-03-24 12:18:16.771708247 +0000 UTC m=+0.125001286 container died 01e74f4670046fc440ec4da06892b58b9ec3b208214e13a0328060135425a389 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:18:17.335 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:17 smithi193 podman[167605]: 2024-03-24 12:18:17.122904857 +0000 UTC m=+0.476197897 container cleanup 01e74f4670046fc440ec4da06892b58b9ec3b208214e13a0328060135425a389 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:18:17.335 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Mar 24 12:18:17 smithi193 bash[167605]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T12:18:17.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-06f23456-e9d4-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-24T12:18:17.580 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:18:17.580 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-24T12:18:17.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T12:18:17.777 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T12:18:20.051 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:19 smithi064 systemd[1]: Stopping Ceph alertmanager.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:18:20.551 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:20 smithi064 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a[86482]: ts=2024-03-24T12:18:20.226Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T12:18:20.551 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:20 smithi064 podman[243475]: 2024-03-24 12:18:20.267294232 +0000 UTC m=+0.148648311 container died 9864e48d66ff759409e7aadc63e6e1d24752f545b830d3f82a75e2ee65bcf817 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:18:21.006 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:20 smithi064 podman[243475]: 2024-03-24 12:18:20.74883218 +0000 UTC m=+0.630186263 container cleanup 9864e48d66ff759409e7aadc63e6e1d24752f545b830d3f82a75e2ee65bcf817 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:18:21.006 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:20 smithi064 bash[243475]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T12:18:21.301 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:21 smithi064 podman[243494]: 2024-03-24 12:18:21.008227643 +0000 UTC m=+0.734404579 container remove 9864e48d66ff759409e7aadc63e6e1d24752f545b830d3f82a75e2ee65bcf817 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:18:21.301 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:21 smithi064 podman[243494]: 2024-03-24 12:18:21.084534803 +0000 UTC m=+0.810711741 volume remove d83ecb71e1e373788c59d31a059cfbcaeac710b6abbb09b82e6a83db31e8d4f7 2024-03-24T12:18:21.301 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:21 smithi064 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T12:18:21.301 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:21 smithi064 systemd[1]: Stopped Ceph alertmanager.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T12:18:21.301 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Mar 24 12:18:21 smithi064 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.807s CPU time. 2024-03-24T12:18:21.801 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:21 smithi064 systemd[1]: Stopping Ceph node-exporter.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:18:21.802 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:21 smithi064 podman[243611]: 2024-03-24 12:18:21.683212174 +0000 UTC m=+0.106708185 container died c7cbb8fc5d4b7512ed65850bc9a27a13fa8eb6f14bc5295e83f4b92a06f20ba5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:18:22.301 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:22 smithi064 podman[243611]: 2024-03-24 12:18:22.128234923 +0000 UTC m=+0.551730934 container cleanup c7cbb8fc5d4b7512ed65850bc9a27a13fa8eb6f14bc5295e83f4b92a06f20ba5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:18:22.301 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:22 smithi064 bash[243611]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T12:18:22.698 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:22 smithi064 podman[243630]: 2024-03-24 12:18:22.396080078 +0000 UTC m=+0.717445898 container remove c7cbb8fc5d4b7512ed65850bc9a27a13fa8eb6f14bc5295e83f4b92a06f20ba5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:18:22.699 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:22 smithi064 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:18:22.699 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:22 smithi064 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T12:18:22.699 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:22 smithi064 systemd[1]: Stopped Ceph node-exporter.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T12:18:22.699 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Mar 24 12:18:22 smithi064 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 5.002s CPU time. 2024-03-24T12:18:23.400 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T12:18:23.591 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T12:18:26.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:25 smithi193 systemd[1]: Stopping Ceph iscsi.iscsi.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:18:26.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:25 smithi193 ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a[62895]: debug Shutdown received 2024-03-24T12:18:36.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:35 smithi193 bash[168026]: time="2024-03-24T12:18:35Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-24T12:18:36.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:35 smithi193 podman[168026]: 2024-03-24 12:18:35.885631506 +0000 UTC m=+10.127419291 container died 16a9c3f9bafb55e64fc964ee88b2122a85d3345d49face013590a19e4bfdf202 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T12:18:36.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:36 smithi193 podman[168026]: 2024-03-24 12:18:36.267471764 +0000 UTC m=+10.509259549 container cleanup 16a9c3f9bafb55e64fc964ee88b2122a85d3345d49face013590a19e4bfdf202 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-24T12:18:36.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:36 smithi193 bash[168026]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a 2024-03-24T12:18:36.446 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:36 smithi193 podman[168062]: 2024-03-24 12:18:36.443621421 +0000 UTC m=+0.555445969 container remove 16a9c3f9bafb55e64fc964ee88b2122a85d3345d49face013590a19e4bfdf202 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T12:18:36.446 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:36 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T12:18:36.703 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:36 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-24T12:18:36.703 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:36 smithi193 systemd[1]: Stopped Ceph iscsi.iscsi.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T12:18:36.703 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Mar 24 12:18:36 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Consumed 3.610s CPU time. 2024-03-24T12:18:38.335 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:38 smithi193 systemd[1]: Stopping Ceph grafana.a for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:18:38.812 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:38 smithi193 podman[168342]: 2024-03-24 12:18:38.491555568 +0000 UTC m=+0.121159428 container died ccaac67d7ff6bb7226d9a3c5515ff4d8a7f695f1e1968872399fd5c648fccd2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, version=8.5, description=Ceph Grafana Container, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-24T12:18:39.085 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:38 smithi193 podman[168342]: 2024-03-24 12:18:38.81413496 +0000 UTC m=+0.443738818 container cleanup ccaac67d7ff6bb7226d9a3c5515ff4d8a7f695f1e1968872399fd5c648fccd2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, vcs-type=git, io.buildah.version=1.32.0, architecture=x86_64, version=8.5, description=Ceph Grafana Container, release=239.1651231664, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, name=ubi8, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T12:18:39.085 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:38 smithi193 bash[168342]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a 2024-03-24T12:18:39.085 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:39 smithi193 podman[168359]: 2024-03-24 12:18:39.0099873 +0000 UTC m=+0.520090754 container remove ccaac67d7ff6bb7226d9a3c5515ff4d8a7f695f1e1968872399fd5c648fccd2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, distribution-scope=public, version=8.5, name=ubi8, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , release=239.1651231664, vendor=Red Hat, Inc.) 2024-03-24T12:18:39.085 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:39 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:18:39.585 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:39 smithi193 systemd[1]: Stopping Ceph node-exporter.b for 06f23456-e9d4-11ee-95cd-87774f69a715... 2024-03-24T12:18:39.586 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:39 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-24T12:18:39.586 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:39 smithi193 systemd[1]: Stopped Ceph grafana.a for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T12:18:39.586 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Mar 24 12:18:39 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@grafana.a.service: Consumed 13.411s CPU time. 2024-03-24T12:18:39.945 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:39 smithi193 podman[168477]: 2024-03-24 12:18:39.693059734 +0000 UTC m=+0.137323328 container died c4156adfae44ce2bba092e3ad6a3e60118733451d274fe444fbd208dabeff47a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:18:40.329 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:40 smithi193 podman[168477]: 2024-03-24 12:18:40.03761792 +0000 UTC m=+0.481881522 container cleanup c4156adfae44ce2bba092e3ad6a3e60118733451d274fe444fbd208dabeff47a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:18:40.329 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:40 smithi193 bash[168477]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b 2024-03-24T12:18:40.585 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:40 smithi193 podman[168496]: 2024-03-24 12:18:40.330571154 +0000 UTC m=+0.638601756 container remove c4156adfae44ce2bba092e3ad6a3e60118733451d274fe444fbd208dabeff47a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-06f23456-e9d4-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:18:40.585 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:40 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:18:40.585 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:40 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-24T12:18:40.585 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:40 smithi193 systemd[1]: Stopped Ceph node-exporter.b for 06f23456-e9d4-11ee-95cd-87774f69a715. 2024-03-24T12:18:40.585 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Mar 24 12:18:40 smithi193 systemd[1]: ceph-06f23456-e9d4-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 4.688s CPU time. 2024-03-24T12:18:41.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:18:41.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:18:41.225 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T12:18:41.226 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942/remote/smithi064/crash 2024-03-24T12:18:41.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/crash -- . 2024-03-24T12:18:41.290 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T12:18:41.290 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-03-24T12:18:41.292 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942/remote/smithi193/crash 2024-03-24T12:18:41.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/crash -- . 2024-03-24T12:18:41.338 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T12:18:41.338 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-03-24T12:18:41.340 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T12:18:41.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T12:18:41.443 INFO:teuthology.orchestra.run.smithi064.stdout:2024-03-24T12:00:00.000113+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-24T12:18:41.444 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T12:18:41.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[SEC\]' /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T12:18:41.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]' /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T12:18:41.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[WRN\]' /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T12:18:41.618 INFO:teuthology.orchestra.run.smithi064.stdout:2024-03-24T12:00:00.000113+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-24T12:18:41.618 INFO:tasks.cephadm:Compressing logs... 2024-03-24T12:18:41.618 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T12:18:41.660 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T12:18:41.707 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T12:18:41.707 INFO:teuthology.orchestra.run.smithi193.stderr:: No such file or directory 2024-03-24T12:18:41.713 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-03-24T12:18:41.713 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-24T12:18:41.717 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T12:18:41.717 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/objectstore_tool.6.log 2024-03-24T12:18:41.717 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T12:18:41.717 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T12:18:41.718 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/objectstore_tool.7.log 2024-03-24T12:18:41.718 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T12:18:41.719 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T12:18:41.719 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T12:18:41.720 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.5.log: -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-24T12:18:41.720 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T12:18:41.720 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log 2024-03-24T12:18:41.720 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T12:18:41.720 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T12:18:41.725 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T12:18:41.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T12:18:41.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-24T12:18:41.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T12:18:41.727 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T12:18:41.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: 57.6% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-24T12:18:41.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T12:18:41.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T12:18:41.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T12:18:41.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log 2024-03-24T12:18:41.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T12:18:41.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T12:18:41.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T12:18:41.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.5% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T12:18:41.764 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.b.log: 68.2% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-24T12:18:41.831 INFO:teuthology.orchestra.run.smithi064.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T12:18:41.839 INFO:teuthology.orchestra.run.smithi193.stderr: 94.5% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/objectstore_tool.6.log.gz 2024-03-24T12:18:41.845 INFO:teuthology.orchestra.run.smithi193.stderr: 81.8% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T12:18:41.945 INFO:teuthology.orchestra.run.smithi193.stderr: 84.8% 90.0% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T12:18:41.945 INFO:teuthology.orchestra.run.smithi193.stderr: -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T12:18:41.956 INFO:teuthology.orchestra.run.smithi064.stderr: 89.1% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T12:18:41.968 INFO:teuthology.orchestra.run.smithi064.stderr: 91.2% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T12:18:41.988 INFO:teuthology.orchestra.run.smithi193.stderr: 90.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T12:18:42.095 INFO:teuthology.orchestra.run.smithi193.stderr: 92.0% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T12:18:42.135 INFO:teuthology.orchestra.run.smithi193.stderr: 91.1% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T12:18:42.239 INFO:teuthology.orchestra.run.smithi193.stderr: 94.0% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/objectstore_tool.7.log.gz 2024-03-24T12:18:42.869 INFO:teuthology.orchestra.run.smithi064.stderr: 94.0% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T12:18:44.355 INFO:teuthology.orchestra.run.smithi064.stderr: 89.1% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T12:18:44.670 INFO:teuthology.orchestra.run.smithi193.stderr: 90.6% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T12:18:47.679 INFO:teuthology.orchestra.run.smithi064.stderr: 90.8% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T12:19:14.311 INFO:teuthology.orchestra.run.smithi064.stderr: 90.0% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T12:25:23.683 INFO:teuthology.orchestra.run.smithi064.stderr: 93.9% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T12:26:47.373 INFO:teuthology.orchestra.run.smithi193.stderr: 93.6% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T12:29:08.777 INFO:teuthology.orchestra.run.smithi064.stderr: 93.3% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T12:29:35.615 INFO:teuthology.orchestra.run.smithi064.stderr: 93.6% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T12:29:54.955 INFO:teuthology.orchestra.run.smithi193.stderr: 93.3% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T12:30:15.156 INFO:teuthology.orchestra.run.smithi064.stderr: 93.4% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T12:30:15.159 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-24T12:30:15.159 INFO:teuthology.orchestra.run.smithi064.stderr:real 11m33.489s 2024-03-24T12:30:15.159 INFO:teuthology.orchestra.run.smithi064.stderr:user 6m11.515s 2024-03-24T12:30:15.159 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m17.239s 2024-03-24T12:30:50.604 INFO:teuthology.orchestra.run.smithi193.stderr: 93.7% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T12:31:16.658 INFO:teuthology.orchestra.run.smithi193.stderr: 93.5% -- replaced with /var/log/ceph/06f23456-e9d4-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T12:31:16.660 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-24T12:31:16.660 INFO:teuthology.orchestra.run.smithi193.stderr:real 12m34.987s 2024-03-24T12:31:16.660 INFO:teuthology.orchestra.run.smithi193.stderr:user 5m29.453s 2024-03-24T12:31:16.660 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m16.695s 2024-03-24T12:31:16.661 INFO:tasks.cephadm:Archiving logs... 2024-03-24T12:31:16.661 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942/remote/smithi064/log 2024-03-24T12:31:16.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T12:32:09.275 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942/remote/smithi193/log 2024-03-24T12:32:09.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T12:32:57.973 INFO:tasks.cephadm:Removing cluster... 2024-03-24T12:32:57.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 --force 2024-03-24T12:32:58.217 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T12:32:58.889 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 06f23456-e9d4-11ee-95cd-87774f69a715 --force 2024-03-24T12:32:59.111 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: 06f23456-e9d4-11ee-95cd-87774f69a715 2024-03-24T12:32:59.774 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T12:32:59.774 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T12:32:59.793 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T12:32:59.814 INFO:tasks.cephadm:Teardown complete 2024-03-24T12:32:59.815 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T12:32:59.825 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T12:32:59.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T12:32:59.836 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T12:32:59.938 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T12:32:59.946 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T12:32:59.947 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:32:59.950 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:32:59.964 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:32:59.989 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:32:59.998 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:32:59.998 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-24T12:32:59.998 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +866us[ +856us] +/- 82ms 2024-03-24T12:32:59.998 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -22us[ -32us] +/- 62ms 2024-03-24T12:32:59.998 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -921us[ -931us] +/- 86ms 2024-03-24T12:32:59.998 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:33:00.006 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:33:00.006 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-24T12:33:00.006 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +873us[ +873us] +/- 82ms 2024-03-24T12:33:00.006 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 +6800ns[+6443ns] +/- 62ms 2024-03-24T12:33:00.006 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 -924us[ -924us] +/- 86ms 2024-03-24T12:33:00.006 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:33:00.007 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T12:33:00.015 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T12:33:00.015 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T12:33:00.024 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T12:33:00.148 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T12:33:00.218 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:33:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.251 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:33:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.279 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T12:33:00.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T12:33:00.327 INFO:teuthology.task.internal:Duration was 3256.479853 seconds 2024-03-24T12:33:00.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T12:33:00.336 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T12:33:00.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T12:33:00.338 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T12:33:00.402 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:33:00.429 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:33:00.646 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T12:33:00.646 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-03-24T12:33:00.646 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T12:33:00.668 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-03-24T12:33:00.669 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T12:33:00.693 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T12:33:00.693 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T12:33:00.711 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T12:33:00.762 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T12:33:00.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T12:33:00.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T12:33:02.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T12:33:02.414 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T12:33:02.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T12:33:02.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T12:33:02.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T12:33:02.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:33:02.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-03-24T12:33:02.571 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:33:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:02.616 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:33:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:02.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T12:33:02.642 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T12:33:02.643 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942/remote/smithi064 2024-03-24T12:33:02.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:33:02.729 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942/remote/smithi193 2024-03-24T12:33:02.729 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:33:02.773 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T12:33:02.774 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T12:33:02.776 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T12:33:02.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T12:33:02.833 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T12:33:02.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T12:33:02.842 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T12:33:02.842 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:33:02.844 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:33:02.856 INFO:teuthology.orchestra.run.smithi064.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 12:33 /home/ubuntu/cephtest 2024-03-24T12:33:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 12:33 /home/ubuntu/cephtest 2024-03-24T12:33:02.876 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T12:33:02.887 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T12:33:02.908 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T12:33:02.933 DEBUG:teuthology.nuke:shortname: smithi064 2024-03-24T12:33:02.933 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:33:02.956 DEBUG:teuthology.nuke:shortname: smithi193 2024-03-24T12:33:02.957 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:33:02.960 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:33.339692', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3JepwCWgr2yIJfaYMnolJhwy5Vh+NFlByQ2I6OMMzvfbKd8fFl+1ZS7elOZvNvwMQvB4EBeUM23bQLAmhU8R0='} 2024-03-24T12:33:02.986 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617942', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:33.340900', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOk9H8eEFFWVNlMKDLkBlGm/7Z0sxrZzum1Fu/7ki/WiPNgwmx2N/KtZ6zWh2I40SNDK2JBOx8Ug/KITGNVkyCM='} 2024-03-24T12:33:03.006 INFO:teuthology.orchestra.console.smithi064:Power off 2024-03-24T12:33:03.006 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:33:03.030 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-03-24T12:33:03.031 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:33:03.046 INFO:teuthology.orchestra.console.smithi193:Power off 2024-03-24T12:33:03.046 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:33:03.056 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-24T12:33:03.069 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-03-24T12:33:03.070 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:33:03.093 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-24T12:33:07.058 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:33:07.094 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:33:07.169 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-24T12:33:07.202 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-24T12:33:11.173 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:33:11.204 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:33:11.283 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-24T12:33:11.312 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-24T12:33:15.286 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:33:15.314 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:33:15.397 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-24T12:33:15.422 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-03-24T12:33:15.498 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-03-24T12:33:15.523 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-03-24T12:33:15.627 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3256.4798531532288 failure_reason: '"2024-03-24T12:00:00.000113+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-24T12:33:15.628 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T12:33:15.719 INFO:teuthology.run:FAIL